I think trailers are great for a lot of applications. But I don't like them, having had experience both with trailers and without. There are three reasons I don't like them for my application, and I think these reasons are worth considering for others, but as with all things a person has to settle on what works for them. There are folks like BillieBob and Road (and many others in this thread) that have excellent trailer setups that do what they need to do and there's no reason others can't do the same.
1) Vehicle Design Limitations: There is only 1 off-road vehicle maker I'm aware of that provides designed specifications for towing in off road environments, and that's Land Rover. The off-road specs for towing on some Land Rover vehicles are as much as 1/3rd the weight of the 'normal' towing capacity for on-road travel. Other manufactures do not publish off-road specs, so people see they can tow 3500lbs with their JK and haul a 3450 lbs trailer off road, and that is well beyond what is safe for that vehicle -- the forces over uneven terrain are amplified hugely as compared to on-road towing, which is what the 3500lbs rating is talking about. If the majority of your travel is gravel roads, then this isn't a big deal. But if you go off road into tricky terrain, while the trailer may not feel like it's slowing you down/can't follow you, it does increase the risk of damage to the two vehicle.
Related to this, there is also a flood of 'off-road' trailers hitting the market now, and not all are created equally. Putting Jeep tires and a graphics package do not an off-road worthy vehicle make! So, between the increased risk of trip-ending failures in both the tow vehicle and the trailer, I prefer to go without.
2) Time to setup: For us, we like to explore all day and actually spend very little time in camp. Some nights, we'll stay in the back-of-beyond, other nights we will find ourselves having to stealth camp at a Wal Mart or something similar. The trailers I've seen (with a few well-designed exceptions) typically take longer to setup and teardown than a vehicle-based system, and they are typically more obviously a "camp".
3) Agility: Being able to turn in the vehicle's own radius is a huge asset off road. Trailers can get in the way on really tight switchbacks and some tracks. If you don't do this kind of track, then this is a moot point, but if you do, it'll be harder with the trailer and can take the enjoyment away a bit. A trailer + vehicle will always be less agile than the same vehicle without a trailer, but again -- this may not be a need for everyone.
Trailers have a lot of advantages though -- more payload, space, etc. in the tow vehicle, and the ability to base camp are both huge assets. Plus, I love the idea of "hook on and go"; we design our vehicles to be like that without a trailer, but that's not possible for every one who have other uses for their vehicles too. Ours is a DD, but we don't use the cargo space for hauling since we have another truck for that, so our Canyon will be set up where all we need to do is grab the dogs and hit the road (Our old JK was setup this way too).
