ABS increases your stopping distance and is less effective off-pavement.
ABS was designed to repeatedly releases your brakes (pumps your brakes) faster than you could, allowing you to maintain steering ability in panic situations. This increases your stopping distance.
In dirt you need the dirt/gravel/etc. to pile up in front of your tires to help you stop.
The last thing you need going over a bumpy trail is for ABS to kick in and now you can't stop when you want to... or even worse - when you are at the edge of a precipice.
With my FJ when I am in 4Lo the ABS is automatically disabled.