Either way, OP, you can't go wrong with either. A SR5 with ATRAC will get you out of most situations, a TRD Off Road with read locking differential and crawl control will get you out of a few more. It all depends on what you intended uses are. An SR5 is incredibly capable out of the box, especially after a tire change. A TRD Off Road, a little more. Full disclosure, I bought a TRD Off Road over an SR5 for a manual transfer case lever and aesthetics.
Also, it's your $, buy what you want. During the brief time I spent in auto sales, too often I would see customers have their purchase influenced by a friend/coworker/family member, etc. I would always ask them if those individuals were planning to make the payments, if not, then they should have no say in what vehicle you buy. You're the only one that needs to be happy with the car.
Regarding new vs lightly used, it's sometimes easier to get a better deal on new (especially of TRD trims levels). I bought mine new because I got it for $1,500 over the cost of a 3 year old, 36K mile, from NJ, that was showing signs of rust. Also, go on the last 2 days of the month. Dealerships are willing to take bigger losses on the sale, if it means they'll receive larger volume bonuses from Toyota.
ATRAC Demonstration
ATRAC used here
Stock SR5 in Chile