I worry about this every time I camp, but with the exception of certain obvious high-value items (any electronics, guns if I've brought them of course), I've always left my camp set during day excursions. It's a gamble, I guess, but I've yet to have anything stolen. I usually camp in the same spot for multiple days, so re-packing daily isn't really an option.
That said, one of the reasons to have a relatively fast/light setup is so you can pack and go without hassle. I dunno, like I said, I worry about it every time...