In these engine the starter contacts do go bad and they are an easy fix. Could be the problem but, I would make sure all the relays are good, and check the starter connections. Also check the positive battery cable.
Hopefully its just the contacts. I replaced them on my truck, and buying them from Toyota was 7 bucks for two. Takes about 30-40 mins to change em out.
Heres this.

http://www.4crawler.com/4x4/CheapTricks/Starter.shtml