If you can prove that the verbal agreement existed then the verbal agreement will hold up in court. If you by chance had a witness to the agreement, it will stand. If it does go to court and you bring up the verbal agreement with proof, chances are he will buckle and they will catch it which in turn will make you not responsible for the money that was already paid.

Document EVERYTHING and get a report done asap. Worse comes to worse, everything is on file and ready to go. May end up never having anything come of it but better to play it safe.