Let me be fair:

Pre 07 350z = Good bang for the bucks. Forget power, just improve on wheels, tires, suspension, and you'll have a nice handling car

07 350z = Very good in power because of the new HR motor. Expensive Z never the less

E46 M3 in place of an 04Z is not wise for OP's budget because it's about 10-12K more than the G. Insurance is a pita. Maintenance is a b1tch and costly.

If OP has more budget, then E46 M3 is definitely the way to go. Judging from his posts, I doubt it. So overall, the 04Z he bought is a good choice.