I would start with your fan switch. Do you have a volt meter? If so, check if you're getting voltage to your blower. If not, it's definately just the switch. I'll putmy money on the switch.

If it still blows once in awhile, then the blower is fine, power is just not getting there. If its cold when it does blow, then the system is charged and doesn't need to be charged.