Start by changing the sparkplugs. also, check for any exhaust leaks and unplugged holes in the firewall/floorboard. As far as slapping a muffler on it... try a decent used catback like a greddy or Racing Beat. Rotaries are harsh on exhaust and you need a good quality setup.

You may also need to check the MAF sensor to see if it is dirty, or if you have any air leaks between it and lower intake manifold. That can cause the car to run rich or inconsistant. You can also adjust the maf a little to dial in the reading a little more (more suited for an experienced gearhead).