I've seen people do this before. If you are careful enough to get the trim off, then go for it. Just wet sand and clean them off real good. Once you get a smooth surface go ahead and lay primer and paint. Although, you may want to do all your trim this way because once you paint this, the rest probably won't match.