Its not about just about the extra weight, but also about where it is.

The weight is far from the center of mass of the car so it shifts the center upwards - which makes a difference in the handling. To get the "effect" of that mass, you have to multiply the weight X distance from center of mass.

The net effect of having the sunroof may be equivalent to riding around with a 100lb dumbell on floor. (Only applies when cornering)

This is the reason the new M3s have CF roof panels. Well, that and for show