For one do you have alright tires?
Could be ball joints or something in that area
ive had the same problems and vehicle handled substantialy better when i got tires and got a camber kit all the way around the car. I used to feel like i was gonna die when it rained