Most of the violence and stuff associated with drugs comes with the fact that the drug market is an illegal market. Its usually people trying to either protect their drug operation from theives, snitches, or the police.

In Amsterdam and other places where the use of "soft drugs" like MJ and hash is tolerated, the use of hard drugs (heroin, crack, etc) is much, much lower than in the US.