I've had many friends that would have held me back if I stuck with that lifestyle. I still hang out with them occasionally because they never wronged me at all, but there's a certain point where you have to put yourself first. You don't have to drop friends, just make sure you keep your priorities straight. They don't need to have intelligent conversations for you to keep going, they aren't dumbing you down. Just keep your shit together and it wont matter.