Lifes too short to worry over shit like this IMO. I don't worry about it, if it doesn't work out it doesn't work out move on.
If you think your friend is being shady/scandolous then lose him as a friend