You shouldn't assume anything because not only can you not distinguish the good from the bad but you admitted you can't distinguish muslim from non-muslim so it's illogical that you say it is a factor you consider at all. It's like a blind person saying they are suspicious of blonde people. Not one of the terrorists who carried out an attack in the US was wearing muslim clothing when they committed their crime and some of them were not middle eastern looking so it is illogical to use that as evidence of a threat. I don't see a logical reason to assume middle eastern looking people are any more of a threat than any other kind of person and as far as I can tell.
I don't care if you hurt their feelings either. You're right that being aware of the risk doesn't mean you exaggerate it. But assessing risk based on characteristics that are not significantly correlated with their danger to you is.
No, it is limiting one method of self defense but that is quite different than not being able to defend yourself. If someone punches you in the face, you can't say, "Well I don't have my gun so therefore I can't defend myself". Do your first amendment rights no longer exist because you can't commit libel?