Yep. I even get that with friends, but it's worse around strangers.
I definitely think it's partly true that we give off some kind of vibe, but I also think that it's a vicious circle and it begins with the initial, irrational feeling that we're unwanted which leads to the seemingly-unfriendly behaviour that gives off the aforementioned vibe, which causes people to react to us negatively, in turn reinforcing the initial anxiety.
In my case, my friends don't react to my anxious behaviour the same way strangers do, so the anxiety isn't reinforced and is more manageable, but around strangers, it can be pretty bad. It all depends on how they react, the vibe they give off (certain people are more likely to trigger the initial fear, I find) and the mood I'm in.
Of course speed, MDMA, a bit of weed, opiates, benzos and alcohol all prevent this from happening in the first place.