So it follows; if quiet people are often perceived as a) arrogant/stuck-up, b) just nervous/timid, or c) abnormal, they're much more likely to experience rejection and other negative reactions, and many people don't tend to tolerate obvious differences from the norm very well; especially if they imagine that the 'different' person also thinks they are better than others.
From my experience, I'd say that quite a few people think a) when they encounter a quiet person, and there do seem to be many insecure people around. If a person remains quiet, it leaves room for others to imagine all kinds of things about them. I think that being quiet in a social setting is sometimes more likely to give off an air of quietly sitting in judgement.
If the quiet person is seen as just nervous, they might suffer other people patronising them and seeing them as weak/timid in some way. I've had this quite a lot.
If seen as abnormal, they might be lectured on how to become more 'normal', or be laughed at, or feared. I've been bullied to be more 'normal'. I've also been called a serial killer
That's a pretty big downside, and I've experienced all of those things. Thankfully, quiet people can often be perceived as trustworthy, intelligent, cool, sophisticated, interesting, patient... . If only this was for the vast majority of the time