It's not necessarily either imo. People can look a lot at people, for all kinds of reasons, this is true, for sure. But they do this to *everyone*. People with SA absolutely have bad confirmation bias and remember the negative times and then this flavours the memory of it all. I will remember the times people stare at me and assume it's because (insert a negative reason), but sometimes, it can literally be the opposite (they find me attractive). I get it a lot because I am physically intimidating too, it's just how it is. But I tend to put my own special SA spin on things "they hate me, they think I am ugly, blah".
A very very significant % of it is confirmation bias and because we are looking out for it, see below...
Often forgotten in these threads, to know someone is looking at you, you kinda have to be looking at them
I just imagine people bothered by this stuff going around looking for people to see if they are looking at them, and them actually staring annoyed with people, so they stare back "see, everyone looks at me funny, they all hate me".