I don't know, for me eye contact was immensely difficult as a kid/teenager. I think it's because of a deeply rooted fear of being judged. My therapist tells me I'm something of an HSP (highly sensitive person), which I'm struggling to accept. But apparently a lot of introverts are HSPs so it shouldn't be that surprising. Anyway, HSPs are hard wired to be more sensitive towards the whole gamut of human emotions (including the pain one feels when rejected from a job interview, a potential partner, etc). For me, it runs much much deeper than those things. Even being left on read really hurts like you wouldn't imagine.
Anyway, to answer the original question, making eye contact is scary because what if they're not interested in what you have to say? What if they're secretly judging you the whole time and looking down on you, thinking of you as a weirdo? Looking at their mouth is so much easier for some reason. There's just something so.. confrontational about looking someone in the eyes.