Speaking personally, I blocked out a lot of people I found annoying. People I thought were too stupid, too proud, too violent, too lazy, too exclusive, too immoral, etc... If you asked me, most people ended up being excessive somehow. Took some time to admit this, but it eventually taught me something: it wasn't them, it was me.
Maybe this is going to be counterintuitive: There probably is no reason why you get irritated anymore, maybe there was once, but you've just learned to associate negative feelings and motivations with a certain person. They probably haven't done anything, and you react to them negatively out of habit more than much else. It spoils relationships and it's a drain on your energy. So maybe get out and take a walk, a deep breath, or remind yourself how cool the person is, or go find someone else to hang out with for an hour or so to distract yourself.
Some folks I know still try to make it out like it really is an irritating person, but their reasons are such bull and I wonder if they know it deep down. It's just a person, nothing more. How you deal with it from there is up to you.