You should've told them what you are about, that you have social anxiety, tell the truth. A lot of musicians have SA or are aloof in some way so don't feel alone. With that, that isolation can cause you to produce beautiful music.
Have you ever considered starting your own project and making your own music? You could produce a few songs, make a myspace site and write you are looking for musicians to play with. You have to make that time be your time.
To be honest, based on what you wrote, I think you have a lot of potentially musically and creatively, I can tell when someones feel music and when someones doesn't.
Also check out and listen to Neutral Milk Hotel. Your picture reminded me of their images on their album "In The Aeroplane Over The Sea."
I hear ya. I've always wanted to start bands or join local ones I liked, but I never had the nerve or know-how. There's circles of local musicians that I halfheartedly try to fall in with and it never goes anywhere. It's the "You guys playin' cards?" syndrome. I stick to solo recording which is fun but limited.