There is a way where you won't be left out. What you can do is to ask questions about what they are talking about. That will usually get them describing it, and then that will usually bring up even more discussion about it or it will lead off into something else. For your situation they probably would've told you that you need to start watching it because it's a good show. You still won't feel like you're in the inner circle of the discussion, but you'll feel more involved and the group will notice that.
I've tried this once or twice with the guys at work but they mostly just dismiss me as an idiot for not knowing anything, *meh*.