re: I'm going to become a 40 year old virgin.
No, it sounds like you've developed the makings of being a person who can become truly committed to another person. Most people are promiscuous and will have sex with people without any emotional attachment. As someone who's introduction to sex was like this, I would much rather I had someone that actually cared about me than to just lose my virginity.
It's easy to lose virginity, it seems incredibly hard (if not impossible for some people) to find a person you can truly love and who will love you back.
edit: I should add that trust is important as well.
Worry less about sex but think of other things instead.