This is a great topic, I am in my early 40's and was sexually frustrated for most of my life. I would estimate I had actual sex maybe 150 or 200 times, and most of those times were with only 2 people, so most others were either one night or 2 or 3 times maximum. In simple terms, like someone else says, it doesn't really change you as a person and you don't wake up the next day with renewed confidence for the most part. Of all the women I was with, i think it was 13 total, only 4 or 5 of the people the sex was great, and the rest were pretty horrible experiences. So sex alone is not always good, i would say mostly it is bad. Staying with the same person, I have read and experienced that after a year and a half, 2 years, 2 and a half years, it kinda reaches a point where boredom sets in, so you can't just have consistent good sex with someone for 30 years, you end up just going through the motions.
I think a big reason i was not very sexually active is that a few women i really liked broke up with me and it scarred me so badly that I was kinda in a numb state for really years after. I could not easily move on to another person and if i didn't like them or they were annoying or clingy or whatever, I just figured i may as well stay alone instead of go through a stale relationship. The heartbreak or whatever i had to deal with a few times was so painful that I often wished I had never met those people, so there is always downsides to everything. Very hard to just look for sex and stay detached, sooner or later someone will come along that you are CRAZY about and when they cut you off, it is a horrible empty feeling and me personally I get suicidal. Even the little contact i had with those women (I never had a real relationship), I honestly felt more lonely with them than when I was alone, because if you see someone once or twice a week and they are working or have kids or responsibilities, you have SO MUCH downtime to be lonely. If you are just single and alone, you at least don't have to obsess about a person all week and miss them. Missing someone and not being able to see them is TORTURE. It is almost not worth the sadness you feel just to be able to see them a few times a week or whatever. People have their own lives!
As far as confidence, sex, even with some women who were gorgeous, in my case did not help my confidence. I was always so negative and down on myself that sex itself could not really help my confidence and make me feel happy with a new outlook.
Just on an animalistic level, they all say which is true, that if you have regular sex, it is WAY healthier than to just stay celibate. I am probably way angrier and more frustrated when I am not having sex than when I am, and I rarely ever had sex my entire adult life. I think people who don't have sex are in more of a funk, and ironically it is a vicious cycle. The more you don't have sex, the more you emit a negativity or off-putting personality into the world, which repels people more. If you are having sex, you will emit endorphins or whatever they are called which attract a ton more people to you.
The problem with me is that i think most people in society are with someone, break up, and eventually move on to someone else. I can't do that and if someone I like breaks up with me, I want to just stay home for years or even a full decade and not date, and you can't be that way in life, I get scarred and devastated too easily