I think for me it was because I had plenty of good friends when I was younger. I had a group of mates that I'd known for a long time - and that I knew would basically do anything for me. I remember calling one of them up in the middle of the night when I was scared in London one time and he talked me down. That same guy once said that I could make anything sound funny - it makes me feel good that I can do that. I need to be in the right mood though - especially nowadays.
Also I guess having girlfriends when I was younger - and getting attention from women. That helps a lot - it changes as you get older though, especially once you hit your 50's. Looks fade.
Having a family of my own has helped me a lot - knowing I have two people that love me, no matter how stupid I am. (and have definitely been)
All this talk about self-esteem though I find a bit silly tbh. I know on a few levels my self-esteem is not good at all. Because I know I've never been any good at holding down a job, and I never had a real career. I did very well at Uni but I've never once had a job that has even vaguely challenged me. They were all just low level jobs that I could do with my eyes closed. My problems with anxiety or whatever always got in the way. So I know I'm way behind many other people my age in that respect - and that I'll never own my own home. That embarasses me but I've come to terms with it now. Not much I can do about it anymore - I'm a bit old to start with all that nowadays.