the truth is, most people dont think of me at all, i mean if you are not rich or good looking, why would they
You don't have to be rich or good looking though. If you are fun to be around and/or charismatic and/or interesting, people will think of you and want to be around you. You can learn to be interesting.
The vast majority of people on earth aren't "good-looking", they're average looking (unless you use a lot of makeup, etc). Working out and getting muscle can change the way people see you. An average looking guy who works out and has alot of muscle will likely get more attention than a guy with a handsome face who is super scrawny or obese. And that's something we have alot of control over.
The vast majority of americans gain like 1.5 pounds of fat a year after age 25 yet lose like .25 pounds of muscle a year. You think that's natural or destiny or due to genetics? While those are all factors, it's mainly our poor diet and lack of exercise, which we have control over.