Love won't feed me, or my partner for that matter. Fearing loneliness has never been as bad as fearing going hungry, sleeping in the streets, having to do degrading things for money, winding up dependent on others for care (especially in old age), having to show up at a job I hate forever, being gravely ill without the money to help myself, never having the money to help the people I love, having little to no control over where I can afford to live, etc.
I've watched the misery and skyrocketing health problems of people who have to work at jobs where their boss harangues them constantly or even screams at them. Heart and health issues run in my family on both sides, and my mother spent the start of this week in the hospital with blood in her lungs - made worse by workplace stress.
If I smartly invested all that money, I'd probably never have to fear those things. So, yeah, call me shallow but I'd rather be safe and lonely than in danger and with one single companion.