So 36 is technically middle aged
No it's not. "Middle aged" is its own term, it does not mean "in the middle point of your life expectancy".
If you say someone has "one foot in the grave" you don't literally mean they have one foot inside in their grave, "technically" or not technically. When you say someone reaches the "age of majority", that doesn't mean they've reached the age where the majority of their life is probably finished. When someone calls their spouse "baby" that doesn't mean they think they are extremely young and undeveloped, it's just a term.
Language doesn't work by you doing fresh interpretations of idiomatic terms on the fly.