Oh my, that is tough... I can't say I know how you feel, I would lie to you if I said that. My dad was also distant with me and my siblings and, as well, he's so caring with other children now. Maybe it's a copying mechanism, I don't know.
It's so sad your dad used to do those things to your brother and to you as well and of course it wasn't right for him to do that. But I hope you get to be in a place where you can avoid this trigger. Just know that you are not alone and even if there are people who judge you for how you look, there will be others who won't.
To me what triggers me is the feeling I will never get to experience love or a more intimate relationship with anyone. I've never been close to anyone, I've always felt left behind while the others around me are playing the game of love between each other. It's that moment when you go and meet new people and they end up liking each other, dating each other, having fun together and you are just there, alone, watching them all having fun and only waiting for them to come back once they are done with their dates. I feel like I would never be granted to chance to even try love because of some kind of curse that was placed on me.
Yesterday I had a sudden depressive mood, couldn't get out of bed, was crying all the time, I didn't want to talk to anyone nor see anyone, just wanted to sleep. It was a rainy day too, so terrible combination.
But yeah, my trigger is my own mind and I know only I can stop it, just don't know how or how to gain confidence after being rejected so many times and never had experience love before. (There are various types of love, I know, and I mean them all... I don't know why, but no matter what I do, people always leave).