I do not understand how someone calling you a Ginger finds that insulting to you. It would be like if someone said "you Brown haired person"

I really don't get how they imagine it is a bad thing to have red hair.
I think it makes them look like a moron saying it like it is some kind of insult. It's not. Just look at them weird since what they are saying makes no sense