When I said, kill I mean, do something bad. I know for example, if I responded back with an insult, it would only prove his point. This isn't big enough to file a complaint for. If I complain, it proves his point. If years later I take revenge by giving his kids a pain in school, will look like a jerk.
So what can I do ? Just ignore it ? I want to ignore it, but it keeps happening.
If there's a specific person who's making these comments to you on a regular basis, then yeah, it's big enough to file a complaint over. It's workplace harassment and you shouldn't have to "just ignore it".
First, you tell him that you want him to stop the insulting comments. He might not even realize it's doing any harm at all, and that could be the end of it right there.
But if he still keeps it up, well, you don't just walk into your boss's office and complain verbally.
What you'd need to do is to start documenting every one of his inappropriate comments. Note the date and time, and names of any witnesses who were around to hear it. You could pull these witnesses aside and ask if they'd be willing to back you up, should you go forward with a formal complaint (plus for all you know, other people could feel the same way about the guy).
Once you have some documented cases, take it to your HR department or his supervisor (or yours), or whoever's in a position above you both.
If it's bothering you enough to go on this kind of a rant about it, it's legitimate enough to file a complaint over.