I got attacked.

[Kalist]

Even tho he's older than me, he was so small.

I didnt want to hit him back, I just let him do his thing, and ignored him as much as possible.

Well it's good you didn't fight back on some relations.
But on others... It isn't.

When you don't fight back, he gets the feeling you are weak. And when you are weak, he seek to you since you are the easiest to beat up, so he can release his anger.
In my eyes, this guy finds the happiness in hurting others and making them feel bad. Some people needs to make others feel bad for them to feel better. Since they may have a bad and/or harsh life.