While you get 5 points for a registered user comment, you only get one for an anonymous comment, so that could be it. Maybe no one has trolled you, just gotten some anonymous comments.
And a troll is one who flies around JU and craps on blogs. It can be anonymous or a registered user. Thus when they crap, they are trolling.
Another likely explanation is the following:
You get 5 points for every reply to an article of yours, whether from someone else or from you (as long as they are registered users). However, if you reply to one of your own articles in the forums, you get fewer points. You still get the 5 if you reply to your own article there on your own page, but if it is from the forums, the point value is less (2 or 3 I think).
I believe this was done (at least partially) in response to "shotgun self-replying" or people posting lots and lots of one liner replies on their own articles.
Oh, and points suck.