You surely know what a ban is. It simply involves preventing a user from performing certain actions on a website or disallowing him or her from even entering it. Banning is a common (though imperfect) technique of combating so-called trolls, spammers, aggressive users, etc. "I will ban you by IP" is probably one of the most popular threat out there.
What is a hellban, though? Sounds terrifying. I met this concept a couple of years ago and found it really interesting. Suppose we have a really annoying, toxic user on a website. The idea is quite simple. Instead of preventing him or her from leaving any comments on the website, we actually allow them to do so. The gotcha is that no one (except for the admins) will actually see his comments! Why do trolls like to troll? To feed upon other users' negative emotions, to have fun looking at their aggressive reactions. The best tactic is to completely ignore trolls and not to feed them. In most cases, they just lose interest and leave the site looking for other prey. If no one notices a troll, no one reacts to his provocations, he has no point in trolling.
That's the main idea behind the hellban. From the troll's perspective, the comments are visible on the website, but for some reason, no one responds to them. From other users' perspectives, the toxic comments are not visible at all. Think about this: what would a troll do if you ban him? He'll create another account or use an open proxy to change his IP address and continue to misbehave. If, however, he doesn't see any reaction to his actions, he'll have no reason to make further attempts.
You may develop this concept even further and, for instance, allow hellbanned users to see all the comments (including ones posted by their hellbanned fellows)! That's a pretty interesting scenario: there will be a "shadow" community on your website that cannot contact ordinary users (well, most likely they'll understand what's happening pretty soon).
So, hellban does seem like a potential solution in fighting trolls, but no there's no doubt that it has some shortcomings. Still, you might want to apply it in one of your projects. And, once again: do not feed the trolls.