It's not as if its a terrible thing to think "Aw, you should thwack 'em!", I get it, I've been there and done that too. Making the odd remark like that isn't a big deal but I can end up on the receiving end of a lot of them and cumulatively they send an unintended message. It's much better if people just keep in the back of their mind that this might happen, and speak up if necessary.