"Is there something wrong with being more tactile?" Is this a judgement? The most irritating thing he's noticed, here, is that he can't tell when he's being judged. It's not upsetting so much as it pops him two steps to the left of on center.