On a serious note I hate wearing headphones/earbuds in such situations. I prefer to know if I have to do something quickly rather than sit unaware of the nutcase on the other side.
Nah, @Wombat. The only time I’d take them out is for announcements. There’s actually very little crazy in aggressive crazy kind of way. PSOs discourage a lot of it. And the line I’m on is generally good. And I travel during business hours. Most daros are a sleep at that time. So no, earbuds stay in 
Unless someone is like, hey you sexy man, my name is Bianca. Then I’d definitely take them out and ask what happened.
I catch public transport (bus only, no trains here) a couple of times a week. I wear headphones and listen to podcasts, but like to sit close to the back so I can keep an eye out for idiots. Only had a couple of minor issues in many years. In both cases the police were mysteriously waiting a couple of stops down the road and removed said idiots from the bus without fuss…