I personally just remove one headphone from one ear. They luckily tend to make it short and, if not, then they probably need my presence anyway, so I leave them entirely then. No point in trying to commit to headphones when it definitely seems like it is not happening there and then.

I am either part of a conversation or I am not. Unless its a thing between 5+ people, you will get that choice and if you cannot choose, I will choose for you.

Zero tolerance policy for dialogue-blueballing.