Taffy
My first season I did exactly the same things. It gave me a pseudo-confidence in situations that were totally new and unknown to me. An "I'm in charge" mindset if you like. Time, space, breaking contact and selective deafness are your friends with scenarios like this.
Blow the whistle, or whatever your action is, and stop the clock. Take your time, and move to space. Gives you the room to think your process through and just gives a bit of perspective - you're keeping your emotional chimp in the box (read the book, worthwhile).
If it is a confrontational situation with a player, break the contact. For example, if you give a decision and they jump up and stare you down, then move to the penalty location, turn away, or talk to another person. Card if needed and move, don't engage in a cock swinging contest to see who's going to blink or react first, don't give them the opportunity. Firm and decisive, but remove the confrontation.
And selective deafness; a useful skill to develop, but as you've noted - a fine line from one week to the next. Personal tolerances vary and there is nothing you can do about that, but recognising a comment said in the heat/frustration of the moment is different to something said pointedly and directed abusively towards you. Ignoring a voice and engaging someone else or using downtime to acknowledge you've heard something and it isn't acceptable can be more successful than creating a scene over something that could have been managed discreetly.
Good luck!