Yes, often it is, and sometimes the ref can be at fault for this.
However, it is also used by spectators to simply explain away why their team is not performing as expected.
We do appear to have got ourselves into a state whereby the referee is the default villain for anything that doesn’t go your teams way on the pitch. Of course, rugby being the way it is, with much of it being reliant on human interpretation mixed in with the subjective element of “materiality”, blaming/criticising the ref is always going to be an open goal for those with a need to explain away their teams performance.
it’s because of this, that I get so irritated by ex-players criticising the refs in the media. These people portray credibility, and as such legitimise the ongoing criticism, especially amongst those who are rabidly one-eyed, game-ignorant or South African.