Anyone who thinks that the standard of refereeing at the top (test) level is just fine is living in cloud cuckoo land. There are a couple of very good referees (Owens, Joubert, perhaps Walsh), then there is a huge daylight gap to the next lot. They are generally too slow around the park, too slow to react to what they see and the interpretations are so widespread from referee to referee that the credibility of the game is being compromised. RobLev's comments are partially right, but law knowledge does not automatically make a great referee.
I think the difficulty is less in the quality of the referees in question and more about where they are refereeing. In general the IRB panel refs are broken into distinct sub sections. Top 14(France), AP(England), Rabo and S15. Each of these league demad a different type of refereeing as they are different types of games much of the time.
In France you have referees who will happily see 2 guys beating the bejaysus out of each other in back play and play on only to wag the finger at the next break in play. They referee attritional rugby where teams try to over power eachother and win scrum PKs. This is often slow and hard to fall behind. You just need to keep a lid on the players.
In England they referee a game that can go from quick set moves to slow bring mud wrestling matches. Lots of kicking and PKs. Teams use scrums to engineer PKs much of the time, not to restart games.
The Rabo is a mixed bag. You can have anything and everything there from Leinster v Munster to Zebre v NG Dragons. For me the Rabo has the greatest spread of playing styles and referees are exposed to many more types of games and decisions. It is no coincidence for me that referees like Owens and Rolland who were/are amongst the best out there came through this structure.
S15 is fast with fast ball at rucks and sometimes little competition at breakdowns. Scrums and lineouts are to restart a game not to win PKs in most cases. It's a more user friendly version of the game and is better to watch but again referees only see so much there.
As there is no world league referees are going to bring a different views to games outside of their country. They are used to the trends of their native league and will apply them by default. Remember that their route up the pathway in their union will have been shaped by critique and coaching as to the type of game the union wants played there and that can differ largely from country to country.
At the end of the day the English will moan that the scrums were butchered, the New Zealanders and Aussies will moan the ball was slow and the French will just moan about whatever.
As the song says "It's the world in union". We all have our different cultures, that's part of what makes the game great!