Ice hockey sorts this out by each team having two strips: "home" colours (predominantly a solid colour) and "away" colours (predominantly white, with flashes of your preferred colour).
Since every match has a deemed home and away side (even at something akin the RWC) it's always clear in advance what strip each side should wear.
And if rugby did something similar the ref would know in the colours in advance and could wear a contrasting colour.
And it would be easy to call, too - scrum white, pk red.
Of course ice hockey sorts the ref's kit out too - refs always wear black and white vertical stripes. So they never have to change, and there's no problem distinguishing them from any of the teams, even in the heat of a melee ?
Why couldn't rugby do something similar?