Do we agree that refereeing your own club can work with no problems, but is a risk and therefore in general best avoided?
exactly, so.
Neutral refs work better. This is, indeed, precisely why Societies were formed.
At age-group level club-refs have the advantage that this is, at least, normal practice and everyone is expecting to see a club-ref.
If two adult teams meet for a league game they are expecting to see a society ref. A ref from the home club is therefore slightly on the back-foot from the start.
In general I can't see why any benefit from a ref belonging to one club. Yes it's managable, but I why choose it if other options are available?