Referee assessment and development is a complex process and takes up a lot of time for all involved.
Your Society Training Team, the Match Observers, Assessors, Advisers, and Referee Coaches have specialised training courses and assessment themselves, as well as CPD in the same way referees have.
Feedback is just one aspect of the development and assessment process and can be structured or casual. I am intrigued to know what makes you so obsessed with asking for feedback from groups such as physios and spectators - at grassroots level some feedback from captains, coaches and players can have great value on both formal and casual basis and some is totally useless. Likewise you might have conversations with others, and make your own choice as to whether you use it or not.
Few of them would be able to offer structured, objective, constructive feedback, let alone turn it into solutions, advice and positive development proposals without undertaking the RFU assessors / trainer courses.
Many referees already get feedback from captains through the formal Society Match Card process, and engage in less formal feedback with some players, blazers, and even spectators. So do do Match Observers / Assessors as over the seasons we build up contacts and friendships with Coaches, Committee men, spectators and some players - all of whom's experience and judgement we might respect, accept or take with a pinch of salt.
Ask who you want for casual feedback but filter that feedback carefully. The formal assessment and feedback process in the Society is key as the foundation for your development, but of course other feedback you might get is useful.