One word - Advantage!
A fuller answer:
If the incident was that far from play you would not see it, in all probability, so I'm not to sure about your extrapolation. If you did see it play advantage. Make sure the offenders and non offenders know you've "got it". That way the score would stand and you are able to go back to the offender and deal as required.
The basic rule we are given is "Foul play is always material" Similarly at younger levels "Advantage does not apply to foul play". Yes, you never should say "never" or "always", as there is always an exception to the rule. But it is a good starting point.
I've watched too many games (at semi pro level) where referees have "managed" fould play with a chat "because it is not material" only to see the inevitable punch up as players get frustrated. Players, and supporters get frustrated at my level when they experience / see foul play that I don't give because I don't see it they get frustrated. So when they know you've seen it and still "do nothing about it" - except a "Don't do it again, please".