There are multiple ways you can connect to another host or player:
They are within your distance circle
- Their ping is below your ping assist value
The Geo-Filter is disabled
- They are in your allow list
Regardless of what allowed you to connect to a host you'll remain connected to that host until communication has stopped for over 2 minutes.
For example if a player is within your circle, you need to leave the game they are in for over 2 minutes and their circle and square has disappeared from the Geo-Filter before they can be filtered again.
This is done so that people can't use the Geo-Filter to boot people out the game. If this didn't exist you could have your filter set large, then shrink it once you start losing to boot other players.