this is fantasy, so not everything can be explained, when it comes down to something like this they can just blame it on magic- that makes anything possible 😉
But I like the idea of people from the outside always getting distracted, and taking a different route away from Storybrook as you mention.
I think Kitsis and Horrorwitz did say, we will see how the curse and the town effect outsiders at some point. So that will be very interesting.