What if some people are so desperate that they decide to save their own lives by crossing the town line? After all, if, say, Granny isn’t in Storybrooke when it’s destroyed, would she die as well? Though that seems truly terrible. They would only have their cursed memories, so, as far as they’re concerned, their entire town would have suddenly, for no reason, vanished. Unless destroying Storybooke also removes the curse, in which case they’d know who they really are.