Well since everything is always “for as long as I can remember” I would hazard a guess he just has no memory of anybody ever entering the town…kinda like when most were surprised to see August even though Regina was fuming and demanding Emma do some recon.