The book of Peter Pan was written in 1902, the the story itself was set much earlier in London. Like early 1800s? And we know that Bae falling into the portal took place centuries before the FT we know, so if time is linear, then yes, he would have landed in our world roughly during the time of Peter Pan. And probably went/found a way to get to Neverland, stay there for awhile and then jump back to our world at some point and grow up to be Neal Cassady.
Maybe because he feared his father maybe coming close to finding him, so he jumped again.