We don’t know Hook’s true backstory, only the bit he said to Bae about having been abandoned by his father too. What he says his father and he boarded a ship with plans to travel the realms, but when he woke up in the morning his father was a fugitive and had fled in the night to avoid capture. This implies Hook was old enough to remember some of the tale, but we don’t know if he was Bae’s age, or Felix’s or a little tyke barely able to remember.
I’ve said this before, but I’m skeptical about the story of Hook’s father being true. Not only it was awfully similar to Rumple’s own that we were going to see in Manhattan before it got cut, but Hook had also expressed the intent to get the secret to kill Rumple from Bae in that very episode, so I can’t help but think he concocted that in order to get Bae to spill the beans. I’m waiting for Good Form to confirm its validity before I accept it as the truth.