Goldenkey- you’re right about what BF said to Ba. This is part of the transcript from “The Return” :
(The Blue Fairy tosses him a transparent looking bean.)
Baelfire: What is it?
Blue Fairy: A magic bean. The very last one that is known to our kind. The others got away from us. You just use it wisely, and follow wherever it leads you. It will save you both.
(The Blue Fairy disappears. Baelfire returns home.)
So she says the others ‘got away’ from us…if Bae went to neverland, he meets up with Tinkerbell, she’s got one of the beans…he uses it to leave Neverland for a land without magic…maybe he spent all his time trying to find his father as Rumple spent trying to find Bae.