My guess is that Rumple is referring to the Blue Fairy. Seeing as Cinderella gave him a quill which she had enchanted that bound his magic. Effectively leading to Thomas’ disappearance, as well as Rumple’s imprisonment in the cave-cell that the dwarves had fashioned for him to ensure his magic would still be moot inside his cell. So that’s probably part of the history he’s referring to. It also wouldn’t surprise me if he hadn’t run into at least one fairy, that wasn’t too keen on him granting ‘wishes’ – “for a price”. When that would be encroaching on the fairy’s use of fairy dust and it would also mean they would be ‘obsolete’, as it were. Since from what I gather fairy dust only helps you achieve your dreams under your own power and Rumple’s deals while costly, do tend to be enacted at a much faster pace.