The way Jefferson remembers is part of his particular aspect of the curse. It’s not an exception to the “only thing that can remember” in my opinion. In the Hat Trick episode, Jefferson said:
It’s hard enough to live in a land where you don’t belong, but knowing it? Holding conflicting realities in your head? It will drive you mad.
[adrotate group="5"]