I kind of got the feeling that maybe Dr. Whale had figured out what was going on in SB in general but didn’t actually remember anything. And feeding into the stereotypical doctor persona he maybe assumed he was this important person in FTL since he ended up being a doctor in SB. But then when the curse broke and he remembered who he was, he looked bummed because maybe he realized he was either some lowlife in FTL or kind of a not so important character (kind of like a side-kick). So he recognized that now people are going to realize he’s not really this big important doctor and they are going to know that he’s either a creep or a nobody. Kind of like he just lost his social status in his mind and in everyone else’s.