Perhaps Rumple’s father is considered cowardly because he didn’t want anything to do with Rumple or his mother. Men who get women pregnant then just turn their backs on them are considered cowards for not taking responsibility for their actions.
Rumple’s biggest fear could be that he wouldn’t be a good dad to Bae just like his father wasn’t a good dad to him.
As for the spinsters, perhaps they are his mother’s sisters.