He left his wife and young heir back in Philadelphia to be down at his Carolina castle with his lover and his art.