My guess is that early on Joe
a) thought Elle was a normal girl, as in not a delusional, emotionally immature narcissist
-and-
b) was kind of feeling himself as he rose up the ranks at his big law firm and had some condescending attitudes about Elle making "girl money". Ew.
Then, my guess is that Joe realized that he had taken a wrong turn, heard the howling winds of the ceaseless money vortex, and plotted his exit.
Elle stayed behind in fantasyland, did not grow or learn, and grafted her crazy onto fellow fabulist "Professor" Rick.