I think he was quite a ladies man in his younger days. Though fathering and attempting to deny a child is a step above again!
My elder sister had a 'horsey' friend when they were younger who continued in competitive equestrian events including charity events. In her late 20s she was sat beside Captain Phillips when he was still married to Anne at one of these events, and he was described as extremely charming and wonderful, attentive company by my sister's friend at this lunch. I think she was quite besotted, particular as he was the exact opposite of her non horsey loving husband.
He apparently had a terrific sense of humour also, so I'm guessing he could charm any rider.