Well, he became a stateless refugee as a toddler, living on the charity of relatives as they had no cash, mother in a psychiatric hospital and father off living the high life with mistresses, living with different relatives every school holiday. He would have scrubbed plenty things at Salem and Gordonstoun and as a naval cadet, living without heating, and, then as a young officer went through the thick of WW2 at sea and being Mentioned in Dispatches. His mother spent WW2 in occupied Greece (and hiding a Jewish family, hence her status as Righteous Amongst The Nations) so they would have had little contact, each worrying about the other; she only agreed to leave Greece in the early 1960s when he was worried about her during the military coup. His older sisters all married quickly to get out of depending on family charity. They all married German princelings so were on the opposite side during the war, again with little contact between him and his siblings. The sister closest in age to him died with all her family in a horrific plane crash just before the war and that's why there are pictures of him aged 16 in a procession with German Army and Nazi officers; it was the funeral of his sister, German brother-in-law, and three nephews, including the child that Cecilie gave birth to during the crash. That all sounds pretty stressful. His mother had to break up some of her inherited jewellery to give Elizabeth a decent engagement ring fit for a future queen as all he had was his navy pay and couldn't afford that.
Since then, it's been very, very different but I would suspect that his childhood spent being paid for by relations and with his beloved mother locked away had some effect on his personality as well as the stress of his first 25 years or so.
ETA: my father was a couple of years older than Philip and chose the Merchant rather than Royal Navy as the Navy officers pay at the time was calculated on the basis of having a private income. Philip may have had the background and relatives but he wouldn't have had much money, and that was why GVI and QEQM disapproved of him.