Chelsea, like most areas in London, has a wealthy area right beside a poorer one, e.g. the Worlds End estate. Being from Chelsea doesn’t really mean much at all, it definitely isn’t an indicator of “poshness” whatever “posh” means.
Chelsea being considered posh is a very recent phenomenon. In victorian London it was a bit of a slum area, and was seen as an unfashionable district, so it started to attract bohemian literary types. In the 60s it was considered pretty counter culture with Mick Jagger living in a flat there, and Viv Westwood (RIP) opening her SEX boutique on Kings road. It was hippyish, but not posh.
It wasn’t considered posh until ~the 80s when the aristocracy/Gentry started buying the comparatively cheaper flats in Chelsea for their offspring, which wouldn’t be too far from the family homes in South Kensington and Mayfair.
Maybe 50 years from now Chelsea will be a slum again and people will be baffled that it was ever deemed posh.