It's complex. I agree with your point that class is the great divider, but don't think comparions re: degrees of oppression are helpful. A poor white man doesn't have the connections, resources etc that a wealthy black man has, but a wealthy black man isn't immune to the effects of racism - a kind of prejudice that the white man doesn't have to deal with. And poverty can't be totally separated from ethnicity either, e.g. Irish travellers/Romani are overwhelmingly from deprived backgrounds. Disabled people are also far more likely to live in poverty. Ideally, diversity means taking into account ALL of these factors so that the people being included reflect a diverse range of experiences.
Sorry if that response was a bit wordy.