Serena was often, moreso earlier on in her career, referred to as being 'manly'; compared to a gorilla, and also being accused of being a man. Some commentators including the head of the Russian tennis federation and the former male champion Ille Nastase referred to her and Venus as the 'Williams brothers'. So, yes, it has been there in the background for much of her career.
I'm a huge tennis fan and I saw the sea-change even amongst fellow players as to how both Serena and Venus were treated and talked about over the years. Both the negative and pejorative comments and slurs were more about Serena because of her more muscular physique.
However, she's not the only female tennis player to have had her physique questioned: Martina Navratilova had similar accusations made against her (apart from 'gorilla' obviously), as did Amelie Mauresmo and Sam Stosur.
And for all these tennis players, the comments also came from other players who would say things like playing against them was 'like playing against a man'.