I think originally, the X-Men were supposed to be a metaphor for communism, rather than a metaphor for minorities or LGBT people, then Claremont came along and well, we all know the rest. What also didn't help is that very few X-Men were people of color with the notable exception of Storm...ugh. Kitty Pryde saying the N-Word three times and comparing it to the term "Mutie." What made it worse is that the X-men constantly sat around their mansion feeling sorry for themselves most of the time. Even though most of them could easily pass for human, maybe that's why other superheroes didn't like being around them.
The metaphor also falls apart in universe as well, I mean really, what's the difference between Wolverine and Captain America, one was born with his powers and the other one got them from drugs? So being born with powers is somehow worse than getting them from some random accident or an experiment?