Not only were the X-Men not popular figures in-comic, but mutants in general were not liked, especially evil ones like Magneto or anyone associated with him. As Pietro stated, he wanted to not always be associated with/referred to as a mutant.
Out-of-comic reason was Stan wanted to write the Twins.