Well, it's hard to keep a hero relevant if they're not fighting, specially since back then the stories were more about the action than anything.
The "we don't want to fight anymore" was likely just an excuse for them to not join the X-Men, which works well enough given the situation, but them joining Avengers a week later, yeah... At the very least, Avengers#16 does mention that they don't want to be treated like they're different:
So that works well enough.