I disagree because Pietro comments their fondest desire would have been to become X-men but that now they had a compromise with the Avengers,Wanda agrees with him and even thinks over if they are making a good decision.
Also even the X-men didnt know where Magneto was when he was a baby, Xavier send him to the Muir Island to be taken care of by Moira but he was isolated there in a cell so even if they had joined the team, they would not have had contact with him.
Still I donīt think it was a bad decision to stay with the Avengers,they did what they thought was best at the time.