The Avengers did actually face the X-Men in the first X-Men vs Avengers mini back in the day. And the Avengers actually did pretty good in the fight. Never hurts to have a Captain Marvel and a Thor around.
Course the present X-Men is a different matter. It's every freaking mutant on the planet including the super villains. If the X-Men don't self implode (and they will IMO), it would take the combination of the entire super hero community to fight them if it came to that.
Again though if it comes to a war, I think the X-Men will again be divided. Guys like Wolverine and probably Storm will end up on the "right" side of history. They usually do.