A lot of these ideas are just flat out terrible. So, the X-Men, who are sworn to protect a world that hates and fears them, are instead hiding and not interested in actually protecting. Forget about the Chitauri, or Ultron, or the fact that Thanos has come to kill everyone. They're just hanging out somewhere not doing anything. It makes no sense, and makes the X-Men look extremely shitty.