For me personally I feel its only a Defenders team if one of the founders are on it, most especially Doctor Strange, he just always felt like the leader to me. Surfer, Namor, and Hulk were kinda just there because they had to/wanted to be and left when they needed/wanted to, but Strange's Sanctum was like the base.