Favreau agreed to rush Iron Man 2 so he could also do Avengers, but it didn't make the money they hoped it would so Joss Whedon did Avengers instead.
There was some tension between Ed Norton and Marvel Studios after he rewrote a significant portion of The Incredible Hulk and didn't receive credit for it. He also didn't get the pay and screentime he wanted for Avengers, so they recast.