There was already a decline with Avengers Age of Ultron, it made less at the box office than Avengers. After that came Civil War, technically an Avengers movie and so packed with characters, even Spideys "return" and it made less than Age of Ultron at the box office. That says something.
I believe the biggest mistake Marvel can do is trying to change the actors and pretend its the same character. If Downey jr. is out, Tony Stark should be gone too and replaced for Riri Williams (though i dont believe that will be the better idea).
The longer i think about it the more i believe Marvel will take a hit when
A) Actors change
B) Characters will be replaced
C) Reboot
Well, its a lose lose situation