Here's the thing about Peter from my perspective. I think he's an AMAZING actor. But when I watched the first few episodes of his Doctor, I thought "I don't know about this". Something felt off. PLUS, I was super ticked that he wasn't going to be playing the Cardinal on Three Musketeers anymore. But that's neither here nor there.
Then, towards the end of the first season, I started to slowly see him as the Doctor and by the first part of his second season, I was all on board with him as the Doctor. I've since re-watched his seasons and they're a lot more appealing and enjoyable to me now. Sure, I still hate things like the sonic sunglasses and what not, but those things are minor quibbles.