That really shouldn't stop you from enjoying them. Wear it, own it, be proud of it
If you wear it with confidence, it'll seem like a natural fit.
(warning: wall of text!)
...I've no idea what we're talking about here or what prompted this, especially since the episode is more than a few weeks old by now. Are you referring to Fenton, the old grump? If so, I recall talking about him earlier and bringing in race and class because he kept lumping people in groups as "the other." He referred to Rigsy and the victims as something like, "those kind" or "you people," and was just generally hateful all around. I don't remember exactly what I said in my review post, so I'd only be positing a guess between race or class regarding Fenton, but one thing to consider is that racism and classism often intersect, especially in city life.
But grouping people together and then blaming them via stereotypes is classic racist/classist behavior to begin with, because it doesn't seek to identify the individual or question the systems that are in place. Additionally, it was quite clear that the writing painted him as an antagonist, and so such hostile worldviews would be easier to put into an antagonist role. We're not looking at a guy who sees the world as roses, after all; he's not only deeply cynical, but extremely bitter at certain individuals. Who is he showing hostility against, and what do they have in common? Well, one thing's for sure: he definitely doesn't like the neighborhood that the episode's set in, nor does he like the graffiti art made in tribute to the victims -- everything's black and white to him with no imagination to conceive otherwise (which, on another level, is a good comparison to the villains, who provide really vibrant visuals in ways that delightfully don't make sense) -- and yet graffiti art is gaining more and more mainstream acceptance and even commissions, but traditionally it's seen as a nuisance made by people of color.
But in the real world, if someone is that bitter against a particular type of people, it's never ever handwaved as "that's just who he is," because then it not only excuses his behavior, but de-legitimizes any sort of harm he's done or he could enact; in the real world years of that kind of view would be called out. Indeed, I would hazard a guess that the writer envisioned the character as an older man to properly imply that he's got decades and decades of grudges bubbling under that bitter exterior; but that also means decades of development to get him to that level of douchbaggery that just dampens any level of imagination and thus empathy (and racism/classism benefits greatly from a lack of empathy). The Doctor notes that he's incredibly unique in that his lack of imagination means he's too stiff for psychic paper, something that's never happened before.
Now, I don't remember much about what I wrote about the episode weeks ago, but I appreciate that the show depicted that endings don't always fix things. The Doctor said as much when he made his observation that those who deserve to live don't always get it. And there's nothing to indicate that Fenton was fundamentally changed for the better from the adventure, unlike Rigsy whom the Doctor predicts a bright future, and the train conductor who's just happy to be alive (I love how Fenton just nonchalantly ties his boots in the end, not even dwelling for a moment what he just went through). And that's the complexity of reality. We live in a world where -isms don't need to be blatant, and are often very much subtle and thus "acceptable," making them insidious -- one of the biggest problems of "I don't see color" is that it fails to take into account the historical nature of racism, while simultaneously trying to sweep those present institutional, systemic problems under the rug, so there's a bit of erasure there in the name of conformity, and Fenton clearly adheres to a very strict definition of conformity. It also tries to dismiss accusations of racism from people who experience racism on a daily basis, whether blatant or through microaggressions. Lastly, it tries to suppress difference when differences could be positively celebrated, but consider: Fenton's big concern is cleanup and erasure and uniformity, but Rigsy's unconventional skills and his imagination ultimately help save the day. Imagining everyone as the same and equal and just like each other is not the same as imagining everyone with equal rights but the freedom to be different to add positive variety to the fabric of society. To me, that comparison could very much act as a legitimate analogy with a deeper social viewing.
Though, I will readily admit that there isn't much in dialogue per se for Fenton and the show rarely delves into social commentary, but rather I'm mining much of this from the strength of Fairbanks' acting. Fenton's an archetype yes, but Fairbanks, I think, imbues the character with a good deal of nuance and enough of a personal motivation that makes all this hypothesizing possible in the first place. If Fenton was played as just a generic grumpy guy, it'd be easy to write him off as a jerk for the sake of being a jerk, aimless and just pissing on everyone because he has a chip on his shoulder. But the way Fairbanks sneers at Rigsy and properly times his disgust is really intentional, so that Rigsy's not just a focal point, but a way to read Fenton. And there's a great deal of credit here that I'm placing in Jamie Mathieson for making all these choices that he did; the setting, the graffiti, the characters were all carefully chosen. Why they were chosen, only Mathieson can explain, but they're all deliberate.
As to your last item, Rigsy impressed the Doctor (not an easy thing to do), and the train conductor was a jovial but otherwise innocent bystander caught up in the events, the type of person that the Doctor saves all the time. And the Doctor postulating that Fenton didn't deserve to be saved is pretty damning considering that in the past he offered to save the likes of Davros and the Master; but they're important whereas Fenton is not only evil and unimaginative, but also unimportant (and that his bitterness makes him even less important), not worth the effort of saving. To the Doctor, Rigsy and the conductor were worth saving, but Fenton was an unfortunate and unintentional save.