I'd feel better about the loyalty if it was two-way, but I don't see that, most of the time. It's very much Alfred serves Bruce. I think the first time it really bothered me was reading old issues when he leaves Bruce for Bruce's own good, he said (after Gordon got shot). What does he do - go work for Tim. Oh, I get it's too keep him in the comics. But if you actually think about it, it's so problematic. There's nowhere else for him to go? No interests to pursue? Nothing else in his life but serving? Serve and only serve? And it's not like Jack Drake is paying him then.
I do see his relationship with the others that way - all perception, of course. At the very least, they matter less than Bruce. His relationship with them may be better, but he generally expects them to do what's best for Bruce. The Bruce-amnesia-story particularly gets it for me. But I'm particularly salty about lots of their family dynamics, and Alfred is definitely not exempt from that for me. And it ties in so much with Bruce's increasing emotional damage for me, too. Partly coincidental, just on the timing of the changes making him Bruce's guardian and Bruce's psche changing. But other times, I can see action and attitudes that I think contribute. Once the idea is your head, you see it everywhere (like noticing cars that look just like the one you bought that you never noticed before you had a reason to think of that car model in particular).