Baron Helmut Zemo and/or the Masters of Evil
Count Nefaria
Dormammu
Grandmaster
Immortus
Kang the Conqueror
Kree Empire
Loki
Michael Korvac
Scarlet Witch
Skrull Empire
Squadron Supreme
Thanos
Ultron
Zeus
It's Ultron, then Kang, then Zemo.
It's helpful to remember that while Zemo's grudge started out being almost totaly about Captain America by now he has a grudge against the entire organization.
And if you take his time as an anti-hero into account his grudge is almost justified.
Cap (and by association the Avengers) will ALWAYS see Zemo as a villian no matter what good he does while forgiving Black Widow and Wolverine for thier past crimes.
...and Hawkeye took his girl, which is reason enough to hold a grudge right there.
Helmut Zemo may have started out as a Cap baddie, however I see him as a legacy villain. Unlike the Red Skull, Heinrich Zemo was created to be Cap's opposite in a team that would bring together villains from other heroes' books. Heinrich was the leader and founder of the Avengers opposite, the Masters of Evil. And Helmut, his successor, led them in their greatest victories (something that not many super-villain teams can claim having, even if temporary). As such, I'm pretty confortable listing him as one of the top Avengers baddies.
Immortus is another one that I feel people tend to underestimate and overlook as an Avengers baddie, simply because he is another version of Kang, and a more manipulative one than an openly confrontational one. However, one has just to look at Avengers Forever to see how dangerous Immortus can really be.
Peace
logically scarlet is the avengers greatest villain both in-story and out-of.
she ushered in the biggest sea change for the books themselves. her attacks led to a company-wide crossover, which then drove the line of the x-books and provided the tone for avenger books for years. in-story she has the highest avenger and non-avenger kill-count of any avenger villain.
ultron has never had the sort of antagonistic influence over the avengers that wanda has, and he certainly never hurt them as much as she did. he definitely never killed as many avengers as her. kang has about two stories where he's actually had much influence over the avengers, and neither took place in an avenger book proper.
How did we get this far without any mention of Count Nafaria?
The guy went from being an Italian version of Auric Goldfinger to powering himself up into a team busting Superman-clone. Dude literally bought his way into the A-list power-wise.
The only reason he isn't murdering the Avengers as we speak is because he wants to be a Don and run his criminal empire from behind the scenes old school style.
Going out in the field is for the hired hands. He has people for that.
Being a guy that hcan beat up Thor just guarantees nobody will mess with his underlings.
I place Nefaria alongside Korvac. Sure, he had a couple of awsome showings (the Shooter/Byrne three parter and Nefaria Protocols), but, that was it. To me, a definitive Avenger foe has to be a recurring one. And unlike Korvac, Nefaria had another couple of pretty lame appearences (his first appearence, against the original team _ even though it did almost cost the Wasp her life, and that ridiculous jobbing against Bendis' New Avengers _ "Oh, I can stand against all the mightiest Avengers but a few bullets, Wolverine scratching my face and a couple of love taps by Ms Marvel will do me in..."). Loki and even Thanos come first in my list than those guys.
My all-time favorite Avengers story is Assault on Olympus, by Stern, Buscema and Palmer, but Zeus (which can be seen as the main baddie of that particular tale) doesn't even make my list. I consider Ares more of an Avenger villain than him. So, IMHO, a great showing might give someone a place in the list of greatest Avengers villains, but, IMO, shouldn't earn anyone a top spot (also looking at you, Thanos).
Peace
Kang would have to be my go-to villain, since he's the villain that is at the center of 2 of my favorite Avengers stories of all time, the Celestial Madonna saga & Avengers Forever.
Yeah I never really saw Thanos as a villain to the Avengers he was always Captain Marvel, Silver Surfer and Adam Warlock villain. He was just made an Avengers villain to say hey we need Thanos to fill the Darksied role for the team.
Avengers Forever one of my all time favorite stories on Kang I have not read Celestial Madonna how is it?
I wouldn't say it stands well the test of time (Englehart's writing can be really dated, and they really drop the art bucket towards the end, making the whole experience a little less palatable), but it sure set up a lot of the MU history for the following decades. It's not one of my top favorites, but I'd say it's required reading for any true Avengers fan. However, don't expect a nuanced Kang as the one in Avengers Forever. Kang in CM is pure, all evil baddie.
Peace
i think its very weird that the avengers don't have that many noteworthy villains to call their own.