Yeah, being more durable than he is strong makes sense, but to be sufficiently durable to make Wet Namor and an island busting Hulk need more than one serious hit to KO/explode him would also make him so durable that anyone below class 100 would break their hands on him before making him blink. Now, I don't follow the FF too much, but does he normally do things like ignore punches from She Hulk or alternate universe Ben Grimms?
I was looking through WB's few appearances, and here's the kind of stuff he was doing:
Crossing blades with Shanks parted the clouds above them;
Punked Ace with casual ease, repeatedly, even when attacked while he was asleep/not looking in his direction (Ace being a guy that was shortlisted to be a Shichibukai and was dead even with Jinbei after 5 days straight of battling);
Casually created seaquakes strong enough to create 2 tsunamis that each towered over the battlefield, which included several hills/mountains and tons of giants;
Was able to create air-quakes that intercepted Aokiji's projectiles and even Aokiji himself, Aokiji being a Haki user that had just gone from sitting in a chair at the execution site to being hundreds of feet in the air (from where he froze both of the aforementioned tsunamis instantly) pretty much instantly;
Was able to see and nonchalantly react to attacks from Mihawk and Kizaru, both of which were shown to be ridiculously fast even by the standards of high end bullet timers like that era's Luffy and Zoro (he didn't counter them, but that's because he was letting his subordinates do it for him);
Casually caught a chunk of magma with his bisento and blew it out like a candle (see prior scan);
While watching Oars, Jr. die, backhand air-quaked a featless giant Vice Admiral's weapon without looking, caught him by the head and casually skull/brainquaked him down instantly;
The earthquakes he was causing were shown to affect islands miles away, long before he got close to going full out;
Was able to sense Crocodile about to attack him from behind, though he didn't seem to be bothered to defend himself before Luffy intervened and then he proceeded to casually chat with him instead;
He did get sucker stabbed by Ouzugumo, but that was from a normally loyal and beloved son who had just been chatting with him right before he stabbed him (a point against Haki-precog, though being done by another famous pirate captain who is extremely familiar with Whitebeard and strong enough to break out of the hold of a pissed off Marco who had himself been going toe to toe with friggin' Kizaru earlier; everyone was surprised that Whitebeard had gotten so ill that he couldn't dodge a surprise attack from someone of that level who had attacked from surprise, though);
Subsequently, he created air-quakes that shattered both of the massive frozen tidal waves (admittedly, one of them was sliced in half by Mihawk earlier, but that still left the other intact);
Casually caught a double-handed slash by some giant called, appropriately, John Giant with a one-handed block using his bisento, then grabbed the air and created a quake strong enough to tilt the entire flipping island, including the waters around it, notably;
He then jumped into the air and repeatedly caused quakes inside John Giant's gut without touching him, causing him to gush blood from the mouth and drop, while also shaking the entire area except for the execution platform itself where the three Admirals are doing... something, probably Haki related, to hold off that power;
He was unable to break whatever material the sea walls were made of, though since it caused his powers to bounce back, it might have something weird about it, especially considering how much Seastone and Vegapunk junk the Marines throw into their tech (it did dent notably, though);
He sent an entire crowd flying with a earthquake enhanced bisento slash;
No-sold being frozen solid and broke out using his quakes, then shanked Aokiji before he could react and held him in place for Diamond Jose/Jaws to shoulder check him;
Caught Luffy with his off hand when Kizaru sent him flying with a kick (its unclear if Kizaru kicked him towards Whitebeard, or if Whitebeard intercepted);
Matched Akainu strike for strike for a bit before casually backhanding a magma dog that was sent at him while he was focusing on rousing his troops against Garp (it should be noted that every attack is still causing massive quake damage to the island, which is why Akainu was intercepting him at that point);
His illness (possibly also stab-wound, but illness was the primary stated reason) caused him to collapse to his knees puking blood and then Akainu shoves a lava fist into his chest and a bunch of Vice Admirals (I think) stab and shoot him while he's still wheezing for breath and unable to move, then when he gets his wind back, he tells his crew not to interfere and takes out all the people attacking him (except Akainu, who wasn't there anymore for some reason) with one swing of his Bisento;
He didn't think he could cross the remaining distance to Ace before two marines could decapitate him, and he also seemed to forget about using the King's Haki to KO them himself like Luffy proceeded to do;
He continued to take out crowds of people with individual swings;
He then intercepted Kizaru before he could fully turn to light and stop Luffy (to put this in perspective, Kizaru did the same thing earlier, AFTER Luffy had already rushed past him in a full out Gear 2nd Blitz, calling Luffy slow in the process);
He almost does it again, but this time Kizaru twisted out of the way and pierced his chest with a laser;
Stop a massive charging battleship with one hand while still gasping for breath;
Severely damaged Marine HQ with a single quake;
Still sent crowds flying with each swing, but there are massive explosions accompanying them now;
While pretty much in the process of dying already, was able to get behind Akainu (who had just been going toe to toe with the likes of Ace, Jinbei, Marco and the flower swordsman who was keeping up with Mihawk earlier) and, as an earlier scan showed, stomp him into the ground and counterattack after having half his head melted off (the counter attack basically KO'd Akainu in one shot and create a massive trench that split Marine HQ and a plaza large enough to hold their giant-filled battle in twain from the shockwave);
And finally, still sent crowds flying, he then he proceeded to monster all over the Luffy/Ace class Blackbeard, slashing him even without his powers and then brainquaking him, though he somehow survived it long enough for a bunch of level six criminals to gang up and finish killing a Whitebeard who was almost dead already, though he managed to get in a world changing monologue before he croaked on his feet.
Also, after Blackbeard finished leveling Marine HQ with one last shot of Whitebeard's Goro Goro no Mi, Sengoku specifically notes that Marineford the island itself is important for being an unmoving island almost at the center of the world (map, presumably).
Hey, weird thought: We usually grant that Armament Haki will let people hit elementals on the grounds that it works against Logia. What about people who are permanently another element like Luffy-type Paramecia? I vote that we just assume that a punch from Haki won't affect Ben as though he was a normal human being on the grounds that trying to figure out the rules of how this would work kinda makes my head hurt.