I can't say I remember if the Federation claimed any area on the other side of the wormhole. The Bajorans certainly did given that they set up a colony which the Dominion wiped out before announcing they were claiming any rights.
Maybe space and Gamma Quadrant Law is different, but under most law empty land with no previous owner belongs to the first person to settle there. The Dominion would have a valid claim if they were there before the Federation, but they weren't. They were the ones who were entering territory that was already being settled by others. And there reaction was 'kill first, make demands and ask no questions at all.' You don't get to kill your new neighbor because he built a house on empty unused and unclaimed land year your house. If the land was unclaimed, it goes to the first person to settle there regardless of whether someone else wants it. It would be different if the Federation arrived 80 years later to find that the Dominion had naturally expanded into that area already. Then the Federation would have no claim and would have to listen to the Dominion. It would also help if the Dominion properly defined their boundaries so others would know where not to go, but claiming the entire Gamma Quadrant at that point was entirely false as other inhabited worlds near the wormhole didn't know what the Dominion was or knew it only as a rumor, which proves they were not sovereign in the area. They were aggressively expanding not as a natural course but in response to having a new neighbor.
The Federation had every right to contest the Dominion's claims to the area around the wormhole for these reasons, and at least on earth the Federation would win. 'We'd have conquered this area in 80 years' is no defense against 'well we're here now and we're here first.'
A similar situation occurred in TOS when the Gorn attacked a Federation colony which they considered an invading force in their territory, wiping it out. Kirk pursues the Gorn ship for revenge and eventually comes to realize the Gorn's point of view, which leads to negotiations. That simply was not possible with the Dominion because they could not tolerate any power other than their own or any sentient species which was not under their rule. Other nations like the Klingons and the Romulans respect powerful rivals and can coexist with other powers as equals. The Dominion is fundamentally incapable of that.