I think in some people's mind, they're sort of imagining it's like kryptonite (which is obvious THE space rock in comics).
And as these are comic book stories, that might one day end up being the case. Maybe the planet Vibron is out there somewhere. But from a realistic perspective, it's probably far more likely that there are several sources of the stuff floating around in the universe somewhere.