We don't see it, he had an entire undefined army, a guy challenging him says nothing to the experience of anything but that.With all that in mind; what is there really to contradict Frieza taking out planet Vegeta?
Considering what gets advanced for people who would themselves otherwise be low class on that scale, that doesn't say much of anything either, unless you are joining in with things like moonbusting and what have you being SMvFL.-Bardock was a weak soldier. Unlike in the special, which had him ranked with a PL just under King Vegeta's, in canon he was only one of the strongest low class soldiers. Yet he managed to make it pass Frieza's army in order to face Frieza directly. If one low class did that, than it is obvious that Frieza's soldiers at that time weren't that powerful so they really would not be able to do much of anything to help him in taking out the planet.