That's what the story said. I agree it doesn't make sense as I don't think 12 is enough to start a sizeable human colony, let alone a human-dominated space empire. But we're told they ended up establishing a stable presence in space, tried to be peaceful, got invaded a few times, then decided to be warmongers. Outside the starting population I can't say that it doesn't make sense.