A large population of Inhumans could work if we treated them like Pre-Meji Japan or Ming China, a society so esoteric that the outsiders find them weird and frightening in a way.
Too bad Marvel decided to greatly increase the population of Inhumans within the population of humans.