Most of Batman's villains aren't especially physically dangerous; that's why I never cared much about Bane before King's run. Breaking Batman physically is relatively easy in concept; breaking him mentally or emotionally is a lot harder.
How physically threatening is the Joker? Answer: not very. He's a threat because he's an unpredictable, unmoored psychopath, insanity being a pretty good pattern for most great Batman villains. Hence Arkham Asylum. If the level of physical threat was what made for good comics, they'd be nothing but fight scenes. How much of a threat is Luthor to Superman--as compared to Doomsday, for example? I think we can agree Doomsday is the greater physical threat but not way up there in terms of great Superman villains. (I'd rather read about Terra-Man.)
It's not Two-Face's look, it's not his obsession with twos, and it's definitely not his physical threat that make him a great villain.
Two-Face is interesting because of the moral quandary he presents to Batman. Before he was Two-Face, he was Batman's friend and ally and Batman's not willing to give up on curing him, especially psychologically. Also, his duality is a nice mirror for Batman's own duality.
That said it has been a long time since a really good Two-Face story in the comics. His arc in All-Star Batman was pretty good but I don't remember seeing him in a significant role since then.