I think the Gaiman curated Sandman line is a good start for DC fantasy books, but there are lots of other possibilities.
I'd love to see something done with Starfire I...
featuring a strong female lead who is a warrior woman leading a rebellion against some kind of oppressor in a swords and science setting, maybe adding elements of planetary romance-a earth women cast into a strange world where she discovers all she is capable of.
Maybe do something with Arion (perhaps a new take on the Atlantis Chronicles) and/or Skartaris (though Warlord has never felt right without Mike Grell being on board in some fashion or other).
I'm not sure if DC retains the rights to Arak or whether Roy Thomas kept them when the series was done at DC, but a new take on the shamanic native American warrior exploring the dark age setting of Europe has a lot of strong storytelling possibilities.
A new Madame Xanadu series, more in line with the sensibilities of the Matt Wagner series from Vertigo than what they did with her in JL Dark.
Ideally, I'd like to see them do something along the lines of Ink and Zoom, OGN by some established fantasy writers using DC properties, with 2 lines, a young adult fantasy line and more adult fantasy line (a la Gaiman's curated line but not Sandman specific). Fantasy is probably a bigger seller in the YA market and in the broader book market (where Potter books and that ilk dominate the YA market and stuff like A Song of Fire and Ice, the Dresden Files, etc. draw in older readers) than in the direct market and it would be a chance to get new eyes of readers who may not specifically be comics readers on stuff in comic format that can create new customers and grow the overall market. There's a lot of untapped potential there.
-M