While I'm unsure if something worse would take the Joker's place if Batman killed him, something just as bad probably would. Nature abhors a vacuum. As for why Batman doesn't kill, Superman put it best in Mark Waid and Alex Ross' Kingdom Come: "When you strip everything else away from Batman, you're left with someone who doesn't want to see anybody die." That's as good an explanation as any in-universe.