I really love this notion. He's always hanging around, but only occasionally engaged.
A step further: he takes a hand from time to time. The JSA gets used to it. They get gobsmacked by somebody like The Royal Flush Gang. The Spectre's answer to their protest that he stood aside amounts to "Dude. They were robbing a bank. Not committing real evil."