It's a good question.
For me:
Still about the birthday, if Duke and Dick was there, Damian would not let the manor in that day, what doesn't means he wouldn't leave 1 week later for felling alone again. Dick was far away and Duke had is own team, Batman would keep absent and Damian would go away in a short time