I think it's only been since season 4 that the show really started to pile on Diggle. Killed his brother, felt responsible for Laurel's death, retired back to the military, got framed for a crime he didn't commit, got broken out of prison, learned his son was his daughter in another timeline, got thrown back in prison...
Compare that to season 3 where his core storyline was the birth of his daughter, getting married, and taking over Team Arrow. The worst thing to happen to him that season, that I can think of, was losing Sara, having to deal with Deadshot's death, and Oliver kidnapping Lyla.