If he brought back Samantha it's doubtful that she'd let Oliver be any sort of father to William, especially after Oliver has been outed as the Green Arrow. She didn't want William to even know that Oliver was his father.
And being a father changes a man (not that I know from personal experience) and Oliver felt this was a big change to his life and how it changed his life afterwards.