They backpedaled because they were afraid Ultron would kill them too. And them attacking the Avengers, who had nothing to do with their parents' deaths show they have no problem endangering innocents' when it suits their goals. No one acts like this was not the plan at all. Ultron even says to Tony, "You'll have to catch Dr. Banner".