The only time they were not was during Vision and the Scarlet Witch and HoM and HoM was so OOC. I also don't see this need to compete for his affection. He hasn't historically been a great father. His children (regardless of bio related or not) are all great on their own. Their relationships to each other could be more healthy than anything with him.