This is correct.
Moreover, if we're going to start demanding women have a certain level of low-testosterone to compete, we are going to find that some cis women with high testosterone, which exist, will be magically barred from competing. Caster Semenya is a real life example of this occurring. It's as unfair to ban her as it would be to ban Phelps for his ridiculously long limbs.
Trans women do not magically come to dominate sports. The reality of anti-androgens and estrogen's impact on the body is real, and bone density and muscle mass are actually lost in the process of transition.