Glasses do have reflections and are noticeable. When you look through the glass it refracts the light so you can see through the wearer's face (at a certain angle). All of these things are weird about glasses--which us glasses wearers are fully aware of, living with them every day. We can't help but notice what glasses do. So when I see someone wearing glasses with no glass in them, that calls my attention to it and I start looking at that instead of what I should be looking at in the scene. Other people may have a different experience--but I was talking about myself.