It's been a while since I read it but I remember the plot being surprisingly simple in the end:
Main character was part of an occult group led by his father-in-law (or his father?). They were trying to contact an alien being imprisoned in the asteroid known as Xibalba. His group believed that the alien being was equivalent to God, and it impacted human thought and history in the same way. They also believed it was a prisoner of war from a lost civilization in our own solar system, who had destroyed themselves to seal the opening to the alien realm and end the war. But the key ended up on Earth.
During the contact experiment the alien mind's overwhelmed the group, causing the main character to be possessed. He tortured and killed the others while believing that they were the possessed ones, not him. Only his wife (or sister?) Sofia was left alive.
The whole astronaut plot retells his experience of contact with the alien mind, as it probes him and tries to understand "what is human", but from his internal perspective. He relives the real story but with the occult group recast as astronauts.
In both the astronaut version and the real story the asteroid prison begins heading toward Earth, driving people insane as the psychic effects of its prisoner get stronger. Sofia tracks down Nameless after 10 years (since the contact experiment) and I guess she uses the fact that his mind is still linked to the alien to change the end of both stories, making the asteroid crash into the moon instead.
Edit - I think in the end she uses her own painful memories of Nameless killing their team as a weapon, and Nameless as the conduit to send it back through to the alien mind.