Heimdell doesn't leave his post in Asgard because he is the watcher of the rainbow bridge.
No human has ever SEEN him. He never left ot fight in the battles on Midgard .
They just assumed he was white because that is what white people do. They white washed him. Just like Jesus.
Boom.