Simple, because people in upper income brackets traditionally vote Republican, but they in particular having been put off by his xenophobia, which showed a loss in his support with this voting group in 2018.
The political center is always the bigger loser during economic downturns, when the rot sets in, political radicalism prevails.