He really did spend 4.5 billion years in there, it's just that: 1. from the Doctor's perspective it was a single run of a couple days (with no memory at all of the repeats, despite it happening countless times); 2. time flows at a much slower pace in the pocket universe that contains Gallifrey; and 3. the teleporter that kept reconstructing him maintained his age for him.
Let's also remember that it was a Time Lord torture device, essentially, and no one's better at time manipulation than them.
As it is, the ballpark age of 2,000 (give or take a couple centuries?) seems like a safe bet. Billions of years passed by for us the viewer (and for the unbreakable wall, of course), but not for the Doctor himself. The same way how the Eleventh Doctor spent all that time from ancient Rome to modern day in the Pandora's Box but it didn't actually count towards his age because time was frozen (from our perspective).