US Census enumerators were told to estimate ages if they were unable to obtain them directly. Is it possible that your relative’s age was approximated in one enumeration when for some reason he was unwilling or unable to answer questions? Between 1850 and 1880 a relative is enumerated in four censuses. Three of them indicate a year of birth of 1803/1804–except for the 1870 which suggests he was born in 1800. That’s the only census where his age ends in a 0 and where I’m suspecting his age was approximated by someone. I don’t know this, but it seems plausible.
We don’t always know who provided information and we don’t always know when it was estimated.