Are you using a map when you search the census for your ancestor? If you don’t have appropriate, contemporary to your census problem year maps, you could easily be making mistakes or looking in places that are not quite right.
With indexes, manual searches of the census are not always necessary (but sometimes they are). Maps though, are not optional. You need to know where locations are and how they fit together. Even if you think you know the location, get a map. In fact, making assumptions about locations can create a few brick walls.