In some denominations, the minister would take the church records with him when he moved to another congregation. This is more likely to happen in frontier churches and in denominations that tended to keep less detailed records. Catholic priests tended to not do this, but there are exceptions.
As a result, the records may be in the last church the minister ministered at, the hands of a descendant of the actual minister, a local historical society or library that happened to obtain the records, or somewhere else.
Any of these places could be quite a distance from where the actual church was located.