Not all homestead applications in the United States were successful. Part of the process of completing the homestead application was to post a public notice that the claim was nearing completion–and often that notice was published in the newspaper.
Homestead claims that were completed generated a land patent which transferred title to the claimant. Those patents are indexed on the Bureau of Land Management website (https://glorecords.blm.gov/ ). Claims that were not completed did not generate a patent and consequently do not appear in that website. Incomplete claims are generally not indexed. The incomplete files are at the National Archives and can contain significant information on your ancestor. One needs the location of where the property was located to obtain the incomplete claim records.
If the incomplete claim got to the point of requiring a notice to be published, the newspaper notice would contain the name of the claimant and the legal description of the property. This would allow the researcher to obtain a copy of the incomplete file.