A site can ask a crawler to stay away from a page without making that page private. It can also want a crawler to fetch a page precisely so the crawler can see an instruction not to index it. Those two situations sound contradictory until the fetch, the search listing, and access to the content are treated as separate decisions.
The distinction matters for anyone building software that reads websites, and for publishers deciding what that software should be able to find. These documents describe the technical arrangements. They do not settle every question about permission to reuse material.
The request made by robots.txt
RFC 9309, published in September 2022, formalizes the Robots Exclusion Protocol first defined in 1994. A crawler identifies the rules matching its user-agent and applies them to paths. The document explicitly distinguishes these rules from access authorization.
That makes Disallow: /private/ a request to a cooperating crawler, not a lock on a directory. The same RFC’s security section warns that listing paths can make them discoverable. A publisher should not put confidential material behind an otherwise public URL and expect the crawl file to protect it.
The instruction a crawler must see
Google’s robots.txt guide explains a separate problem: a blocked URL can still appear in search when other pages link to it. Google may know that the address exists without fetching the page’s content.
The guide also explains why blocking crawling can frustrate a noindex instruction. A crawler generally has to fetch the page to see that instruction. For a public page that should disappear from Google’s index, making it unreadable to the crawler can therefore be the wrong move.
This is a practical distinction between a document and its entry in a catalogue. The publisher may need to let the catalogue inspect the document in order to update the entry. Neither action changes whether an ordinary visitor can open the URL.
The check made by the server
HTTP’s authentication framework describes challenges and credentials used for protected resources. Here the server participates in deciding whether to provide access. The arrangement is different from publishing a crawler preference and relying on a client to honor it.
For a team maintaining a website, the useful question is specific: should automated readers avoid fetching this path, should a search engine omit it, or should unauthorized visitors be unable to read it? A single “hide from bots” setting can conceal those different intentions. Check what the setting actually changes before relying on it.
Published . Corrections