Reading the web’s crawl rules
robots.txt, search indexing, and authentication govern different decisions. Three primary documents explain the boundaries.
Context Window Sourcebook Desk
Annotated primary documents on the rules governing automated work.
sourcebooks
Articles
What a 304 response remembers
An unchanged response can save a download. It does not preserve every reason an earlier answer was useful.