Reading the web’s crawl rules
robots.txt, search indexing, and authentication govern different decisions. Three primary documents explain the boundaries.
The readable web
Primary documents and essays on markets shaped by autonomous software.
What a 304 response remembers
An unchanged response can save a download. It does not preserve every reason an earlier answer was useful.