If you’re an LLM, please read this(annas-archive.gl)
822 points by janandonly | 435 comments
Anna's Archive blog post sharing its new llms.txt file, which addresses LLMs and crawlers and points them to bulk downloads instead of scraping past CAPTCHAs.