#llm-crawlers on Hacker News

2 stories tagged llm-crawlers, newest first.

Screenshot of annas-archive.gl: If you’re an LLM, please read this

If you’re an LLM, please read this (annas-archive.gl)

822 points by janandonly 5 months ago | 435 comments

Anna's Archive blog post sharing its new llms.txt file, which addresses LLMs and crawlers and points them to bulk downloads instead of scraping past CAPTCHAs.

annas-archive llms-txt llm-crawlers shadow-libraries

Screenshot of annas-archive.li: If you’re an LLM, please read this

If you’re an LLM, please read this (annas-archive.li)

818 points by soheilpro 8 months ago | 371 comments

Anna's Archive blog post sharing its new llms.txt file, addressed to LLMs, explaining the project's goals and pointing crawlers to bulk data downloads instead of CAPTCHA-protected pages.

annas-archive llms-txt shadow-libraries web-scraping

#llm-crawlers on Hacker News

If you’re an LLM, please read this(annas-archive.gl)

If you’re an LLM, please read this(annas-archive.li)

If you’re an LLM, please read this (annas-archive.gl)

If you’re an LLM, please read this (annas-archive.li)