14 stories tagged ai-crawlers, newest first.
313 points by sammy0910 | 170 comments
Blog post by the developer of sprinklz.io describing how his site slowed after an HN post due to heavy AI bot and scraper traffic, found via NGINX logs.
245 points by ingve | 222 comments
A blog post on Musing Mortoray questioning why AI companies are not prosecuted for hacking, presumably over aggressive data scraping practices.
1282 points by zdw | 636 comments
Blog post by Konstantin Ryabitsev with numbers showing AI crawlers consume more git.kernel.org CPU rendering commits than all legitimate access combined.
401 points by petercooper | 367 comments
A site owner's account of a year spent fighting scraper bots on a 1.5 million-page website; the capture shows only a Cloudflare verification page.
365 points by tosh | 126 comments
Cloudflare Radar's AI Insights dashboard showing traffic trends from AI bots and crawlers, filterable by industry and crawl purpose.
728 points by taviso | 772 comments
Tavis Ormandy's essay examining Anubis, a proof-of-work anti-AI-crawler gate, covering the problem it targets, cost numbers, alternatives, and workarounds.
369 points by stephendause | 257 comments
New York Times article on Cloudflare blocking AI data scrapers by default; the captured page shows only a CAPTCHA verification challenge.
437 points by scotchmi_st | 230 comments
Cloudflare blog post announcing pay per crawl, a feature letting content owners charge AI crawlers for access to their sites.
329 points by LinuxBender | 232 comments
Ars Technica article reporting that open source developers face overwhelming AI crawler traffic, leading some to block entire countries to keep FOSS sites online.
803 points by todsacerdoti | 487 comments
LibreNews article by Niccolò Venerandi on how LLM scrapers from AI companies are overloading and taking down FOSS projects' infrastructure.
532 points by OptionOfT | 207 comments
Xe Iaso's blog note describing how Amazon's AI crawler is overloading their self-hosted git server and an nginx ingress snippet used to block it.
644 points by blendergeek | 244 comments
Project page for Nepenthes, a tarpit that traps LLM web crawlers in endless generated pages of links and optional Markov-babble text.
566 points by ta988 | 335 comments
A diaspora* post by Dennis Schubert reporting that LLM training crawlers like GPTBot, Amazonbot and ClaudeBot account for about 70% of the project's web infrastructure traffic.
497 points by cdme | 423 comments
Blog post by Robb Knight showing that Perplexity AI fetches pages with a generic browser user agent, bypassing server-side blocks on AI crawlers.