Bot traffic from Yandex(radar.cloudflare.com)
561 points by walrus01 | 226 comments
Cloudflare Radar page showing bot traffic statistics for Yandex's autonomous system (AS13238) over the past seven days; only a verification challenge loaded.
#web-crawlers on Hacker News
8 stories tagged web-crawlers, newest first.
OpenAI agents tried to bruteforce a UN website's API fields(swarmcha.se)
64 points by intunderflow | 48 comments
Blog post analyzing how OpenAI agents scanned UNCTAD's statistics API about 16,500 times in 2026 using proxies, obfuscation, and Google's XSS game.
OpenAI bots meddled with multiple US Government agency sites(bbc.com)
98 points by Betelbuddy | 155 comments
BBC News article reporting that OpenAI bots interfered with multiple US government agency websites.
Ban me at the IP level if you don't like me(boston.conman.org)
526 points by classichasclass | 392 comments
A Boston Diaries blog post complaining about the Thinkbot AI web crawler, which ignores robots.txt and asks to be blocked by IP across 74 addresses from 41 network blocks.
Perplexity is using stealth, undeclared crawlers to evade no-crawl directives(blog.cloudflare.com)
1183 points by rrampage | 652 comments
Cloudflare blog post reporting that Perplexity uses stealth, undeclared crawlers to evade websites' robots.txt and no-crawl directives.
Faking a JPEG(ty-penguin.org.uk)
387 points by todsacerdoti | 90 comments
Blog post describing how the author extended Spigot, a Markov-chain gibberish site for abusive crawlers, to quickly generate fake JPEG images.
Cloudflare's new marketplace lets websites charge AI bots for scraping(techcrunch.com)
381 points by boristsr | 246 comments
TechCrunch article reporting that Cloudflare is launching a marketplace letting website owners charge AI companies' bots for scraping their content.
Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
507 points by Jeannen | 248 comments
A Tell HN post proposing websites add an ai.txt file describing the site for AI crawlers and training datasets, analogous to robots.txt, with comment discussion.