7 stories tagged robots-txt, newest first.
14 points by caaqil | 7 comments
Stanford FSI news article on a study finding AI chatbots recommended the Japanese Communist Party to left-leaning voters in Japan's 2026 election due to source access.
526 points by classichasclass | 392 comments
A Boston Diaries blog post complaining about the Thinkbot AI web crawler, which ignores robots.txt and asks to be blocked by IP across 74 addresses from 41 network blocks.
1183 points by rrampage | 652 comments
Cloudflare blog post reporting that Perplexity uses stealth, undeclared crawlers to evade websites' robots.txt and no-crawl directives.
339 points by turkeytotal | 246 comments
404 Media news article reporting that Reddit now blocks search engines other than Google, following its AI data licensing deal with Google.
497 points by cdme | 423 comments
Blog post by Robb Knight showing that Perplexity AI fetches pages with a generic browser user agent, bypassing server-side blocks on AI crawlers.
319 points by schappim | 267 comments
OpenAI documentation page describing GPTBot, its web crawler, including its user agent string and how site owners can identify or block it.
507 points by Jeannen | 248 comments
A Tell HN post proposing websites add an ai.txt file describing the site for AI crawlers and training datasets, analogous to robots.txt, with comment discussion.