5 stories tagged web-crawling, newest first.
81 points by panoramas4good | 16 comments
Signals Corps blog post analyzing HTTP headers from 217 million crawled web responses for text aimed at AI agents, including prompt injection attempts.
328 points by chmaynard | 343 comments
LWN article by Jonathan Corbet updating on AI scraper bot traffic overwhelming websites and the role of residential proxy networks.
462 points by jeffpalmer | 179 comments
Cloudflare changelog post announcing a Browser Rendering crawl endpoint that crawls entire websites with a single API call.
578 points by wilsonzlin | 97 comments
Wilson Lin's blog post describing how he built a web search engine from scratch in two months using 3 billion neural embeddings.
1750 points by walz | 278 comments
Riley Walz's project showcasing about 5 million unedited YouTube videos with default IMG_XXXX filenames uploaded from iPhones between 2009 and 2012, shown in random order.