Ask HN: Weird archive.today behavior?
112 points by rabinovich | 39 comments
112 points by rabinovich | 39 comments
526 points by classichasclass | 392 comments
A Boston Diaries blog post complaining about the Thinkbot AI web crawler, which ignores robots.txt and asks to be blocked by IP across 74 addresses from 41 network blocks.
219 points by cnst | 153 comments
68 points by netdoll | 57 comments