How the hell do people find this kind of stuff. Like, is someone sitting at a computer randomly looking at peoples’ robots.txt files for interesting things?

• Category: Web Stuff
• Tag: crawler, google, robots.txt, search, seo