End January Update
I’m flying off in 3 days and I’m antsy, so I’m posting this blogpost a little early since I’ll be out the whole day tomorrow anyway. Sunday at SCAPE: Ground...
fun with robots.txt
discourse.32bit.cafeI’m thinking of using robots.txt to exclude all commercial search engines, SEO crawlers, AI/ML scrapers, etc. Because there’s nothing wrong with Google, OpenAI, etc. that can’t be fixed with the sort of antitrust enforcement not seen since AT&T was broken up into the Baby Bells. I know that honoring robots.txt is strictly voluntary, but I at least want to signal that I do not consent to having my work crawled by commercial robots. I do, however, want to explicitly include useful bots or bots u...