GeistHaus
log in · sign up

Preventing LLM Web Site Crawlers

matttproud.com

I was recently doing several searches on the public World Wide Web around some niche technical topics. The search results were straight disappointing. The search topics and terms would lead to very specific, canonical documents, ones that have so many inbound links that preferring any other documents from a ranking standpoint would be lunacy — in a sane and just world that is. One we don’t live in, though: so many of the highly-ranked links were ad farms that contained LLM-generated junk.

0 pages link to this URL

No pages have linked to this URL yet.