Observation:

- People started deploying anti-scraping measures to fight LLM scraping
- Web indexers can't index stuff anymore
- Search results are even worse than before
- The only way to retrieve the information is to use models that were trained in pre-anti-scraping times (or beat anti-scraping)

If I'm right, anti-scraping can actually push people towards LLM's (who currently absolutely have the capacity to circumvent most anti-scraping).

If you think you share knowledge worth finding, please consider this before deploying countermeasures!

#scraping #search #llm