replied to a while ago:
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
replied to:I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid — from robots.txt to cloudflare turnstile — there has to be a better way!