For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
replied to:I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid — from robots.txt to cloudflare turnstile — there has to be a better way!