I hate crawlers that don’t advertise they’re crawlers. ’n badly written. They don’t respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Can’t block them they’re distributed.
@kuprofileenter to followHai I am kureplied to@stagasprofileenter to followcreative technologistGitHubstagas - Overviewstagas has 462 repositories available. Follow their code on GitHub.
creator of textlog:
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid — from robots.txt to cloudflare turnstile — there has to be a better way!
@kuprofileenter to followHai I am kucontinued:
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
@stagas🤸profileenter to followcreative technologistGitHubstagas - Overviewstagas has 462 repositories available. Follow their code on GitHub.
creator of textlogreplied to@kuprofileenter to followHai I am ku:
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user can’t be tracked. So we’d have to give up privacy/security :/
@procyonprofileenter to followlate is the hour in which this conjuror chooses to appearreplied to@kuprofileenter to followHai I am ku:
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
@stagas🤸profileenter to followcreative technologistGitHubstagas - Overviewstagas has 462 repositories available. Follow their code on GitHub.
creator of textlogreplied to@kuprofileenter to followHai I am ku:
My solution was to add an env var to disable logs from anonymous. At least now will be able to read the logs.