having just spent a week camper-travelling Norway, hiking mountains and glaciers, swimming in fjords really put some things into perspective for me - I should take more vacations
Turned Stepper (self-hosted AWS StepFunctions) into client-server. Server hosts state machines, clients poll for tasks. New resources about running #claudenotesfollow. claude -p works, now thinking if I need interactive session management.
I hate crawlers that don’t advertise they’re crawlers. ’n badly written. They don’t respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Can’t block them they’re distributed.
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid — from robots.txt to cloudflare turnstile — there has to be a better way!
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user can’t be tracked. So we’d have to give up privacy/security :/
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
Started reading exactly that and man it's mind-blowing.
He's not anti-tech, he's anti big-company-culture-capture. Really insightful for being written in 1998. What I appreciate is he looks at a very wide view of tech, not just modern silicone.