I hate crawlers that don’t advertise they’re crawlers. ’n badly written. They don’t respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Can’t block them they’re distributed.
Any conversation
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid — from robots.txt to cloudflare turnstile — there has to be a better way!
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user can’t be tracked. So we’d have to give up privacy/security :/
I find myself building bespoke tooling for my own and workflows these days, not only because of the obvious reason of software being so much cheaper to build, but also because there are SO MANY options for solving any given problem that it feels impossible to evaluate all of them.
If it's not a crucial piece of tech for a work problem then it often feels like paving my own way is both cheaper mentally (kind of amazing that's true these days) and probably a better fit.
It will become increasingly harder to find niche markets for software when those people can just build it themselves exactly how they want it. In the end, people want to accomplish tasks, and the AI is going to be building the software on-the-fly exactly for that task, why do we need generic software that may or may not be a fit?
Yeah. It makes me think that protocols for exchanging data between apps will become even more important in that landscape. But maybe, not - maybe each exchange is another bespoke AI-driven implementation?
Protocols, infrastructure, containers, those kinds of things. I was actually playing around with an idea for a minimal markup language + runtime that can be utilized by AI to spin up apps more easily, and brainstorming it with ChatGPT, I have a feeling something like that will appear soon to complement MCP.
What’s the difference between a script-kiddy and a real ? Abstraction. You can program your whole life without learning how to meaningfully abstract problems into simpler pieces that TRULY make the problem easier to understand.
I’m not talking about design patterns or vs . These tools can help. But to really simplify something for others, you need an intellectual empathy that many simply never figure out.
I have something that usually builds, tests, and deploys under five minutes. It has been over two hours and forty eight minutes and it is still going on today... 🫣
setting up some of my workflows based on this writeup - ludwigabap.com/…/on-my-personal-…of-the-tools-i-use/
Particularly a big fan of zellij amongst others
Also venturing into having agents enable more of my work - let us see how that goes.