Have recently consolidated my different chat apps (Signal, Slack, WhatsApp, Discord, Google Messages) into a single Matrix client + bridges - I'm 100% sold on Matrix.
I hate crawlers that donโt advertise theyโre crawlers. โn badly written. They donโt respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Canโt block them theyโre distributed.
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid โ from robots.txt to cloudflare turnstile โ there has to be a better way!
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user canโt be tracked. So weโd have to give up privacy/security :/
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
The frequent comments you read about protesters in France or Portugal versus the situation here in America are valid. At the same time, here in America, the police would absolutely roll in with machine guns. That's kind of a counterweight.
Same with the Steam Machine. They could have been awesome offerings for a lot of different use cases, but alas, being able to generate memes with grandma's pictures is of the utmost importance...
@kev๐งprofilefollowrunner. making games, music, software. getting into as much trouble as i can.wrote:
game dev with LLMs assisting on some of the engine legwork I learned to do over the years is the most fun programming I've had since I was 12 years old plucking away at the public library commodore 64
It seems this year is when big budget LLM players overtake independent academic mathematicians, in speed at least. The Jacobian conjecture, a lot of Erdos problems, now even the Navier Stokes millenium problem. Solved due to new model releases, all within this summer.
in math it can definitely prove statements that havent been proven before. I dont think thats the big thing though since it also writes sentences nobody has ever written before and in software it can write programs that havent been written before. Its just that the speed and scalability that has overtaken human mathematicians and programmers
Last weeks post about "if you're not using an LLM router, how do you know wtf is going on" inspired me to spin up a couple of OSS options this week and wow were they right.