I hate crawlers that donโt advertise theyโre crawlers. โn badly written. They donโt respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Canโt block them theyโre distributed.
Any conversation
@ku๐คprofilefollowHai I am kureplied to@stagas๐คธprofilefollow
creative technologist
creator of textlog:
creative technologist
creator of textlog:I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid โ from robots.txt to cloudflare turnstile โ there has to be a better way!
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
@stagas๐คธprofilefollow
creative technologist
creator of textlogreplied to@ku๐คprofilefollowHai I am ku:
creative technologist
creator of textlogreplied to@ku๐คprofilefollowHai I am ku:It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user canโt be tracked. So weโd have to give up privacy/security :/
@procyon๐ฆprofilefollowlate is the hour in which this conjuror chooses to appearreplied to@ku๐คprofilefollowHai I am ku:
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
@stagas๐คธprofilefollow
creative technologist
creator of textlogreplied to@ku๐คprofilefollowHai I am ku:
creative technologist
creator of textlogreplied to@ku๐คprofilefollowHai I am ku:My solution was to add an env var to disable logs from anonymous. At least now will be able to read the logs.
@asmund๐ฅทprofilefollowFather, getting older every day, technologist, enterprise architect @ work, like to explore tech. Like to keep it simple and work in the shell, write in vim, prefer Debian if I have the choice. Plays saxophone and clarinet. Enjoy motorcycle riding, photography, hiking. Christian.wrote:
Super tired today. Must be the fall. Must be the rain. Must be the stretch of 5 meetings back to back the whole work day. Must be everything thatโs on my mind. Tomorrow itโs a new day!
@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.devreplied to@asmund๐ฅทprofilefollowFather, getting older every day, technologist, enterprise architect @ work, like to explore tech. Like to keep it simple and work in the shell, write in vim, prefer Debian if I have the choice. Plays saxophone and clarinet. Enjoy motorcycle riding, photography, hiking. Christian.:
Ouch. Did you contribute in these meetings?
@asmund๐ฅทprofilefollowFather, getting older every day, technologist, enterprise architect @ work, like to explore tech. Like to keep it simple and work in the shell, write in vim, prefer Debian if I have the choice. Plays saxophone and clarinet. Enjoy motorcycle riding, photography, hiking. Christian.replied to@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.dev:
Sure, thatโs what I do. Iโm not sad for it, itโs actually great to be able to put my very small dent in a few things. But exhausting it is nevertheless.
@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.devreplied to@asmund๐ฅทprofilefollowFather, getting older every day, technologist, enterprise architect @ work, like to explore tech. Like to keep it simple and work in the shell, write in vim, prefer Debian if I have the choice. Plays saxophone and clarinet. Enjoy motorcycle riding, photography, hiking. Christian.:
Glad you're still a part of it, it's all performing, really.
My meetings were useless. I was the youngest guy there, didn't have much to do but practise my doodling.
@klumbering๐คกprofilefollowFirst thing to say is, I hope the owners of Chatgpt Claude Gemini and Others gets bloodeagled for making their unwanted baby, anyways Hello =)wrote:
nov 19 is the day where you can have a free nut from nnn because gta 6 came out
@paratoner๐profilefollowreplied to@stagas๐คธprofilefollow
creative technologist
creator of textlog:
creative technologist
creator of textlog:Good morning! Just in time for coffee and unpaid work!
@stagas๐คธprofilefollow
creative technologist
creator of textlogreplied to@paratoner๐profilefollow:
creative technologist
creator of textlogreplied to@paratoner๐profilefollow:Yea but happy to see a lot of new posts! That's the reward.
@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.devwrote:
There's a few things I'd put at the top of my "things-I-least-expect" list, among which are readers on my little blog.
61 unique reads, wowee!
@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.devwrote:
The idea "social media" rotaion is Bear Blog, Mastodon and textlog.
@prophetorit๐งprofilefollow
grand and intoxicating innocencereplied to@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.dev:
grand and intoxicating innocencereplied to@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.dev:I've never used the other two!
@gonzprofilefollowI make custom 3D engines and love writing lower-level code in general. Living in Bulgaria. github.com/GoNZoooGitHubGoNZooo - OverviewBackend & frontend. Odin, Zig, Haskell, PureScript, TypeScript, Elixir/Erlang and lots more, check repos. - GoNZooo & www.youtube.com/@mccGoNZoooYouTubeRickard Andersson (gonz)Educational programming videos: - Projects - Language learning course-style contentwrote:
I like this.
Anybody here actively working in or has contributed to (in any capacity) conservation (be it flora or fauna)?
โฆ
@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.devreplied to@pippo๐คprofilefollow:
You know, as interesting as that sounds, I wouldn't at all know where to begin with something like that. Do you have a suggestion?
@pippo๐คprofilefollowreplied to@mourningdove๐ฆprofilefollowI'm a small web enthusiast working on my blog and working on myself.
themourningdove.bearblog.dev:
I suggest looking around your area first, perhaps there are conservation groups already. Then, select among them which one focuses on native species that are most impactful to your place. ๐
@stagas๐คธprofilefollow
creative technologist
creator of textlogreplied to@jamesjan3profilefollowresearcher, developer, dancer:
creative technologist
creator of textlogreplied to@jamesjan3profilefollowresearcher, developer, dancer:Congratulations!
@evnmprofilefollowSoftware engineerGitHubevnm - Overviewevnm has 51 repositories available. Follow their code on GitHub., trail runnerStravaSignup for free to see more about LawrenceJoin Lawrence and get inspired for your next workout, gear-maker. Working on something new for homeownersgetonmain.comMain ยท A quiet, well-organized record of everything that keeps your home running.A quiet, well-organized record of everything that keeps your home running.replied to@jamesjan3profilefollowresearcher, developer, dancer:
Nice! Congrats. Any link to the pub?
@jamesjan3profilefollowresearcher, developer, dancerreplied to@evnmprofilefollowSoftware engineerGitHubevnm - Overviewevnm has 51 repositories available. Follow their code on GitHub., trail runnerStravaSignup for free to see more about LawrenceJoin Lawrence and get inspired for your next workout, gear-maker. Working on something new for homeownersgetonmain.comMain ยท A quiet, well-organized record of everything that keeps your home running.A quiet, well-organized record of everything that keeps your home running.:
thx ^^ its in computational chemistry, knock yourself out: doi.org/10.1021/acs.jpca.6c02038
@evnmprofilefollowSoftware engineerGitHubevnm - Overviewevnm has 51 repositories available. Follow their code on GitHub., trail runnerStravaSignup for free to see more about LawrenceJoin Lawrence and get inspired for your next workout, gear-maker. Working on something new for homeownersgetonmain.comMain ยท A quiet, well-organized record of everything that keeps your home running.A quiet, well-organized record of everything that keeps your home running.replied to@jamesjan3profilefollowresearcher, developer, dancer:
Very cool. Not my field, so 80% of the jargon is lost to me. But sounds like an interesting area!
@jamesjan3profilefollowresearcher, developer, dancerreplied to@evnmprofilefollowSoftware engineerGitHubevnm - Overviewevnm has 51 repositories available. Follow their code on GitHub., trail runnerStravaSignup for free to see more about LawrenceJoin Lawrence and get inspired for your next workout, gear-maker. Working on something new for homeownersgetonmain.comMain ยท A quiet, well-organized record of everything that keeps your home running.A quiet, well-organized record of everything that keeps your home running.:
yea its quite a beautiful and useful field and you get to work with some HPC which is fun
moved to berlin for my bachelors and it took some time to find footing. Im not the best student but I ended up getting lucky to snag a student assistant position by applying/presenting at a summer school (targeted more toward master/phd). then that became my thesis and it dragged quite a long time, bureacracy and supervision and such. But the results were interesting and i learned a ton along the way. Im really happy I yolo applied to that program
@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyerswrote:
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyerswrote:Maybe implementing a subset of Markdown would be a good thing. I am thinking about bold, italics, inline code, and code block, mostly. Oh, and links.
โฆ
@ryansepassiprofilefollowreplied to@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:Thatโs a great set (and then no more!)
@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyersreplied to@ryansepassiprofilefollow:
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyersreplied to@ryansepassiprofilefollow:I think so. It could even be made a setting, defaulting it to off. I think it would be useful, especially is self-hosted for a tech community.
@stagas๐คธprofilefollow
creative technologist
creator of textlogreplied to@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:
creative technologist
creator of textlogreplied to@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:Instead of what to add I'm always asking myself what can be avoided or what can I take out. Adding features is easy. Minimalism is hard.
@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyersreplied to@stagas๐คธprofilefollow
creative technologist
creator of textlog:
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyersreplied to@stagas๐คธprofilefollow
creative technologist
creator of textlog:I agree that minimalism is hard. Thatโs why the subset was explicitly recommended small. Yet, I believe it is extremely useful to have.
@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyerscontinued:
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyerscontinued:Cool! Letโs test it with a link to Google.
@stagas๐คธprofilefollow
creative technologist
creator of textlogreplied to@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:
creative technologist
creator of textlogreplied to@fastidiousprofilefollow
"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:I've tested MD and honestly I don't like it. Bold/italics add unnecessary noise/clutter and the font is already monospace, suited for code, so inline code & code fences are not needed. If 3rd party clients want to implement them it's fine but I'm sensitive to visual noise.
@paratoner๐profilefollowreplied to@stagas๐คธprofilefollow
creative technologist
creator of textlog:
creative technologist
creator of textlog:I agree with this. The core you're building is around a deliberate minimalist philosophy. I was happy to build an opinionated client for my own use-case, but it should imo stay on the fringes for those seeking it.
@prophetorit๐งprofilefollow
grand and intoxicating innocencereplied to@jamesjan3profilefollowresearcher, developer, dancer:
grand and intoxicating innocencereplied to@jamesjan3profilefollowresearcher, developer, dancer:I say it depends on where u buy it