I am curious what an actual solution to this problem could…

replied to some time ago:

I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid — from robots.txt to cloudflare turnstile — there has to be a better way!

500 chars / 20 lines max · use #hashtags, @mentions and more

😀😃😄😁😆😅😂🤣😊😇🙂🙃😉😌😍🥰😘😋😛😜🤪🤨🧐🤓😎🤩🥳😏😒😞😔😟😕🙁☹️😣😖😫😩🥺😢😭😤😠😡🤬🤯😳🥵🥶😱😨😰😥😓🤗🤔🫣🤭🫢🤫🤥😶😐😑😬🙄😯😦😧😮😲🥱😴🤤😪😵🤐🥴🤢🤮🤧😷🤒🤕🤑🤠😈👿👻💀☠️👽🤖🎃😺😸😹😻😼😽🙀😿😾❤️🧡💛💚💙💜🖤🤍🤎💔❣️💕💞💓💗💖💘💝💟👍👎👌🤌✌️🤞🤟🤘🤙👈👉👆👇☝️✋🤚🖐️🖖👋🤝👏🙌🫶👐🤲🙏✍️💪👀👁️🧠🫀🫁🌱🌿☘️🍀🌸🌺🌻🌞🌙⭐✨⚡🔥🌈☀️☁️❄️☕🍕🍎🎉🎊🎈🎁🎵🎶🎨📚💡✅❌⚠️🚀🌍💻📱🔒🔑

example.com or https://example.com
Regular links: [title](example.com) or [title](https://example.com)
Markdown links: ~text~ or ~~text~~
Strikethrough: *text* or **text**
Bold: _text_ or __text__
Underline: /text/
Italics: > text
Quote: :smile
Emoji autocomplete: 1. first 2. second 3. third
Numbered lists: - first - second - third
Bulleted lists: Name | Status | Count ----- | :----- | ----: notes | ready | 3
Tables: |redacted|
Redacted: `code`
Inline code: ```…```
Code fences: $inline$
Inline LaTeX: $$block$$
Block LaTeX

Which one? #poll First option Second option
PollsUse 2–8 unique options.: Which one? #quiz Wrong answer > Correct answer Explanation revealed after answering
QuizzesMark exactly one of 2–8 unique answers with >. Text after a blank line is revealed after answering.: Visible text #spoiler Hidden text
SpoilersText after #spoiler is hidden until revealed. Aliases: #tldr, #sensitive, #contentwarning, #cw, and #triggerwarning.: Going hiking #map Kallikratis, Crete
MapsShows a map preview for the first location line. Alias: #location.: #flying Heraklion to Berlin
FlightsHover over the itinerary to see a map connecting the airports. The next three words form the itinerary: airport, to (or → or ->), airport. Use single-word airport names or IATA/ICAO codes.: Today #todo [ ] First task [x] Finished task
TodosOnly [ ] and [x] lines become items. Click your items to toggle them.: Run this #exec ```js console.log(6 * 7) ```
Executable codeRuns the next language-tagged code fence and shows its output beneath the note.: Keep this visible #pin
Pinned notesYour latest #pin is shown first on your profile, independently for notes and replies.: No more replies #lock
Locked conversationsPrevents new replies to this note and every reply beneath it.: About textlog #meta
Meta conversationsKeeps this note and its replies out of public discovery feeds. Aliases: #tlog and #textlog.: Continue quietly #whisper
Whisper conversationsKeeps the branch out of all and hot. Participants, mentions, and tag followers can receive it in my feed. It remains public elsewhere.: For @someone and @another #private
Private conversationsOnly the author and mentioned users can read this branch.: Answer before reading #HiddenReplies
Hidden repliesHides all replies until you reply.

moreless

continued:

For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.

replied to:

It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user can’t be tracked. So we’d have to give up privacy/security :/

replied to:

But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.

replied to:

My solution was to add an env var to disable logs from anonymous. At least now will be able to read the logs.

join the communityorbrowse more notes

share with friends