textlog
@stagas🤸profilefollowcreative technologist creator of textlogwrote some time ago:
flat
I hate crawlers that don’t advertise they’re crawlers. ’n badly written. They don’t respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Can’t block them they’re distributed.

share with friends

@stagas wrote on textlog https://textlog.cc/post/1942
500 chars / 20 lines max · use #hashtags, @mentions and more
😀😃😄😁😆😅😂🤣😊😇🙂🙃😉😌😍🥰😘😋😛😜🤪🤨🧐🤓😎🤩🥳😏😒😞😔😟😕🙁☹️😣😖😫😩🥺😢😭😤😠😡🤬🤯😳🥵🥶😱😨😰😥😓🤗🤔🫣🤭🫢🤫🤥😶😐😑😬🙄😯😦😧😮😲🥱😴🤤😪😵🤐🥴🤢🤮🤧😷🤒🤕🤑🤠😈👿👻💀☠️👽🤖🎃😺😸😹😻😼😽🙀😿😾❤️🧡💛💚💙💜🖤🤍🤎💔❣️💕💞💓💗💖💘💝💟👍👎👌🤌✌️🤞🤟🤘🤙👈👉👆👇☝️✋🤚🖐️🖖👋🤝👏🙌🫶👐🤲🙏✍️💪👀👁️🧠🫀🫁🌱🌿☘️🍀🌸🌺🌻🌞🌙⭐✨⚡🔥🌈☀️☁️❄️☕🍕🍎🎉🎊🎈🎁🎵🎶🎨📚💡✅❌⚠️🚀🌍💻📱🔒🔑
example.com or
https://example.com
Regular links
[title](example.com) or
[title](https://example.com)
Markdown links
~text~ or ~~text~~
Strikethrough
*text* or **text**
Bold
_text_ or __text__
Underline
/text/
Italics
> text
Quote
:smile
Emoji autocomplete
1. first
2. second
3. third
Numbered lists
- first
- second
- third
Bulleted lists
Name  | Status | Count
----- | :----- | ----:
notes | ready  |     3
Tables
|redacted|
Redacted
`code`
Inline code
```…```
Code fences
$inline$
Inline LaTeX
$$block$$
Block LaTeX
Which one? #poll
First option
Second option
PollsUse 2–8 unique options.
Which one? #quiz
Wrong answer
> Correct answer

Explanation revealed after answering
QuizzesMark exactly one of 2–8 unique answers with >. Text after a blank line is revealed after answering.
Visible text #spoiler
Hidden text
SpoilersText after #spoiler is hidden until revealed. Aliases: #tldr, #sensitive, #contentwarning, #cw, and #triggerwarning.
Going hiking #map
Kallikratis, Crete
MapsShows a map preview for the first location line. Alias: #location.
#flying Heraklion to Berlin
FlightsHover over the itinerary to see a map connecting the airports. The next three words form the itinerary: airport, to (or → or ->), airport. Use single-word airport names or IATA/ICAO codes.
Today #todo
[ ] First task
[x] Finished task
TodosOnly [ ] and [x] lines become items. Click your items to toggle them.
Run this #exec
```js
console.log(6 * 7)
```
Executable codeRuns the next language-tagged code fence and shows its output beneath the note.
Keep this visible #pin
Pinned notesYour latest #pin is shown first on your profile, independently for notes and replies.
No more replies #lock
Locked conversationsPrevents new replies to this note and every reply beneath it.
About textlog #meta
Meta conversationsKeeps this note and its replies out of public discovery feeds. Aliases: #tlog and #textlog.
Continue quietly #whisper
Whisper conversationsKeeps the branch out of all and hot. Participants, mentions, and tag followers can receive it in my feed. It remains public elsewhere.
For @someone and @another #private
Private conversationsOnly the author and mentioned users can read this branch.
Answer before reading #HiddenReplies
Hidden repliesHides all replies until you reply.
@ku🤔profilefollowHai I am kucontinued:
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
@stagas🤸profilefollowcreative technologist creator of textlogreplied to@ku🤔profilefollowHai I am ku:
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user can’t be tracked. So we’d have to give up privacy/security :/
@procyonprofilefollowlate is the hour in which this conjuror chooses to appearreplied to@ku🤔profilefollowHai I am ku:
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
join the communityorbrowse more notes