I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site’s servers. More here.. For that reason, NPR has blocked Fediverse servers like Mastodon.
I have experienced this on my personal server as well (DDOS), but at the time didn’t know why, exactly.
This isn’t the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.
Heh. Ever heard of the Slashdot effect?
For the fifth fucking time, this is not the same thing .
I added a rule in my .htaccess for their user agents to direct all requests to a static page on my domain.
Same thing used to happen with reddit giving sites the hug of death lol
It’s unfortunate but kind of expected.
Its not at all the same thing
Why not have the users fetch the previews themselves. The user’s browser, that is.
Or get the preview on the first user seeing the post in that instance.
For sure there is more users than instances. So the amount of requests would be grater but at least they wouldn’t happen at the same time.
But if fediverse ever grow to be as big as other social media, then a single post/link can be viewed by thousands of users at the same time causing the same effect.
Sounds like every instance is doing the preview generation. Maybe it could grab the preview data it needs on the original instance, and pass that around with the post…
Well you can’t really trust the original instance to be truthful with its preview.
I’m not exactly sure why a web server can’t handle legit web traffic, just because it’s a lot? I agree that smaller actors should be given a little breastfeeding but solving it on the other side is bad architecture. A legit request can never be compared to a a ddos attack, the overloaded net has to throttle and then the instances need to retry with a generous enough delay.
Options for solutions to another systems problems are all going to introduce more problems. Sure, the requests could honestly be routed to a cache and it would help this problem, att least until it breaks or is exploited. But it would never be truly solved that way.
I’m not exactly sure why a web server can’t handle legit web traffic, just because it’s a lot?
Its not because it’s a lot, it’s because it all happens in the same instant, which does not resemble any sort of normal web traffic.
but solving it on the other side is bad architecture
Solving it on the side that’s creating the problem is bad architecture? 🤔
But it would never be truly solved that way.
Several potential solutions have been posited, if you just look through the comments.
Lots of normal traffic is instant. Throttling must be handled by for in this instance NPR or its providers routers because even if hammering is a bad practice, we are participating in why it happens at all if we just build structures to mitigate it.
Yes that’s why I included a solution, and explained explicitly that it can be solved but only temporarily
Lots of normal traffic is instant
Not instant from a thousand different servers. That’s what a botnet looks like. AKA DDOS. That’s why I explained that. Why are you acting like you don’t understand this?
Throttling must be handled by for in this instance NPR
No. It mustn’t. AP servers can fix it on their end.
Yes that’s why I included a solution
Which you then went on to explain wasn’t a solution…?
You may suffer from irritation of your ego my friend. Know that you are not your ideas or projects, you are more than that. I feel like all of your questions were previously answered sorry.
You may suffer from the dumbs, my friend. My questions were answered. Before you showed up.
it seems like ideally it shouldn’t attempt to preview the link before a user tries to view the content. i guess that would increase latency for users though
Sure would.
So we need people to just mass post links to bad websites, is what you’re saying.
i’m feeling less ambivalent about those reddit repost bots for a moment
So linking my own website here is a bad idea? Glad I haven’t done that yet.
Is this the same thing as the “Reddit hug of death”?
Similar, except those are caused by users visiting a link, whereas the Fediverse issue is federated servers gathering info before users even see a link.
How many servers are we talking about? 10,000 ?
Or being slashdotted.
Or the Digg Effect
Is there a reason why the publishing instance is not downloading the preview image, and then submits it with the post to the other instances? This way they would not have to fetch it themselves.
Yeah, thats how it should be: one download, everyone else downloads from that server, not the original source.
Seems abusable though
It is, but that’s what defederation is for: If there are instances that are being sketchy, they are supposed to get cut off from the rest of the Fediverse.
Seems abusable though
If one instance doesn’t trust another, they shouldn’t federate in the first place.
Seems abusable though
Exactly
Add metadata with a checksum of the og one with signature and let each client check that up.
If it’s a signed checksum then maybe. But if the og instance is doing the abusing then it won’t help.
Yeah there must be some trust indeed of the og instance. A scoring system could help with that.
Or … Or we could stagger the requests of the different instances so that they don’t overload the linked site. Why reinvent the wheel?
Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.
Either publishing server should serve the metadata - although it would be prone to bad actors, so a refetch in the next 0-180 seconds would be appropriate. Or just add a random 0-60 second delay before fetching the data.
I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It’s no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.
Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don’t honour robots.txt (or aren’t yet listed).
To be honest, if a website has no caching of their server-rendered parts, they are also doing something wrong! Very little sites really need to re-render a public page on every view.
Around 2010, I used to work at an MSP, and once had a customer that ran a custom LAMP stack website for local radio UK stations web presence. It was a bit slow, running on a dedicated physical server, but every time it snowed, the website would 500 out - because of too many requests to the “School Closures” page. The site had no caching other than Apache’s internal cache, but PHP pages could not be cached because “reasons”, and the original developer was “not available”. I optimised their MySQL indexes, increased the number of Apache workers, and it improved the render time per page and number of concurrent connections. Stress testing looked good.
Anyway, some months later, 2AM one snowy morning, I was on call, and it went down again. This was when I discovered the school closures problem. After diagnosing, and then liaising with the customer to discuss the options, I was told “just make it work”. I ended up modifying the PHP script to add some “on spec” caching that saved the HTML to a temp file in ramdisk, and only re-ran the query-and-render code if the cached copy was more than 5 minutes old. It was hacky, but it did the trick, and I was finished by about 4 AM. This hack was later applied at their request to a number of trouble-points in the source tree, as and when things broke.
We recommended adding a Varnish cache, and found a more generalised caching library but they wouldn’t/couldn’t authorise engineering work for this - they were running on a shoestring - and they already had the tech break-fix contract with us, and decided it was sufficient to just spot fix things as they broke rather than fix the root cause.
I left that MSP in 2012, so I don’t know what happened after that, but as far as I know, the site could still be running… unlikely, but possible.
that don’t honour robots.txt (or aren’t yet listed).
Throw’em in the tar pit.
Limited mental capacity these days, so I don’t have the spoons to properly research and implement a solution at the moment. I installed Nepenthes last year, but I’ve never managed to get round to integrating it into my site. It’s low priority at the moment.
that don’t honour robots.txt
That’s essentially all of them in my experience.
At least anything “AI”
It’s almost like the software that powers the internet was designed for a different time and is only still used because there’s monied interest in keeping it around. We need GNUnet.
Oh wow GNUNet I havent heard of that in a while. How is it nowadays?
Or i2p.
is there something that makes this different to 17 other proposals for a new internet or is it just a case of
Now this is interesting. Thanks for sharing
GNUnet helps building a new Internet
GNUnet is a network protocol stack for building secure, distributed, and privacy-preserving applications.
With strong roots in academic research, our goal is to replace the old insecure Internet protocol stack.
GNUnet provides privacy-by design
It provides [improving addressing, routing, naming and content distribution](https://www.w3.org/2014/strint/papers/65.pdf) in a technically robust manner - as opposed to ad-hoc designs in place today.
GNUnet provides privacy-by design
Money doesn’t want us to have privacy.
Also https:// itsfoss.com/news/mastodon-link-problem/
