Is the fediverse about to get Fryed? (Or, “Why every toot is also a potential denial of service attack”)
ar.al/2022/11/09/is-the-fedive…
CC @stephenfry @gretathunberg
#fediverse #mastodon #stephenFry #gretaThunberg #smallTech #smallWeb
Is the fediverse about to get Fryed? (Or, “Why every toot is also a potential denial of service attack”)
Every time I post something that gets lots of engagement, I essentially end up carrying out a denial of service attack on myself. What does this say about the design of Mastodon and ActivityPub and the future of the fediverse?Aral Balkan
Katherine Cox-Buday
in reply to Aral Balkan • • •If I understand correctly, this entire post is predicated on the assumption that folks are running github.com/mastodon/mastodon. I wonder if this would be the case with a more efficient implementation of ActivityPub, e.g. github.com/superseriousbusines…
Which is to say, the implementation might be the limiting factor, not ActivityPub itself.
GitHub - mastodon/mastodon: Your self-hosted, globally interconnected microblogging community
GitHubAral Balkan
in reply to Katherine Cox-Buday • • •Katherine Cox-Buday
in reply to Aral Balkan • • •Aral Balkan
in reply to Katherine Cox-Buday • • •Simon Lucy
in reply to Aral Balkan • • •@katco
It's a plain truism in performance engineering that if you free up one bottleneck, say the overhead on the queue itself, then the bottleneck will move and it may well mean that waiting on read or write locks becomes mutual deadlocks.
Or more prosaically running out of connections, handles to a dependent service. All those are resolvable with effort.
But it could be considerable effort.
Jesse
in reply to Aral Balkan • • •Ricardo
in reply to Aral Balkan • • •JamesAlderman
in reply to Aral Balkan • • •Marc Ochsner
in reply to Aral Balkan • • •Paul Wilde :blobcatnim_new: :dontpanic_nobg:
in reply to Aral Balkan • • •great post @aral , let's hope it gets spread around the fediverse so those whose popularity can cause problems (it's weird saying that!) are aware of what their own success can do! (and what they can do to help out!) 😁
@stephenfry @gretathunberg
Aral Balkan
in reply to Paul Wilde :blobcatnim_new: :dontpanic_nobg: • • •@paul Thanks, Paul. Hopefully, it will also get all of us thinking about what we can do to combat the centralisation tendencies present in the ActivityPub protocol and Mastodon server designs. (As well as considering what a web where we all own and control our own places might be like.)
@gretathunberg @stephenfry
Trolli Schmittlauch 🦥
in reply to Aral Balkan • • •Babu Menos 🎵
in reply to Aral Balkan • • •Aral Balkan reshared this.
Aral Balkan
in reply to Babu Menos 🎵 • • •Babu Menos 🎵
in reply to Aral Balkan • • •Karin Sorenby
in reply to Babu Menos 🎵 • • •Aral Balkan
in reply to Karin Sorenby • • •@kcsorenby Thanks, Karin :)
@babumenos
Adam
in reply to Aral Balkan • • •Aral Balkan
in reply to Aral Balkan • • •Me, keeping an eye on my task queue as a conversation starts on my latest post 👀
#sidekiq #mastodon #scale #activityPub
Mindiell
in reply to Aral Balkan • • •Aral Balkan
in reply to Mindiell • • •Aral Balkan
in reply to Aral Balkan • • •I should probably stop replying to people now, shouldn’t I?
But I won’t!
I WILL NOT LIVE IN FEAR!
brunoc
in reply to Aral Balkan • • •chrismckee
in reply to Aral Balkan • • •I may need to rtfm
Alejandro AR
in reply to Aral Balkan • • •teledyn 𓂀
in reply to Aral Balkan • • •A long time ago, before even Geocities, I had a blog that had enough traffic to get noticed, and I was approached by a researcher at the University of Georgia offering me free hosting so long as uptime wasn't critical.
The site was called Dragonfire. The experiment was squeezing as much power out of a commodity white box as technically possible. It was overclocked to the hilt, water cooled, and the name was chosen after some coffee spilled on the case and started boiling.
Clemens aka data
in reply to Aral Balkan • • •Our instance is not that big, but seems to be integrated in the network, we hit 500k 1.5 weeks ago, steering towards 1M events/day.
It's really interesting when you're hosting the hardware yourself (especially as a hobby project) and can't easily scale up CPUs, but switching to nvme really payed off. We currently run 2 sidekiq processes at 10 treads each, with basically zero queue backlog.
stux⚡
in reply to Aral Balkan • • •Uncle Patrick
in reply to Aral Balkan • • •Marvin
in reply to Aral Balkan • • •Aral Balkan
in reply to Marvin • • •@ClaudetteK Alas, I have failed :)
@Saket
Marvin
in reply to Aral Balkan • • •Mr.Generation
in reply to Aral Balkan • • •cupofjoakim 🐸
in reply to Aral Balkan • • •Joeri
in reply to Aral Balkan • • •nixponey
in reply to Aral Balkan • • •Cedric
in reply to Aral Balkan • • •for $50 a month, I imagine I would also check the state of the queue.
It remembers me when I was using Heroku. A tool I wrote, hosted there, started to attract more users. It was a bit stressful.
just read the instructions
in reply to Aral Balkan • • •Aral Balkan
in reply to just read the instructions • • •@dgold Thank you ;)
🌏😵💫
Jack Deeth
in reply to Aral Balkan • • •Aral Balkan
in reply to Jack Deeth • • •@JackDeeth The Small Web stuff I’m working on is basically optimised for 1.
(The reason being that there’s a huge amount of complexity that is added when you move from 1 to 2. Or, conversely, a lot you can simplify in the experience if you’re optimising for just one person who is the owner of the server.)
But yes, it would be very interesting to see what design properties a solution for small instances would have.
anthony symkowick
in reply to Aral Balkan • • •@JackDeeth I think this toot is the best portrayal of the need for a small-web alternative to the corporate insulation/advertisement model. Interaction on the web has a fundamental cost.
I'm with you on the single-tenant solution being the better one, so how can we build a one-click product for nontechnical users to easily own, maintain, and bear the cost burden of their own instances?
Though I'm not sure a really efficient algorithm can solve the high cost for Stephen Fry's instance
Alasdair Allan
in reply to Aral Balkan • • •An escape pod was jettisoned during the fighting - Raspberry Pi
Alasdair Allan (Raspberry Pi)Aral Balkan reshared this.
Olivier Forget
in reply to Aral Balkan • • •I was wondering about real "big-world" scalability issues here. Thanks for posting this clear illustration of the problem. Yikes.
I would have thought AP would also include some gossip-style ability to spread posts around without the one instance being responsible for pinging ALL the followers. Guess not?
PS: apologies for the x Sidekiq jobs I have just triggered with this reply.
MrFroggyG
in reply to Aral Balkan • • •Am I understanding correctly that if every one of your 22k followers ran their own instance, it would seriously up the processing power required of your instance?
It feels like there’s an optimal number of followers above which someone should spin up their own instance, but otherwise, you might be straining the system more?
©
in reply to Aral Balkan • • •Adam Dalliance
in reply to Aral Balkan • • •So when I do a post, my server queues up jobs to tell every subscriber.
Suppose that makes sense. They all gotta hear about it. But it feels pretty inefficient.
Seems like it should be possible to do something like the Bittorrent protocol and fire off a dozen jobs to a selection of servers asking them to pass the message onwards.
Aral Balkan
in reply to Adam Dalliance • • •@pre (And every reply.)
I mean, of course, everthing is always a work in progress and hindsight is always 20/20. I just hope we can acknowledge some of the core design decisions that we take for granted that we’ve actually inherited from Big Tech and see how we can go forward differently. (Or at least apply social pressure where protocols and server designs might incentivise centralisation.)
rick
in reply to Aral Balkan • • •Aral Balkan
in reply to rick • • •nicolas ⁂
in reply to Aral Balkan • • •eobet
in reply to Aral Balkan • • •Jesper Mårtenson
in reply to Aral Balkan • • •I’m happy that my instinct to set up my own instance this time (tapped very little in on mastodon in 2019) instead of joining an established one, seems to be the right one.
Very insightful post, thanks for that.
Aral Balkan
in reply to Jesper Mårtenson • • •Liz Ellis She/Her🌳
in reply to Aral Balkan • • •great article and really useful. I’m moving to a dinkier server.
PS no need to reply!
Aral Balkan
in reply to Liz Ellis She/Her🌳 • • •@LizEllisPhD Haha, thanks + glad to hear it.
PS. I suck at not replying :)
🐧 Jonathan Treffler 🇺🇦🇵🇸
in reply to Aral Balkan • • •I think a potential long-term solution is extremely easy self hosting for everybody. Basically plug an ethernet cable into a router sized box and done.
There are software projects that make selfhosting pretty easy already, but not easy enough and off-the-shelf hardware with it preinstalled is just not there yet.
(1/x)
Matteo Carandini
in reply to Aral Balkan • • •Jim Jones
in reply to Aral Balkan • • •At the risk of causing these problems by replying to a popular person with an engaging post, I have a couple of questions...
1. Is there a possibility that code solutions exist to optimize against this behavior that arises from popular users?
2. Would setting replies to unlisted help resolve some of this storm?
3. If (2), could a configuration change to Mastodon instances that made replies unlisted by default (but changeable on a per post basis) to public help resolve the issue?
Emma 🇪🇺🇳🇱🇺🇦
in reply to Aral Balkan • • •I'm quite curious about cost calculations for various platforms.
JamesAlderman
in reply to Aral Balkan • • •Kat MCP(NT4) MCSE(Win2K)
in reply to Aral Balkan • • •husku
in reply to Aral Balkan • • •Jonas N
in reply to Aral Balkan • • •Dreeg Ocedam
in reply to Aral Balkan • • •Joe Wood
in reply to Aral Balkan • • •Alx 🐈
in reply to Aral Balkan • • •I still couldn't get in depth to your 'Small Web' concept, but I'll do it ASAP. ~
And thank you also for being so polite to answer individually to every message :)
Aral Balkan
in reply to Alx 🐈 • • •@alx Thanks, Alessandra, that’s very kind of you to say. And, hey, we’re all newbies at all this to one degree or other. Here’s to figuring things out and making things better together :)
PS. If there’s anything that doesn’t make sense about the Small Web stuff I’ve written, etc., please just ask. I’m always trying to improve/simplify how I explain things and it’ll help me to know which bits are confusing.
Seth
in reply to Aral Balkan • • •Borghal
in reply to Aral Balkan • • •Ross Young 🐺
in reply to Aral Balkan • • •Duncan Hart 🇮🇲 🇦🇺 🌏
in reply to Aral Balkan • • •JacksonBates
in reply to Aral Balkan • • •LB 👆: Does anyone know if all ActivityPub implementations (Pleroma, Misskey and all the others) suffer from the scaling challenges Mastodon does?
If Aral (see the post I'm replying to) was running, say, microblog.pub, would he have the same issues?
Joel Webber
in reply to Aral Balkan • • •This is a really helpful framing for the problem, and seems like something we're going to have to grapple with sooner rather than later. A few questions spring to mind:
- What is the current cost of maintaining an identity, as a function of followers, activity, media, etc?
- How much can that be optimized in the code? Through protocol improvements?
- Assuming the optimized cost is still non-trivial, are there significant advantages to shared hosting?
Hosker 🇬🇧
in reply to Aral Balkan • • •❣️ the energy.
If everyone has their own instance/server wouldn't this example require 23k Sidekiq jobs instead of 3k?
Isn't that LESS scalable, for everyone.
You highlight a #fediverse problem to be solved but not sure everyone going DIY is the answer. (If this is the case.)
Silmathoron ⁂
in reply to Aral Balkan • • •See e.g. infosec.exchange/@WPalant/1093… for instance
Jannis Göing
in reply to Aral Balkan • • •Blackett Music
in reply to Aral Balkan • • •Sam Nicholl
in reply to Aral Balkan • • •Dan
in reply to Aral Balkan • • •Jason Green
in reply to Aral Balkan • • •There’s some tension between the points you make and the notion of the small instance hosting a community of exercise enthusiasts or esperantistoj.
As both you and @profcarroll point out, a well connected small instance can be quite resource intensive. A poorly connected instance is a virtual ghost town. Could some of this be solved by using lighter weight ActivityPub software? (Misskey or Pleroma)
Mika
in reply to Aral Balkan • • •In your post you talk about your server (which I assume is some sort of virtual private server) having 12 sidekiq threads and can deliver an ActivityPub update to 12 other servers at a time.
Do you have any idea how much performance the virtual server provided by mastohost has in values that allow a comparison to other VPSes? As in, what is "a sidekiq thread" for the server running it, do you get one sidekiq thread per CPU? Is CPU speed of the server, the web connection, or speed of the recieving server (e.g. if it needs to acknowledge recieving the ActivityPub push) the limiting factor for the speed at which such a sidekiq job executes? Maybe @hugo can provide some info on this?
Shelby Kinney-Lang
in reply to Aral Balkan • • •Jokke
in reply to Aral Balkan • • •Craig Grannell
in reply to Aral Balkan • • •I had to laugh a bit at “Do you know what happens to your email if Google says (rightly or wrongly) that you’re spam? No one sees your email.”
Yeah. That’s me these days. Hence moving press folks over to my own domain (but a different address from personal email).
Dane
in reply to Aral Balkan • • •Anarchic Teapot
in reply to Aral Balkan • • •hildabun
in reply to Aral Balkan • • •I'm very new to mastodon, and still trying to learn the basics of how it is structured. Your article has been very informative and, though coming from a very different and more-practical perspective, increasingly seems to raise concerns along the same lines as I've had as I've been reading about this.
Specifically: that this thing seems to be too centralised (lots of little centralised servers, but still centralised), and (if I'm reading correctly) only a single level of federation?
Paul Stone
in reply to Aral Balkan • • •GunChleoc
in reply to Aral Balkan • • •@neilhimself FYI ⬆️
@aral @stephenfry @gretathunberg
Roomey
in reply to Aral Balkan • • •Could I ask you a few naive questions, as I am not up to speed on the fediverse architecture.
It sounds like the problem of exponential job growth are technically solvable in code (rather than inherent in the system), is that correct?
Can the jobs not get gracefully queued and balanced, with the result popular people are just processed more slowly?
Can the jobs be batched so newer jobs can do batch updates and allow older jobs to be dropped?
Thank you
Dylan Sale
in reply to Aral Balkan • • •This is concerning. It seems to imply that decentralization is impossible long term for most people.
I see this going in the direction of celebrities creating Mastodon instances and starting to post ads to users on their instance, and possibly charge users for the benefit of getting priority access to their posts. (You can see them before the sync jobs run for other instances).
I guess if you want freedom from that you just need to live with a worse experience, at least you have the choice.
Hartley Wintney 🦌
in reply to Aral Balkan • • •kcarruthers
in reply to Aral Balkan • • •Brett Lempereur
in reply to Aral Balkan • • •Thanks, genuinely interesting and thought provoking. I’m barely known anywhere and my first instinct was to setup my own instance, if only to have some control over my presence.
This does present other challenges you don’t cover - natural follows from things like being in a local timeline just aren’t there. There really is a strong incentive for providers and users for centralisation.
Short of a fully P2P (ala Chord/Pastry) system, genuinely stumped on technical solutions here.
andy
in reply to Aral Balkan • • •FiXato
in reply to andy • • •Aral Balkan
in reply to FiXato • • •robin
in reply to Aral Balkan • • •Ricky Onsman
in reply to Aral Balkan • • •Stephen Baker
in reply to Aral Balkan • • •Keith J Grant
in reply to Aral Balkan • • •I think you're on to something, but the concept of single-user instances is taking too far. Your instance does a lot of work pulling in your federated timeline, but that work only benefits one user: you.
If you have a 300-user instance, but those 300 people follow a lot of the same people, the load-per-user is significantly lower.
There's a happy medium
Maarten Steenhagen
in reply to Aral Balkan • • •Michael
in reply to Aral Balkan • • •This is a great essay, and articulates extremely well a number of concerns that I have had while reading other descriptions of how Mastodon is different or better because of its architecture.
I have been feeling like the kid in the Emperor's New Clothes, looking around me and going "can no-one else see what I'm seeing???"
Thanks for saving my sanity!
Jim Pembroke
in reply to Aral Balkan • • •William Gunn
in reply to Aral Balkan • • •Allus Miserable
in reply to Aral Balkan • • •That was a really interesting and informative read. I'm learning more everyday. Thanks!
DON'T REPLY! 😂
peter kleiweg 🧩
in reply to Aral Balkan • • •jezza
in reply to Aral Balkan • • •Tom Resing
in reply to Aral Balkan • • •Kermode
in reply to Aral Balkan • • •Maybe a new fedi option would be to filter out ppl with over x number of follows?
I don't read Local or Global.
Chapps
in reply to Aral Balkan • • •TJ Radcliffe
in reply to Aral Balkan • • •I've just been arguing with someone else that we need a protocol that keeps the size of any "frictionless" community to a small multiple of Dunbar's number, so it's very nice to see your argument along the same lines.
I'm more interested in creating (small) common spaces than personal ones, though
And I'm struck by the irony of "I want to flee Twitter because Musk might charge for it" being met with, "The best way to Mastodon is to pay $ for your own instance"
Darnell Clayton
in reply to Aral Balkan • • •So it was your fault! 😂🤣😂
I agree with a solo instance. Ironically I had to upgrade my plan as I exceeded my bandwidth limits on the plan.
Christian HB9HOX
in reply to Aral Balkan • • •I guess, the same apply to the Matrix chat system.
Can we make running software easier? I do work in IT (as a developer, a system administrator and now as product owner) and I do not have the energy and time to run my own instances.
Jens Finkhäuser
in reply to Aral Balkan • • •Since I've been in the information-centric networking group at #IETF115, I'd like to add that this kind of tech would be a good complement to the small web notion.
The basic principle is that you're not looking to speak to a server, but asking the network where a piece of information is. So if my machine has already received Aral's post, then it's as legitimate a source of this post as his own, etc. Load spreads out.
@stephenfry @gretathunberg
Stefan Midjich ꙮ҄
in reply to Aral Balkan • • •Laurent Gatto
in reply to Aral Balkan • • •that was a very interesting read, with links for much food for thought (that I will be digesting in the coming days and weeks).
After the Frying risk assessment and you instance-of-one birthday saga, I was hesitating to follow and reply... but did in the end. I will keep the tooting low though 🙂
Jack Beauchamp-Stansfield
in reply to Aral Balkan • • •katzenberger
in reply to Aral Balkan • • •Somewhat agree, strongly disagree with many conclusions.
Single-tenant #instances are a ruthless attack on our planet.
#Mastodon is conceptually flawed with topical communities expressing themselves mainly via a local #timeline. Lists: poorly implemented. Federated #groups: a.gup.pe hacks
Look at the self-descriptions of any instance focused on a large #community, and ask yourself who could afford to *not* register at the *one* place where the *action* is.
@stephenfry @gretathunberg
humont
in reply to Aral Balkan • • •Really interesting article - this seems to be the major problem with choosing "push to notify" instead of "poll for notificatoins".
Hard problem to solve - the current structure disincentives producing good content that people want to follow - a problem that remains EVEN if you're using your solo-instance.
what other possible solutions are there?
FiXato
in reply to Aral Balkan • • •judging by how many of the replies to this post still include at-mentions to Fry and Thunberg despite their replies not actually being relevant to them, I also wonder if support for BCC rather than CC would be useful. ;)
Aside from that, it does sound like (as others in the replies have also remarked) that more smaller instances would actually increase the traffic?
Final note: you mistyped 'extinguish' as 'entinguish'. ;)
Aral Balkan
in reply to FiXato • • •mart0
in reply to Aral Balkan • • •Maybe a solution, especially for accounts with a lot of followers like you, is to be more intentional in your chattiness (and this could be a habit or norm across the fediverse).
You want to reply to everyone, but do all those replies have to be public posts? If you just want to say a "thank you" or a small courtesy back to someone you know ("thanks! how's your cat?"), you could make it a Direct Post. The way Mastodon puts these in thread for you and the other person like any other post (rather than jusr in a separate DM inbox) works great for this!
Unless I am really wrong about how Mastodon handles these, it would only be one jog going out to the one instance of that user, rather than broadcasting to everyone who follows you.
There's probably an analogy here about how you are using a high power concert PA system to broadcast a separate reply to every single person in the stadium audience, when instead you could just mingle into to crowd later and directly chat with each attendee (and Hugo is powering the PA by running on a treadmill backstage).
If you reply to this message from me with a simple "thanks, great point!" It could be a Direct Post. If you have some commentary or correction of value to your followers, keep it Public!
Ben Sahlmueller
in reply to Aral Balkan • • •sheislaurence
in reply to Aral Balkan • • •Minhaz Ali
in reply to Aral Balkan • • •Diabetic Heihachi
in reply to Aral Balkan • • •Self hosting is the way. After I joined, my first thoughts were "K neat, how do I host my own?"
Great article, my initial afterthoughts:
Instances need to be capable of a kind mitosis for when they get too large.
Throw more lvls of decentralization at instance level blocking?
With this 3000 requests loading up issue, perhaps there is a way to redistribute those requests so other instances can share a set amount of resources when they are below a certain load threshold? Hmm.
@lashman
Anders C. Sørby 🔥🌍 🇵🇸🇸🇩🇨🇩🇺🇦🌻⚛️
in reply to Aral Balkan • • •Grey the earthling
in reply to Aral Balkan • • •> Well, there’s only one thing you can do when you find yourself in such a pickle: scale up your Mastodon instance … Or start blocking followers, or unfollowing people, or staying quiet.
Maybe I know too little about how ActivityPub works and/or I've been drinking too much solar.lowtechmagazine.com but would another option be to *wait*?
The queue will resolve itself *eventually*, right? — so what if we treat this slower pace as a Feature, Not A Bug?
LOW←TECH MAGAZINE
LOW←TECH MAGAZINELarry Garfield
in reply to Aral Balkan • • •Pascal Mages
in reply to Aral Balkan • • •Mino
in reply to Aral Balkan • • •John Otto
in reply to Aral Balkan • • •Thank goodness for Fry!
Or thank Fry for goodness?
@stephenfry @gretathunberg @siberianluck
TallBaldDesi
in reply to Aral Balkan • • •Aral Balkan
in reply to TallBaldDesi • • •TallBaldDesi
in reply to Aral Balkan • • •Edgar Robles 🦦
in reply to Aral Balkan • • •maybe part of the problem with breaking up instances is the perceived cost and effort to maintain one . probably the same as maintaining an email server.
Nikolas Becker :verified:
in reply to Aral Balkan • • •IAmDarthMole
in reply to Aral Balkan • • •This may have been broached already and I just missed it but is mastodon designed to handle replies/likes interserver the same way it does intraserver?
Meaning is the massive increase in resources you saw a product of your account being on its own server and everyone else being on different servers?
Thanks!
Aral Balkan
in reply to IAmDarthMole • • •IAmDarthMole
in reply to Aral Balkan • • •IAmDarthMole
in reply to Aral Balkan • • •thanks for the reply! I see I misunderstood your article earlier.
The comment about having your own server if you had a lot of followers was meant to be a workaround to limit resource issues for the many due to the way mastodon was designed to handle interserver communications?
Would I be accurate in stating that your article was stating that the fact it handles intraserver communications so much better actually hurts itself and its mission as a decentralized champion?
Prakash C
in reply to Aral Balkan • • •Shanan
in reply to Aral Balkan • • •Paul Jacobson
in reply to Aral Balkan • • •I just read your post about the overall benefits of more Mastodon instances and how people should be encouraged to create their own.
My son has been working on his own instance by following the official docs. This really needs to be a lot easier to do, though.
#Linode and #DigitalOcean have very easy installation options although these options are pretty expensive compared to a similar VM option at a hosting provider like #Hetzner, for example.
I'm sure this is just a matter of time.