A long time ago, before even Geocities, I had a blog that had enough traffic to get noticed, and I was approached by a researcher at the University of Georgia offering me free hosting so long as uptime wasn't critical.

The site was called Dragonfire. The experiment was squeezing as much power out of a commodity white box as technically possible. It was overclocked to the hilt, water cooled, and the name was chosen after some coffee spilled on the case and started boiling.

LB 👆: Does anyone know if all ActivityPub implementations (Pleroma, Misskey and all the others) suffer from the scaling challenges Mastodon does?

If Aral (see the post I'm replying to) was running, say, microblog.pub, would he have the same issues?

This is a really helpful framing for the problem, and seems like something we're going to have to grapple with sooner rather than later. A few questions spring to mind:

- What is the current cost of maintaining an identity, as a function of followers, activity, media, etc?
- How much can that be optimized in the code? Through protocol improvements?
- Assuming the optimized cost is still non-trivial, are there significant advantages to shared hosting?

❣️ the energy.

If everyone has their own instance/server wouldn't this example require 23k Sidekiq jobs instead of 3k?

Isn't that LESS scalable, for everyone.

You highlight a #fediverse problem to be solved but not sure everyone going DIY is the answer. (If this is the case.)

@ClaudetteK Alas, I have failed :)

@Saket

There’s some tension between the points you make and the notion of the small instance hosting a community of exercise enthusiasts or esperantistoj.

As both you and @profcarroll point out, a well connected small instance can be quite resource intensive. A poorly connected instance is a virtual ghost town. Could some of this be solved by using lighter weight ActivityPub software? (Misskey or Pleroma)

In your post you talk about your server (which I assume is some sort of virtual private server) having 12 sidekiq threads and can deliver an ActivityPub update to 12 other servers at a time.

Do you have any idea how much performance the virtual server provided by mastohost has in values that allow a comparison to other VPSes? As in, what is "a sidekiq thread" for the server running it, do you get one sidekiq thread per CPU? Is CPU speed of the server, the web connection, or speed of the recieving server (e.g. if it needs to acknowledge recieving the ActivityPub push) the limiting factor for the speed at which such a sidekiq job executes? Maybe @hugo can provide some info on this?

I had to laugh a bit at “Do you know what happens to your email if Google says (rightly or wrongly) that you’re spam? No one sees your email.”

Yeah. That’s me these days. Hence moving press folks over to my own domain (but a different address from personal email).

I'm very new to mastodon, and still trying to learn the basics of how it is structured. Your article has been very informative and, though coming from a very different and more-practical perspective, increasingly seems to raise concerns along the same lines as I've had as I've been reading about this.

Specifically: that this thing seems to be too centralised (lots of little centralised servers, but still centralised), and (if I'm reading correctly) only a single level of federation?

@neilhimself FYI ⬆️

@aral @stephenfry @gretathunberg

Could I ask you a few naive questions, as I am not up to speed on the fediverse architecture.

It sounds like the problem of exponential job growth are technically solvable in code (rather than inherent in the system), is that correct?

Can the jobs not get gracefully queued and balanced, with the result popular people are just processed more slowly?

Can the jobs be batched so newer jobs can do batch updates and allow older jobs to be dropped?

Thank you

This is concerning. It seems to imply that decentralization is impossible long term for most people.

I see this going in the direction of celebrities creating Mastodon instances and starting to post ads to users on their instance, and possibly charge users for the benefit of getting priority access to their posts. (You can see them before the sync jobs run for other instances).

I guess if you want freedom from that you just need to live with a worse experience, at least you have the choice.

Thanks, genuinely interesting and thought provoking. I’m barely known anywhere and my first instinct was to setup my own instance, if only to have some control over my presence.

This does present other challenges you don’t cover - natural follows from things like being in a local timeline just aren’t there. There really is a strong incentive for providers and users for centralisation.

Short of a fully P2P (ala Chord/Pastry) system, genuinely stumped on technical solutions here.

I think you're on to something, but the concept of single-user instances is taking too far. Your instance does a lot of work pulling in your federated timeline, but that work only benefits one user: you.

If you have a 300-user instance, but those 300 people follow a lot of the same people, the load-per-user is significantly lower.

There's a happy medium

@katco

It's a plain truism in performance engineering that if you free up one bottleneck, say the overhead on the queue itself, then the bottleneck will move and it may well mean that waiting on read or write locks becomes mutual deadlocks.

Or more prosaically running out of connections, handles to a dependent service. All those are resolvable with effort.

But it could be considerable effort.

This is a great essay, and articulates extremely well a number of concerns that I have had while reading other descriptions of how Mastodon is different or better because of its architecture.

I have been feeling like the kid in the Emperor's New Clothes, looking around me and going "can no-one else see what I'm seeing???"

Thanks for saving my sanity!

That was a really interesting and informative read. I'm learning more everyday. Thanks!

DON'T REPLY! 😂

I've just been arguing with someone else that we need a protocol that keeps the size of any "frictionless" community to a small multiple of Dunbar's number, so it's very nice to see your argument along the same lines.

I'm more interested in creating (small) common spaces than personal ones, though

And I'm struck by the irony of "I want to flee Twitter because Musk might charge for it" being met with, "The best way to Mastodon is to pay $ for your own instance"

So it was your fault! 😂🤣😂

I agree with a solo instance. Ironically I had to upgrade my plan as I exceeded my bandwidth limits on the plan.

I guess, the same apply to the Matrix chat system.

Can we make running software easier? I do work in IT (as a developer, a system administrator and now as product owner) and I do not have the energy and time to run my own instances.

Since I've been in the information-centric networking group at #IETF115, I'd like to add that this kind of tech would be a good complement to the small web notion.

The basic principle is that you're not looking to speak to a server, but asking the network where a piece of information is. So if my machine has already received Aral's post, then it's as legitimate a source of this post as his own, etc. Load spreads out.

@stephenfry @gretathunberg