I wish that #Matrix (and #Synapse) were better designed.
It has become among the most popular open messaging protocols, but it remains a scalability nightmare.
It's literally the Blockchain of messaging. Its current state is given by the sum of the whole chain of events received since t=0. It's a brittle append-only ledger, and any modifications you do to it are very likely to break it.
Do you have a huge 100GB database and you want to clean up old stuff? Sorry, you can't. There are some non-official solutions for compacting the events, but they're all likely to break your db - and none worked so far in my case.
Do you have users on your instance that entered a busy room on matrix.org? Then you'll get all the join/leave events of that busy room on your db, with no way of deleting them, and nothing to do unless your users exit those rooms.
Did you start your instance by toying with the default SQLite backend, and now that it's become big you want to move to Postgres? Sorry, no official guides provided, only unofficial procedures scattered across a bunch of blogs.
Do you want to change the name of your server? Sorry, you can't. All the events are tightly coupled to your server name. The advised solution is to simply start a new instance.
Do you want to keep the same server name but spin off a new instance? Sorry, you can't, because your server name, your keys and the events sent to other servers are all tightly coupled, so your new instance may not be able to reliably exchange messages with servers that were federated to your previous one.
Do you want to migrate your account and conversations to a new server, or even just get a backup of your data? Well, you can just forget it, because there's no way of disentangling the thick mesh of events and moving it somewhere else.
Such a rigid and brittle implementation shouldn't have become an open de facto standard without much questioning about its poor design decisions.
At the very least, some official tools must be provided to enable admins to compact events. If the size of the database is guaranteed to increase indefinitely, then entry barriers against self-hosting are only going to increase.
The Matrix.org Foundation
in reply to Fabio Manganiello • • •there's a lot of confusion here.
Matrix is deliberately designed to *NOT* require all events since t=0. It requires the key-value events (i.e. state events) for the room, but not the actual messages. In turn, room DAGs can and are split into partitions called 'chunks' whenever there's a gap in federation.
So yes, you can't go and manually edit the DAG, any more than you can manually edit commits in a git repository. But you can absolutely purge old data (https://element-hq.github.io/synapse/latest/admin_api/purge_history_api.html) and...
Purge History - Synapse
element-hq.github.ioThe Matrix.org Foundation
in reply to The Matrix.org Foundation • • •...you can compress old state events via https://github.com/matrix-org/rust-synapse-compress-state (which is stable, and should be merged into Synapse).
Similarly, as others have said, synapse_port_db for sqlite->postgres should absolutely work (if not, it's a bug. there have been some in the past, but as far as we know it works currently).
Finally, account portability is in progress (https://github.com/devonh/matrix-spec-proposals/blob/cryptoIDs/proposals/4080-cryptographic-identities.md), but needs funding to complete.
So while we're definitely not perfect, we're not quite that shit either.
GitHub - matrix-org/rust-synapse-compress-state: A tool to compress some state in a Synapse instance's database
GitHubtyil
in reply to Fabio Manganiello • • •weilawei
in reply to tyil • • •Yeah, the implementations really leave something to be desired. I run Prosody, and it gets particularly cranky under certain circumstances. For example, the file upload module and large file sizes do not get along well, resulting in the Prosody process itself going completely zombified. It doesn't happen every time (hard to reproduce) but above 5 MiB, the problems start to show up with increasing frequency.
That said, it'd be better to just improve XMPP and fix the server bugs.
tyil
in reply to weilawei • • •I set up #Prosody last weekend, and I haven't checked file uploads yet. The setup was incredibly simple, and resource usage is next to nothing. Added #Biboumi to make it function as my #IRC bouncer too, and it works like a charm!
This leads me to believe that even the current state of XMPP is already beats Matrix by a wide margin. Its easier, less resource hungry, faster, and there are already multiple good server and client implementations available.