mastodon - Link to source

TIL that there's this major issue with the #matrix protocol where sometimes messages in rooms are just lost. Gone. Poof. We've been losing messages in the pmOS main channel for a few days now, it's pretty frustrating when trying to help folks.

They've apparently known about this for years yet there's no public info I can find about it. The only advice is to "recreate the room", which is quite disruptive when there are thousands of participants (and it'll happen again)

"This is fine." - @matrix

#matrix @The Matrix.org Foundation
This entry was edited (2 weeks ago)
in reply to f_ 🇵🇸

mastodon - Link to source

Clayton

@vitali64sur has been investigating this for us and hasn't been able to find much beyond what he commented here. If this affects the matrix.org room, it's kinda weird you haven't heard of this 😉

We aren't super excited about having to recreate the room, especially if there's no assurance that this will actually fix/prevent it from happening again later. The synapse folks don't think it's a problem with synapse or homeserver configuration... What other options do we have?

@f_ 🇵🇸
in reply to Clayton

mastodon - Link to source

The Matrix.org Foundation

@vitali64sur this thread looks to be talking about 3 different issues (OFTC IRC bridge problems; state resets & "split brains"), in different rooms(?). Which room are you talking about? I linked to the two known problems (state resets, which we're in the final stages of fixing) and CVE-2025-30355 which was fixed in March. It's the latter which was exploited against Matrix HQ, but it doesn't cause message loss once patched; just misordering.
@f_ 🇵🇸
in reply to f_ 🇵🇸

mastodon - Link to source

The Matrix.org Foundation

@vitali64sur if the problem is pmOS's self-hosted IRC bridge, why are linking bugs against the Matrix/Element-hosted OFTC bridge? (github.com/matrix-org/matrix-a…) Or is that a separate issue?

OFTC bridge does no longer create new puppets on new IRC users · Issue #1851 · matrix-org/matrix-appservice-irc

How to reproduce: Join fron IRC, on a matrix-bridged channel, with a nickname the OFTC bridge has never seen before send messages see that there is no matrix-side ghost joining, and the messages do...
GitHub
@f_ 🇵🇸
in reply to The Matrix.org Foundation

mastodon - Link to source

The Matrix.org Foundation

@vitali64sur okay, so the problem on #main:postmarketos.org is malicious federation traffic as mentioned at matrix.org/blog/2025/04/11/thi…. we're fixing this as part of state reset work, meanwhile there are workarounds available. we'll take a look.

This Week in Matrix 2025-04-11

Matrix, the open protocol for secure decentralised communications
Thib (matrix.org)
#main @f_ 🇵🇸
in reply to The Matrix.org Foundation

mastodon - Link to source

The Matrix.org Foundation

@vitali64sur for the sake of completeness, any problems in #_oftc_#alpine-linux:matrix.org are probably related to github.com/matrix-org/matrix-a… and unrelated to pmOS, and separately the #forgejo-chat:matrix.org issue mentioned elsewhere (codeberg.org/forgejo/discussio…) sounds like it's 'just' state resets.

Proposal: Recreating forgejo chat room on matrix from scratch

## Why The current chat room has a very high complexity and cannot maintain/persist the current state.
Codeberg.org
#alpine #forgejo #_oftc_ @f_ 🇵🇸
in reply to The Matrix.org Foundation

mastodon - Link to source

f_ 🇵🇸

Strange? The pmOS.org matrix homeserver has been upgraded mere hours after github.com/element-hq/synapse/…, or is it something else?

Federation denial of service via malformed events

### Impact A malicious server can craft events which, when received, prevent Synapse version up to 1.127.0 from federating with other servers. The vulnerability has been exploited in the wild. ...
GitHub
in reply to achill @ im just home

mastodon - Link to source

The Matrix.org Foundation

@fossdd @vitali64sur um, glad you see. are you talking about irc<->matrix problems? or some kind of matrix protocol issue? there's a class of bugs which cause state to reset (github.com/matrix-org/synapse/…) which we've been working on and almost finished, and there was a recent bug that could cause disruption (but not lost messages): github.com/element-hq/synapse/…. what matrix room actually is this?

Federation denial of service via malformed events

### Impact A malicious server can craft events which, when received, prevent Synapse version up to 1.127.0 from federating with other servers. The vulnerability has been exploited in the wild. ...
GitHub
@achill @ im just home @f_ 🇵🇸
⇧