@ordnung

2 weeks ago

@ordnung
2 weeks ago

We are back online with the database from this morning 15.08.2025 at 05:20 CEST. Every local toot posted between this time and 13:45 CEST is lost so are the two registrations we saw today. Remote toots should catch up the next few hours.

The lost followings from our users to users from mastodon.social should be recovered in most cases. Sadly users from mastodon.social following our users are lost because the domain deletion process removes every existing relation between accounts. #mastoadmin

#mastoadmin

This entry was edited (2 weeks ago)

in reply to @ordnung

@ordnung

in reply to @ordnung 2 weeks ago

We are sorry this happend. As we mentioned we started a process to delete old cached instances that where offline for more than a year. Sadly in one line of over 18.000 lines was a mistake we hadn't expected. We checked the domains before but somehow didn't catch this. We currently have no explanation why exactly this single line was wrong. #mastoadmin

Screenshot showing a malformed script line where there is a mastodon.social that should have been another longer domain.

#mastoadmin

This entry was edited (2 weeks ago)

in reply to @ordnung

@ordnung

in reply to @ordnung 2 weeks ago

We also discussed a manual fix for the following relations by manually altering the database, but this would have cost us many hours more and we couldn't guarantee that it would work at all (and not causing new issues). Therefore we decided to use our daily backups to fix this as fast as possible. We will evaluate if we can increase the number of backups per day in the future. #mastoadmin

#mastoadmin

in reply to @ordnung

@ordnung

in reply to @ordnung 2 weeks ago

If you try to wrap you mind around how something like this could happen, please remember this is a project we run in our spare time. We aim for a good reliable service but shit like this can always happen.
#mastoadmin

#mastoadmin

in reply to @ordnung

@ordnung

in reply to @ordnung 2 weeks ago

Ok we are still analyzing the problem and we found 292 additional severed lines in that 18.000 lines file. But none of these caused any problem, because the domains where invalid. Looks like the multi line edit of VS Code isn't that reliable at all. #mastoadmin

#mastoadmin

in reply to @ordnung

@ordnung

in reply to @ordnung 2 weeks ago

Wow we found the problem it's so... special. The word wrap is on by default in VS Code. And if we select all, press Shift+ALT+I to enter multi line editing, then press POS1 to jump to the beginning of the line. Then press CTRL+V to enter the beginning of the line before the domain name. Until now everything works but we have some line breaks. #mastoadmin

#mastoadmin

in reply to @ordnung

@ordnung

in reply to @ordnung 2 weeks ago

Then we press "End" on our keyboard but instead of jumping to the actual end of the line it jumps to the point where the word wrapped and this is on some place inside the domain name. And this behavior caused the problem. GREAT!

End of post mortem.
#mastoadmin

#mastoadmin

in reply to @ordnung

Bubu

in reply to @ordnung 2 weeks ago

oh no. What a nightmare coincidence of bad luck.

Thanks for taking care of the instance (today and all the other days where nothing goes wrong)!

⇧

@ordnung 2 weeks ago • •

@ordnung
2 weeks ago