The best way to test the resilience of your service is: reboot the server. I mean it. Don't just restart the service, reboot the server. A story in two parts:

Part 1: a while ago I configured ZFS under my Talos Kubernetes cluster, and everything was fine, until I decided to reboot. When it came back, nothing was working, because I forgot to properly configure a way for Talos to read the ZFS volume encryption key.

Part 2: at some point I configured Audiobookshelf to store data on top of said ZFS volume. Everything was working fine for a few weeks, until I had to reboot the server again (for reasons). When it came back, I lost all my downloaded podcasts, because I had a typo on my configuration that was pointing to a directory outside of the PVC, so it mounted as an emptyDir volume.

Honestly, I should have known better. I had issues in the past when some servers went down because of power failures (battery didn't last) and they did not come back properly.

You gotta do a reboot/power test every once in a while, just like you have to test your backups on a regular basis.

#HomeLab #TalosLinux #ZFS #SRE #DevOps @homelab

Wie versprochen der Blogbeitrag über mein neues #Homelab "Konzept".

teqqy.de/selfhosted-setup-2025…

#selfhosted #komodo #gitops @homelab_de

Selfhosted Setup 2025: Mein neuer Workflow mit Proxmox, Komodo und GitOps

Mein Homelab-Rebuild 2025 – weg von Kubernetes, hin zu GitOps auf Proxmox: Komodo, Gitea, Renovate und ein schlanker VM-Stack für zuverlässiges Selfhosting.

^teqqy.de

The self hoster's paradox...

Everything working well: "I'm bored, there's nothing to play with 🥱"

Something breaks: "This is not how I want to spend my Sunday night, why do I do this 😭"
#selfhosting #fediadmin #homelab

In early September, The Matrix Foundation homeserver went down.

I'm extremely proud of our SRE team. They had a Disaster Recovery Plan and monthly exercises to apply it, resulting in no data loss despite a 24h outage.

I've learned a lot about how to properly backup/restore a Postgres database when writing this post with SREs. We also learned how to better prevent and be resilient to human error.

Thanks all for the hugops during the outage!

matrix.org/blog/2025/10/post-m…

#homelab #selfHosting #sre

Post-mortem of the September 2 outage

Matrix, the open protocol for secure decentralised communications

^{Matthew Hodgson (matrix.org)}

Any #ZFS folks want to offer me an opinion on the following? I have 2 ZFS pool of raidz2 with 7 2TB SAS drives each (about 11.5T). I want to add a LOG drive. I have a single SAS SSD that's 300G. When I suggested using a 1TB LOG drive earlier, someone said that was way too much. (which is why I picked up a used 300G SSD)

My question is whether it makes sense (is possible) to partition that drive into 2 partitions and have one pool use 1 partition as a log drive (e.g., 150G) and the other pool use the other partition as a log drive on the same physical device.

Is this going to be worse because it's just too much IO on one device? Is it reasonable? Any other ideas?

Thanks
#homelab #linux #selfhost

I've been on the hunt for some accessible, easily self-hosted (on prem or datacenter) chat solution for a while now, and I've been looking very closely at deltachat/chatmail.

Not gonna lie, the fact that just uses mature SMTP facilities for transport is making me weirdly aroused.

If anyone has experience running their own chatmail server or just using deltachat in general, I'd be glad to hear from you.

#DeltaChat #ChatMail #SelfHosted #HomeLab

I keep plugging the drive sled into the server and the lights never come on and the server doesn’t recognise the drive.

PEBCAS - Problem exists between chair and screwdriver.
#homelab #diy #server

I use mattermost as a free, open source Slack replacement. I like it a lot. I also use authentik as my SSO provider for my #homelab / #selfhosted stuff.

If you don't pay for the enterprise version of Mattermost, you don't get the SSO features. You can fake that by using their free GitLab integration. But you also don't get the capability to switch users from one form of authentication (email/password) to SSO after you set it up.

This is pretty obscure, and it involves a bunch of raw Postgres queries. But I figured it out and wrote a blog on converting Mattermost users from email/password to SSO

Mattermost Converting Users from Email to Gitlab authentication | Paco Hope

If you start your Mattermost Team Edition using email and password for users, but then you change your mind and switch to GitLab authentication, it’s not at all obvious or easy how to convert the existing users.

^blog.paco.to

#SelfHost #SelfHosting #SelfHosted #HomeLab

[source]

This Week in Self-Hosted (19 January 2024)

The latest news (ft. @matrix and @plex), software updates and launches (ft. @doncow #Immich, #Paperless, #PiAlert, and others), a spotlight on #wgeasy, upcoming events, and more in this week's self-hosted recap!

selfh.st/newsletter/2024-01-19…

#selfhost #selfhosted #selfhosting #homelab #homeserver #opensource #newsletter

This Week in Self-Hosted (19 January 2024)

Self-hosted news, software updates, launches, and a spotlight on wg-easy, a WireGuard management container with a web interface

^{Ethan Sholly (selfh.st)}

RT @twicechild41@twitter.com

Bringing up a #synapse (Matrix.org) stack with @traefik@twitter.com, @PostgreSQL@twitter.com, #elementio, #facebook and #telegram bridges from #mautrix and a bot manager from #maubot, with #dockercompose. My first ever guide! github.com/twicechild/matrix-s…

#matrixorg #traefik #PostgreSQL #homelab

🐦🔗: twitter.com/twicechild41/statu…

Matrix.org

Matrix is an open standard for interoperable, decentralised, real-time communication

^Matrix.org