I’ve been on Fedi for a year now and have fallen in love with the platform, so here’s my introduction! I studied computer science with a focus on RTOS and FP/PL, but I’m about to start my final semester of law school. I’ll be practicing at a boutique firm that primarily handles IP cases once I’m barred. I foilboard and I play way too many rhythm games in and out of the arcade.

I contribute to open source projects where I can, and I write up my experience in my digital garden which I’ve been maintaining for over a year now. It’s also a good place to find usage tips for projects/tools that you might want to use.

• RSS feed

My passion for tech also includes privacy, and I’m an advocate for minimizing your digital footprint. GenAI is a scam and its purveyors are causing real harm while they sell it as hard as they can.

Follow me for: #selfhosting #digitalgardening #privacy #lawfedi #section230 #copyright #patents #rustlang #gleam #haskell #RSS #neovim #NixOS #zotero #tmux #alacritty #linux #egpu #qemu #arch #archlinux #GNOME #watches #watchmaking #obsidian #obsidianmd #thunderbird #fpv #mechkeys #mechkeeb #mechanicalkeyboard #matrix #signal #fido2 #passkeys #dancerushstardom

be-far's Digital Garden

Tip You will own nothing, and you will be happy. On my little corner of the internet, I document my adventures in tech and complain about the internet of shit.

^be-far.com

Okay Linux systems/platform folks, I need your ideas.

We run Linux VMs using #QEMU. Periodically we have a workload where there is a significant delay between our orchestrator invoking qemu-system-x86_64 and the guest VM OS starting.

(Guest OS starting in this case is evidenced by clout-init logging written to serial console that we persist.)

The delay is in the range of 10-30 minutes. When it happens, we see a significant increase in the following hypervisor metrics from qemu invocation until the guest OS starts cloud init:

- disk read requests
- disk read volume (kbs)
- % of cpu time spent running the kernel
- % time during which i/o requests were issued

How can we figure out what's going on?

Some additional things that might be helpful to know:

- hypervisor hosts are running debian 10 on intel hardware
- we haven't identified any other patterns when this happens, though it seems to always happen on hosts that are already running other VMs
- the data volume is LVM on LUKS encrypted partition
- hypervisor hosts have 2 physical drives in RAID configuration (the exact one I need to figure out)
- happens with stock debian 11 cloud image as guest OS

Any ideas?

I want to better understand what qemu is doing between that initial invocation when when the guest OS starts cloud-init. Is there any way to get more logging during this time? What are other ways to surveil what might be the bottleneck?