Hi All, We've been tracking a scenario where IO on one of our hosts locks up.We were hopeful that https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.2.15 and specifially 62a4b137d8aff9ce9dc5e96685253190b4ec6e62 (aka 38d11da522aacaa05898c734a1cec86f1e611129) would fix this.
It did not.What we consistently see when we're in time to catch this prior to being unable to log in is an lvcreate process running, not always this specific LV but a variation of:
13469 ? D<L 0:00 | | \_ /sbin/lvcreate -kn -An -s -n fsck_MDKServer /dev/lvm/backup_MDKServer
(The idea here is to create thin snapshots of the origin volume in order to run a fsck on them so that we can recover from filesystem corruption on the server which acts as a backup server for others, once the fsck is done the snapshot is then renamed to save_* after removing the previous save_* LVs.)
This always seems to happen on lvcreate, not lvremove nor lvrename.I'm attaching dmesg -T and ps axf. dmesg in particular may provide clues as it provides a number of stack traces indicating stalling at IO time.
Once this has triggered, even commands such as "lvs" goes into uninterruptable wait, I unfortunately didn't test "dmsetup ls" now and triggered a reboot already (system needs to be up).
To recover we have to "echo b > /proc/sysrq-trigger" since a normal reboot simply doesn't work (presumably due to being unable to close and unmount the filesystems, even emergency sync never completes).
In terms of disk setup, there are a number of mdraids that serves as PVs into the VG. /proc/mdstat and some details from LVM attached as disksetup.txt.
The one thing from lvs that stands out for me is this one:66 fsck_MDKServer lvm Vwi---tz-- 4.00t thin_pool backup_MDKServer
Which to me indicates that the lvcreate didn't complete (the LV isn't marked active above, whereas all other snaps are active, even if not open).
Assistance would be greatly appreciated. I notice that newer kernels are out, so happy to upgrade if there are relevant changes that might affect this. Nothing at https://kernelnewbies.org/Linux_6.3#Block_layer jumps out to me.
This host locks up approximately weekly. It's worse if dmeventd is also running (won't survive a day). We do have the following "non-standard" udev rules, which I'm not sure if they might be an influencing factor:
crowsnest [10:24:37] /etc/udev/rules.d (master) # cat 98-md-stripe-cache.rules SUBSYSTEM=="block", KERNEL=="md*", ACTION=="change|add", TEST=="md/stripe_cache_size", ATTR{md/stripe_cache_size}="32768" SUBSYSTEM=="block", ACTION=="change|add", TEST=="bdi/read_ahead_kb", ATTR{bdi/read_ahead_kb}="512" SUBSYSTEM=="block", ACTION=="change|add", TEST=="device/timeout", ATTR{device/timeout}="180"
We've not aware of any other hosts being affected, at least not to this scale. This is the host where we make the most heavy use of both mdraid as well as lvm (especially thin snaps). In comparison most of our other hosts are toys w.r.t. disks.
A few things I find strange (perhaps due to ignorance):1. It's not just the LV that's being snapped that locks up, it's always more than just that one.
2. Some filesystems are usually still usable (/var). Once the lvcreate triggered the issue, these get fewer over time, once /var is affected we can no longer log in.
To be noted: when this host idles we see ~40-60MB/s worth of IO, goes up to 400-800MB/s and we saw it peak around 1.3GB/s around two weeks back whilst also investigating possible causes (Current DISK READ/WRITE as per iotop).
Filesystems are all ext4 with O_DIRINDEX disabled (standard options otherwise).
Kind regards, Jaco
Attachment:
disksetup.txt.gz
Description: application/gzip
Attachment:
dmesg.txt.gz
Description: application/gzip
Attachment:
ps.txt.gz
Description: application/gzip
