Hello people, we have been experiencing an issue with lvm2-thin on _some_ of our production servers where out of nowhere lvm2/device-mapper starts spamming error logs and I can't really seem to trace down the root cause. This is what the logs look like; Oct 9 06:25:02 U5bW8JT7 lvm[8020]: device-mapper: waitevent ioctl on LVM-CP5Gw8QrWLqwhBcJL87R1mc9Q9KTBtQQmOowipTAFuM7hqzHz6pRVvUaNO9FGzeq-tpool failed: Inappropriate ioctl for device Oct 9 06:25:02 U5bW8JT7 lvm[8020]: waitevent: dm_task_run failed: Inappropriate ioctl for device It writes this to rsyslog at speeds so fast not even tail can keep up over ssh. I'm really lost as to the reason or how to trace this back to a process. We use lvm-thin to host virtual machines via libvirt, so several thin disks under a thin pool one for each virtual machine. The lvm2 pool is created on top of a vg that is on top of a md-raid1 device. /dev/md127: Version : 1.2 Creation Time : Wed May 5 16:56:09 2021 Raid Level : raid1 Array Size : 1953382464 (1862.89 GiB 2000.26 GB) Used Dev Size : 1953382464 (1862.89 GiB 2000.26 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Wed Oct 9 07:31:44 2024 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : U5bW8JT7:1 (local to host U5bW8JT7) UUID : 5072120c:b114a0aa:51438cf8:8ebe58b8 Events : 70525 Number Major Minor RaidDevice State 0 259 0 0 active sync /dev/nvme0n1 1 259 1 1 active sync /dev/nvme1n1 I do not know what this `CP5Gw8QrWLqwhBcJL87R1mc9Q9KTBtQQmOowipTAFuM7hqzHz6pRVvUaNO9FGzeq-tpool` disk is on device mapper, I'm unable to find it, however the tpool suffix suggests it's the thin pool so here is what lvdisplay says about it --- Logical volume --- LV Name lvol1 VG Name lightning-nvme LV UUID mOowip-TAFu-M7hq-zHz6-pRVv-UaNO-9FGzeq LV Write Access read/write LV Creation host, time U5bW8JT7, 2021-05-05 16:57:13 +0000 LV Pool metadata lvol1_tmeta LV Pool data lvol1_tdata LV Status available # open 72 LV Size <1.80 TiB Allocated pool data 81.30% Allocated metadata 49.28% Current LE 471040 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2 The start of the device name suggests it's part of the vg so here's also what vgdisplay says: --- Volume group --- VG Name lightning-nvme System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 52521 VG Access read/write VG Status resizable MAX LV 0 Cur LV 72 Open LV 70 Max PV 0 Cur PV 1 Act PV 1 VG Size <1.82 TiB PE Size 4.00 MiB Total PE 476899 Alloc PE / Size 471098 / <1.80 TiB Free PE / Size 5801 / 22.66 GiB VG UUID CP5Gw8-QrWL-qwhB-cJL8-7R1m-c9Q9-KTBtQQ Any tips on how to trace this back to a process or a cause or a bug or anything are highly appreciated. I literally only have these two log lines to go by, everything else is working as expected until the server crashes from running out of disk space from these logs. Restarting the node fixes the issue/log spam until it starts happening again randomly after several days. Runnig Ubuntu 18.04.5 LTS - lvm2 2.02.176-4.1ubuntu3.18.04.3 - --Fabricio Winter