Help diagnosing lvm ioctl log spam

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello people, we have been experiencing an issue with lvm2-thin on
_some_ of our production servers where out of nowhere
lvm2/device-mapper starts spamming error logs and I can't really seem
to trace down the root cause.

This is what the logs look like;
Oct  9 06:25:02 U5bW8JT7 lvm[8020]: device-mapper: waitevent ioctl on
LVM-CP5Gw8QrWLqwhBcJL87R1mc9Q9KTBtQQmOowipTAFuM7hqzHz6pRVvUaNO9FGzeq-tpool
failed: Inappropriate ioctl for device
Oct  9 06:25:02 U5bW8JT7 lvm[8020]: waitevent: dm_task_run failed:
Inappropriate ioctl for device

It writes this to rsyslog at speeds so fast not even tail can keep up
over ssh. I'm really lost as to the reason or how to trace this back
to a process. We use lvm-thin to host virtual machines via libvirt, so
several thin disks under a thin pool one for each virtual machine. The
lvm2 pool is created on top of a vg that is on top of a md-raid1
device.

/dev/md127:
           Version : 1.2
     Creation Time : Wed May  5 16:56:09 2021
        Raid Level : raid1
        Array Size : 1953382464 (1862.89 GiB 2000.26 GB)
     Used Dev Size : 1953382464 (1862.89 GiB 2000.26 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Oct  9 07:31:44 2024
             State : active
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name :  U5bW8JT7:1  (local to host  U5bW8JT7)
              UUID : 5072120c:b114a0aa:51438cf8:8ebe58b8
            Events : 70525

    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync   /dev/nvme0n1
       1     259        1        1      active sync   /dev/nvme1n1

I do not know what this
`CP5Gw8QrWLqwhBcJL87R1mc9Q9KTBtQQmOowipTAFuM7hqzHz6pRVvUaNO9FGzeq-tpool`
disk is on device mapper, I'm unable to find it, however the tpool
suffix suggests it's the thin pool so here is what lvdisplay says
about it

--- Logical volume ---
  LV Name                lvol1
  VG Name                lightning-nvme
  LV UUID                mOowip-TAFu-M7hq-zHz6-pRVv-UaNO-9FGzeq
  LV Write Access        read/write
  LV Creation host, time U5bW8JT7, 2021-05-05 16:57:13 +0000
  LV Pool metadata       lvol1_tmeta
  LV Pool data           lvol1_tdata
  LV Status              available
  # open                 72
  LV Size                <1.80 TiB
  Allocated pool data    81.30%
  Allocated metadata     49.28%
  Current LE             471040
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

The start of the device name suggests it's part of the vg so here's
also what vgdisplay says:

--- Volume group ---
  VG Name               lightning-nvme
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  52521
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                72
  Open LV               70
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <1.82 TiB
  PE Size               4.00 MiB
  Total PE              476899
  Alloc PE / Size       471098 / <1.80 TiB
  Free  PE / Size       5801 / 22.66 GiB
  VG UUID               CP5Gw8-QrWL-qwhB-cJL8-7R1m-c9Q9-KTBtQQ

Any tips on how to trace this back to a process or a cause or a bug or
anything are highly appreciated. I literally only have these two log
lines to go by, everything else is working as expected until the
server crashes from running out of disk space from these logs.
Restarting the node fixes the issue/log spam until it starts happening
again randomly after several days.

Runnig Ubuntu 18.04.5 LTS - lvm2 2.02.176-4.1ubuntu3.18.04.3

-
--Fabricio Winter




[Index of Archives]     [Gluster Users]     [Kernel Development]     [Linux Clusters]     [Device Mapper]     [Security]     [Bugtraq]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]

  Powered by Linux