Hi, I am seeing occasional hard to reproduce failures to lvcreate a thin LV,
with transaction ID mismatch errors.
The system is a self-managed compute node that uses thin LVs for base system
software and containers - each service has a separate thin LV as its rootfs, and
the system takes a fresh thin snapshot of the installed contents at every boot.
During system bring up we have two concurrent processes adding, deleting, and
renaming thin LVs from a single thin pool:
1 - a login script that creates a thin snapshot of a minimal rootfs for each
user, then launches an LXC container with that rootfs, and leaves the user in
bash running in that container. If any issues occur in any of that process, it
will lvremove the snapshot and retry several times. Although it creates a
container, the script itself runs as sudo root, not inside any
container/namespaces.
2 - a software install service that takes system services packaged as containers
and creates thin LVs based on the container image layer set, and then takes an
additional thin snapshot to be mounted for the current boot. this last snapshot
is multi-step, with an initial lvcreate of a temp name and a final lvrename.
Neither of these processes are in containers or on a VM with a shared volume, so
they should be seeing the same LVM lock files, as far as I can tell.
This overall approach has been stable for a long time, but a recent change has
caused these to overlap more frequently, and we are now seeing failures in
lvcreate with a transaction id mismatch when the install service tries to create
its temporary LV - here's a snippet from one such log:
Error: lvm lvcreate --activate=y --setactivationskip=y --ignoreactivationskip --name=tmp-extract-414e5ec83c02133eae2984ee4
25b22589bca058d --snapshot vg_ifc0/5e9a280e11efbc75ed8f01bdd7b58559c373b451b72921555be5c2eaf93d27b2: exit status 5: /dev/sdh: open failed: No medium found
/dev/sdi: open failed: No medium found
/dev/sdj:ound
/dev/sdk: open failed: No medium found
/dev/sdh: open failed: No medium found
/dev/sdi: open failed: No medium found
/dev/sdj: open failed: No medium found
/dev/sdk: open failed: No medium fou
ThinDataLV-tpool (251:3) transaction_id is 147, while expected 148.
Failed to suspend vg_ifc0/ThinDataLV with queued messages.
Due to some failure recovery loops, these services are running
lvcreate/lvremove/lvrename (on same VG but different LVs) as often as 5 times
per second, which seems fast but doesn't seem like it should be a problem.
Looking through past messages to this list, it looks like previous cases were
due to sharing volumes between containers/vms without a common lock dir, which
we are not doing.
Any thoughts on how to further debug or avoid this issue?
I can provide the lvm metadata backup files if that would help - there are a lot
of them, as once it starts failing, the system retries frequently.
Ihis is on Ubuntu 20.04, with lvm 2.03.07(2)
(ubuntu package version 2.03.07-1ubuntu1)
and a custom kernel built from 5.15.68.
Thanks!
-mike
with transaction ID mismatch errors.
The system is a self-managed compute node that uses thin LVs for base system
software and containers - each service has a separate thin LV as its rootfs, and
the system takes a fresh thin snapshot of the installed contents at every boot.
During system bring up we have two concurrent processes adding, deleting, and
renaming thin LVs from a single thin pool:
1 - a login script that creates a thin snapshot of a minimal rootfs for each
user, then launches an LXC container with that rootfs, and leaves the user in
bash running in that container. If any issues occur in any of that process, it
will lvremove the snapshot and retry several times. Although it creates a
container, the script itself runs as sudo root, not inside any
container/namespaces.
2 - a software install service that takes system services packaged as containers
and creates thin LVs based on the container image layer set, and then takes an
additional thin snapshot to be mounted for the current boot. this last snapshot
is multi-step, with an initial lvcreate of a temp name and a final lvrename.
Neither of these processes are in containers or on a VM with a shared volume, so
they should be seeing the same LVM lock files, as far as I can tell.
This overall approach has been stable for a long time, but a recent change has
caused these to overlap more frequently, and we are now seeing failures in
lvcreate with a transaction id mismatch when the install service tries to create
its temporary LV - here's a snippet from one such log:
Error: lvm lvcreate --activate=y --setactivationskip=y --ignoreactivationskip --name=tmp-extract-414e5ec83c02133eae2984ee4
25b22589bca058d --snapshot vg_ifc0/5e9a280e11efbc75ed8f01bdd7b58559c373b451b72921555be5c2eaf93d27b2: exit status 5: /dev/sdh: open failed: No medium found
/dev/sdi: open failed: No medium found
/dev/sdj:ound
/dev/sdk: open failed: No medium found
/dev/sdh: open failed: No medium found
/dev/sdi: open failed: No medium found
/dev/sdj: open failed: No medium found
/dev/sdk: open failed: No medium fou
ThinDataLV-tpool (251:3) transaction_id is 147, while expected 148.
Failed to suspend vg_ifc0/ThinDataLV with queued messages.
Due to some failure recovery loops, these services are running
lvcreate/lvremove/lvrename (on same VG but different LVs) as often as 5 times
per second, which seems fast but doesn't seem like it should be a problem.
Looking through past messages to this list, it looks like previous cases were
due to sharing volumes between containers/vms without a common lock dir, which
we are not doing.
Any thoughts on how to further debug or avoid this issue?
I can provide the lvm metadata backup files if that would help - there are a lot
of them, as once it starts failing, the system retries frequently.
Ihis is on Ubuntu 20.04, with lvm 2.03.07(2)
(ubuntu package version 2.03.07-1ubuntu1)
and a custom kernel built from 5.15.68.
Thanks!
-mike
_______________________________________________ linux-lvm mailing list linux-lvm@xxxxxxxxxx https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
