On Fri, Sep 07, 2018 at 03:28:28PM +0200, Lars Ellenberg wrote:
> We don't expose that, no.
> But even if we did, that would not be racefree :-)
>
> The last (or even: any?) "close" of a block device that used to be open
> for WRITE triggeres a udev "change" event, thus a udev run,
> and the minimal action will be some read-only open and ioctl from
> (systemd-)udev itself, more likely there also will be blkid and possibly
> pvscan and similar actions. All of them should be read-only openers,
> and all of them should be "short".
> But they will race with the drbd demotion.
True, but did not find any strange interaction with udev during VM shutdown.
This is what udevadm monitor reports:
KERNEL[1174.220256] remove /devices/vbd-10-51712 (xen-backend)
UDEV [1174.222484] remove /devices/vbd-10-51712 (xen-backend)
KERNEL[1174.224405] remove /devices/console-10-0 (xen-backend)
UDEV [1174.226964] remove /devices/console-10-0 (xen-backend)
KERNEL[1174.287215] change /devices/virtual/block/drbd0 (block)
KERNEL[1174.287267] change /devices/virtual/block/drbd0 (block)
UDEV [1174.295811] change /devices/virtual/block/drbd0 (block)
UDEV [1174.301983] change /devices/virtual/block/drbd0 (block)
Strace on the udev daemon gives only these:
[pid 7416] execve("/sbin/drbdadm", ["/sbin/drbdadm", "sh-udev", "minor-0"], [/* 10 vars */]) = 0
[pid 7416] execve("/lib/drbd/drbdadm-84", ["drbdadm", "sh-udev", "minor-0"], [/* 12 vars */]) = 0
[pid 7418] execve("/sbin/drbdadm", ["/sbin/drbdadm", "sh-udev", "minor-0"], [/* 9 vars */]) = 0
[pid 7418] execve("/lib/drbd/drbdadm-84", ["drbdadm", "sh-udev", "minor-0"], [/* 11 vars */]) = 0
and this should be 65-drbd.rules to add/remove symlinks in /dev.
Adding a dump_stack in drbd_release gives two possible code paths,
both from xen_blkback and the first one from workqueue being the
problematic one:
[ 530.698782] CPU: 0 PID: 4 Comm: kworker/0:0 Tainted: G O 4.9.0-8-amd64 #1 Debian 4.9.110-3+deb9u4
[ 530.698783] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 09/12/2016
[ 530.698784] Workqueue: events xen_blkif_deferred_free [xen_blkback]
[ 530.698785] 0000000000000000 ffffffff81331e54 ffff883f82143800 ffff883f741b5660
[ 530.698787] ffffffffc03a309e ffff883f741b5580 ffffffff81245ca8 0000000281a186b5
[ 530.698789] 000000008935f100 8412ffa31a64cc4c 000000000000020a ffff883fa6218280
[ 530.698791] Call Trace:
[ 530.698792] [<ffffffff81331e54>] ? dump_stack+0x5c/0x78
[ 530.698805] [<ffffffffc03a309e>] ? drbd_release+0x1e/0x40 [drbd]
[ 530.698810] [<ffffffff81245ca8>] ? __blkdev_put+0x1e8/0x2a0
[ 530.698813] [<ffffffffc055d6b8>] ? xen_vbd_free.isra.9+0x48/0x60 [xen_blkback]
[ 530.698814] [<ffffffffc055d6f7>] ? xen_blkif_deferred_free+0x27/0x70 [xen_blkback]
[ 530.698816] [<ffffffff81092fea>] ? process_one_work+0x18a/0x420
[ 530.698817] [<ffffffff810932cd>] ? worker_thread+0x4d/0x490
[ 530.698818] [<ffffffff81093280>] ? process_one_work+0x420/0x420
[ 530.698820] [<ffffffff81099329>] ? kthread+0xd9/0xf0
[ 530.698822] [<ffffffff81099250>] ? kthread_park+0x60/0x60
[ 530.698823] [<ffffffff81615df7>] ? ret_from_fork+0x57/0x70
[ 1216.251924] CPU: 14 PID: 297 Comm: xenwatch Tainted: G O 4.9.0-8-amd64 #1 Debian 4.9.110-3+deb9u4
[ 1216.251925] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 09/12/2016
[ 1216.251926] 0000000000000000 ffffffff81331e54 ffff883f82143800 ffff883f741b5660
[ 1216.251928] ffffffffc03a309e ffff883f741b5580 ffffffff81245ca8 0000000281a186b5
[ 1216.251930] ffffffffc0562360 97e48f39d448f082 000000000000020a 0000000000000006
[ 1216.251933] Call Trace:
[ 1216.251935] [<ffffffff81331e54>] ? dump_stack+0x5c/0x78
[ 1216.251947] [<ffffffffc03a309e>] ? drbd_release+0x1e/0x40 [drbd]
[ 1216.251951] [<ffffffff81245ca8>] ? __blkdev_put+0x1e8/0x2a0
[ 1216.251954] [<ffffffff81414910>] ? register_xenbus_watch+0xe0/0xe0
[ 1216.251956] [<ffffffffc055d6b8>] ? xen_vbd_free.isra.9+0x48/0x60 [xen_blkback]
[ 1216.251959] [<ffffffffc055e9fe>] ? frontend_changed+0x9e/0x660 [xen_blkback]
[ 1216.251961] [<ffffffff814120d9>] ? xenbus_read_driver_state+0x39/0x60
[ 1216.251962] [<ffffffff8141542c>] ? xenbus_otherend_changed+0x8c/0x120
[ 1216.251964] [<ffffffff81414910>] ? register_xenbus_watch+0xe0/0xe0
[ 1216.251967] [<ffffffff81414995>] ? xenwatch_thread+0x85/0x120
[ 1216.251969] [<ffffffff810bbfb0>] ? prepare_to_wait_event+0xf0/0xf0
[ 1216.251971] [<ffffffff81099329>] ? kthread+0xd9/0xf0
[ 1216.251973] [<ffffffff81099250>] ? kthread_park+0x60/0x60
[ 1216.251975] [<ffffffff81615df7>] ? ret_from_fork+0x57/0x70
--
Valentin
