There is some more conversation with the multipath-tools bug, but it
seems the udev rules might actually be at play here.
Making the bcache-tools udev rules respect SYSTEMD_READY has
alleviated the problems for me in my testbed. This is a generic
mechanism that other udev rules use (btrfs as an example). Would you
consider looking into including something like this?
To do it is pretty easy, you just add this at the top of the udev rule:
ENV{SYSTEMD_READY}=="0", GOTO="bcache_end"
-Mitch
On Thu, Sep 26, 2024 at 8:55 AM Coly Li <colyli@xxxxxxx> wrote:
>
> I doubt bcache_allocator involved in this issue. After the cache device registered, an extra refcount will be added to the block device, this is expected.
> And Bcache is fine with multipath, because another path has identical UUID, the registration just fails and exits.
>
> Why multipath was not initialized at the first place, this is an open question for me.
>
> Coly Li
>
>
>
> > 2024年9月26日 09:15，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道：
> >
> > Is it perhaps the bcache_allocator interacting with the block device?
> > I dug around a little bit but couldn't conclude if the allocator was
> > interacting with the block device, but there might be a better way to
> > confirm that, which I am not aware of.
> >
> > On Wed, Sep 25, 2024 at 5:55 PM Mitchell Dzurick
> > <mitchell.dzurick@xxxxxxxxxxxxx> wrote:
> >>
> >> Yes, sdc is another path to sdd. The lsblk command looks like this:
> >>
> >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> >> fd0 2:0 1 4K 0 disk
> >> sda 8:0 0 20G 0 disk
> >> └─mpatha 252:1 0 20G 0 mpath
> >> ├─mpatha-part1 252:2 0 1M 0 part
> >> └─mpatha-part2 252:3 0 20G 0 part /
> >> sdb 8:16 0 20G 0 disk
> >> └─mpatha 252:1 0 20G 0 mpath
> >> ├─mpatha-part1 252:2 0 1M 0 part
> >> └─mpatha-part2 252:3 0 20G 0 part /
> >> sdc 8:32 0 1G 0 disk
> >> └─mpathb 252:0 0 1G 0 mpath
> >> sdd 8:48 0 1G 0 disk
> >> └─mpathb 252:0 0 1G 0 mpath
> >> sr0 11:0 1 2.6G 0 rom
> >>
> >> Where I invoke the KVM command like this:
> >> $ kvm -m 2048 -boot c \
> >> -cdrom ./oracular-live-server-amd64.iso \
> >> -device virtio-scsi-pci,id=scsi \
> >> -drive file=image.img,if=none,id=sda,format=raw,file.locking=off \
> >> -device scsi-hd,drive=sda,serial=0001 \
> >> -drive if=none,id=sdb,file=image.img,format=raw,file.locking=off \
> >> -device scsi-hd,drive=sdb,serial=0001 \
> >> -drive file=image2.img,if=none,id=sdc,format=raw,file.locking=off \
> >> -device scsi-hd,drive=sdc,serial=0002 \
> >> -drive if=none,id=sdd,file=image2.img,format=raw,file.locking=off \
> >> -device scsi-hd,drive=sdd,serial=0002 \
> >> -netdev user,id=net0,hostfwd=tcp::2222-:22 \
> >> -device virtio-net-pci,netdev=net0
> >>
> >>> Do you know why multipath failed to get device 252:1 ? And why I don’t see backing device of bcache here ?
> >> I'm not sure at the moment, that's what I'm trying to get to the bottom of.
> >>
> >> The one thing that lead me down this path of bcache being involved
> >> with this issue, is simply that by `sudo apt purge bcache-tools` I see
> >> the device is picked up by multipath-tools successfully, so it seems
> >> something in the bcache-tools package is causing this.
> >>
> >> Is there any specific thing you'd like me to look for, or a command
> >> you'd like me to run?
> >>
> >> I originally assumed that the reason multipath failed was that the
> >> bcache udev rule opens the device file, but that is clearly a
> >> misunderstanding since the multipath udev rules run first, so
> >> _something_ is intereacting with the device early and making it busy,
> >> I'm just not sure what right now.
> >>
> >> On Sun, Sep 22, 2024 at 6:55 PM Coly Li <colyli@xxxxxxx> wrote:
> >>>
> >>> Hi Mitchell,
> >>>
> >>> Yes I can see the log.
> >>>
> >>> [ 1.226296] device-mapper: multipath service-time: version 0.3.0 loaded
> >>> [ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)
> >>> [ 1.277086] device-mapper: ioctl: error adding target to table
> >>> [ 1.300407] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 1
> >>> [ 1.301650] bcache: register_cache() registered cache device sdd
> >>> [ 1.301748] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
> >>>
> >>>
> >>> From the above lines, it seems device mapper starts early, but both dm and bcache encounter a busy device.
> >>> Then bcache registered sdd successfully, and not register sdc. But I don’t see other information about how the backing device was registered.
> >>>
> >>> From the following message,
> >>> [ 4.049395] bcache: register_bcache() error : device already registered
> >>> [ 4.268997] bcache: register_cache_worker() error /dev/sdc: fail to register cache device
> >>>
> >>> I assume sdc is the another path of sdd?
> >>> From the logs, it seems all bcache behaviors are expected, except for no backing device existed/registered.
> >>>
> >>> Also I see multipath rule might execute before bcache one, but explains “[ 1.277039] device-mapper: table: 252:1: multipath: error getting device (-EBUSY)”. I don’t know how it happens, but this is a suspicious one.
> >>>
> >>> Do you know why multipath failed to get device 252:1 ? And why I don’t see backing device of bcache here ?
> >>>
> >>> Thanks.
> >>>
> >>> Coly Li
> >>>
> >>>> 2024年9月20日 21:11，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道：
> >>>>
> >>>> Coly,
> >>>>
> >>>> Are you able to view this?
> >>>>
> >>>> https://gist.github.com/mitchdz/addb44425709fd6df5a6b1b91611b234
> >>>>
> >>>> -Mitch
> >>>>
> >>>> On Fri, Sep 20, 2024 at 5:51 AM Mitchell Dzurick
> >>>> <mitchell.dzurick@xxxxxxxxxxxxx> wrote:
> >>>>>
> >>>>> You are right, it seems to be something else than the udev rules. Let
> >>>>> me paste the boot messages real quick.
> >>>>>
> >>>>> On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@xxxxxxx> wrote:
> >>>>>>
> >>>>>> On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules?
> >>>>>>
> >>>>>> Coly Li
> >>>>>>
> >>>>>>> 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道：
> >>>>>>>
> >>>>>>> Coly,
> >>>>>>>
> >>>>>>> Yes, I made that observation in multipath-tools upstream bug[0].
> >>>>>>>
> >>>>>>> Ultimately the issue is that the bcache-tools udev rule runs
> >>>>>>> bcache-register on the device and therefore it can't be opened by
> >>>>>>> multipath tools.
> >>>>>>>
> >>>>>>> Therefore, the udev rule should wait to run until after multipath has
> >>>>>>> a chance to make the maps. I'm not sure if there's any complications
> >>>>>>> with moving the rules later into the boot process though, WDYT?
> >>>>>>>
> >>>>>>> -Mitch
> >>>>>>>
> >>>>>>> On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@xxxxxxx> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道：
> >>>>>>>>>
> >>>>>>>>> Thanks for the response Coly!
> >>>>>>>>>
> >>>>>>>>> Apologies on my delay, I was out for a bit and then was sick == lots
> >>>>>>>>> of catching up to do.
> >>>>>>>>>
> >>>>>>>>> Getting back to this, I do see that we have the bcache KCONFIG value enabled
> >>>>>>>>>
> >>>>>>>>> CONFIG_BCACHE_ASYNC_REGISTRATION=y
> >>>>>>>>>
> >>>>>>>>> I don't believe this is a race condition issue. If I try to reload the
> >>>>>>>>> devmap table with multipath-tools well after boot, I see the following
> >>>>>>>>> in dmesg:
> >>>>>>>>>
> >>>>>>>>> $ sudo multipath -r
> >>>>>>>>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting
> >>>>>>>>> device (-EBUSY)
> >>>>>>>>> [ 8758.158039] device-mapper: ioctl: error adding target to table
> >>>>>>>>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting
> >>>>>>>>> device (-EBUSY)
> >>>>>>>>> [ 8758.256758] device-mapper: ioctl: error adding target to table
> >>>>>>>>
> >>>>>>>> When you see the above kernel message, can you check whether bcache device is initialized already?
> >>>>>>>> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Coly Li
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> FYI - Since I'm not sure if this is a bug in bcache-tools or
> >>>>>>>>> multipath-tools, I also made an upstream bug report for
> >>>>>>>>> multipath-tools at[0].
> >>>>>>>>>
> >>>>>>>>> If there is anything else you'd like me to try, let me know :)
> >>>>>>>>>
> >>>>>>>>> [0] - https://github.com/opensvc/multipath-tools/issues/96
> >>>>>>>>>
> >>>>>>>>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@xxxxxxx> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Mitchell,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready.
> >>>>>>>>>>
> >>>>>>>>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices.
> >>>>>>>>>>
> >>>>>>>>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful.
> >>>>>>>>>>
> >>>>>>>>>> Just FYI.
> >>>>>>>>>>
> >>>>>>>>>> Coly Li
> >>>>>>>>>>
> >>>>>>>>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道：
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the reply Coly.
> >>>>>>>>>>>
> >>>>>>>>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04
> >>>>>>>>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet.
> >>>>>>>>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current
> >>>>>>>>>>> kernel used in Noble is 6.8.0-40.40.
> >>>>>>>>>>>
> >>>>>>>>>>> Unfortunately you need an account to access pastebin. I can copy that
> >>>>>>>>>>> information elsewhere for you if that would be helpful, but I can also
> >>>>>>>>>>> just gather any extra information you may want from my testbed.
> >>>>>>>>>>>
> >>>>>>>>>>> I also have some steps in the bug report to reproduce this issue using kvm.
> >>>>>>>>>>>
> >>>>>>>>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be
> >>>>>>>>>>> glad to hear :)
> >>>>>>>>>>>
> >>>>>>>>>>> -Mitch
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@xxxxxxx> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello bcache team.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I know this project is done and stable as [0] says, but I have a
> >>>>>>>>>>>>> question if anyone is around to answer.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm
> >>>>>>>>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing.
> >>>>>>>>>>>>> I'm wondering if there was any consideration or support for
> >>>>>>>>>>>>> multipathing when this project was made.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, your new project, bcachefs, might be hitting the same scenario.
> >>>>>>>>>>>>> I haven't had the time to test this though unfortunately.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for your time,
> >>>>>>>>>>>>> -Mitch
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1
> >>>>>>>>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Coly Li
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >
>
