You are right, it seems to be something else than the udev rules. Let me paste the boot messages real quick. On Fri, Sep 20, 2024 at 5:42 AM Coly Li <colyli@xxxxxxx> wrote: > > On my distribution, I see multipath rules is 56-multipath.rules, it should be executed early than bcache one. I am not an systemd and boot expert, but what is the order of your multipath rules? > > Coly Li > > > 2024年9月20日 19:24，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道： > > > > Coly, > > > > Yes, I made that observation in multipath-tools upstream bug[0]. > > > > Ultimately the issue is that the bcache-tools udev rule runs > > bcache-register on the device and therefore it can't be opened by > > multipath tools. > > > > Therefore, the udev rule should wait to run until after multipath has > > a chance to make the maps. I'm not sure if there's any complications > > with moving the rules later into the boot process though, WDYT? > > > > -Mitch > > > > On Mon, Sep 9, 2024 at 8:56 AM Coly Li <colyli@xxxxxxx> wrote: > >> > >> > >> > >>> 2024年9月6日 07:37，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道： > >>> > >>> Thanks for the response Coly! > >>> > >>> Apologies on my delay, I was out for a bit and then was sick == lots > >>> of catching up to do. > >>> > >>> Getting back to this, I do see that we have the bcache KCONFIG value enabled > >>> > >>> CONFIG_BCACHE_ASYNC_REGISTRATION=y > >>> > >>> I don't believe this is a race condition issue. If I try to reload the > >>> devmap table with multipath-tools well after boot, I see the following > >>> in dmesg: > >>> > >>> $ sudo multipath -r > >>> [ 8758.157075] device-mapper: table: 252:3: multipath: error getting > >>> device (-EBUSY) > >>> [ 8758.158039] device-mapper: ioctl: error adding target to table > >>> [ 8758.256206] device-mapper: table: 252:3: multipath: error getting > >>> device (-EBUSY) > >>> [ 8758.256758] device-mapper: ioctl: error adding target to table > >> > >> When you see the above kernel message, can you check whether bcache device is initialized already? > >> Or you may post the boot time kernel message as attachment, or past it somewhere, then let me have a look. > >> > >> Thanks. > >> > >> > >> Coly Li > >> > >> > >> > >> > >>> > >>> FYI - Since I'm not sure if this is a bug in bcache-tools or > >>> multipath-tools, I also made an upstream bug report for > >>> multipath-tools at[0]. > >>> > >>> If there is anything else you'd like me to try, let me know :) > >>> > >>> [0] - https://github.com/opensvc/multipath-tools/issues/96 > >>> > >>> On Mon, Aug 12, 2024 at 12:17 AM Coly Li <colyli@xxxxxxx> wrote: > >>>> > >>>> Hi Mitchell, > >>>> > >>>> > >>>> It sounds like a timing issue of the initialization scripts. I assume the cache and backing devices are relative large, so the initialization takes time. And because the storage is not local, the remote link contributes more time to wait for the bcache device being ready. > >>>> > >>>> If this is the case, then you have to tune the multipath initialization to wait for longer, or compose a customized script to start services depending on bcache devices. > >>>> > >>>> BTW, I assume the bcache Kconfig “Asynchronous device registration” is checked/enabled. If not, maybe check it on can be a bit helpful. > >>>> > >>>> Just FYI. > >>>> > >>>> Coly Li > >>>> > >>>>> 2024年8月12日 10:54，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道： > >>>>> > >>>>> Thanks for the reply Coly. > >>>>> > >>>>> I've been able to reproduce this in Ubuntu Noble and Oracular (24.04 > >>>>> && 24.10). It should be an issue in Jammy but haven't tested that yet. > >>>>> The current kernel used in Oracular is 6.8.0-31.31 and the current > >>>>> kernel used in Noble is 6.8.0-40.40. > >>>>> > >>>>> Unfortunately you need an account to access pastebin. I can copy that > >>>>> information elsewhere for you if that would be helpful, but I can also > >>>>> just gather any extra information you may want from my testbed. > >>>>> > >>>>> I also have some steps in the bug report to reproduce this issue using kvm. > >>>>> > >>>>> Lastly, if there's any steps you'd like me to try or look into, I'd be > >>>>> glad to hear :) > >>>>> > >>>>> -Mitch > >>>>> > >>>>> On Sun, Aug 11, 2024 at 5:31 AM Coly Li <colyli@xxxxxxx> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>>> 2024年8月8日 09:21，Mitchell Dzurick <mitchell.dzurick@xxxxxxxxxxxxx> 写道： > >>>>>>> > >>>>>>> Hello bcache team. > >>>>>>> > >>>>>>> I know this project is done and stable as [0] says, but I have a > >>>>>>> question if anyone is around to answer. > >>>>>>> > >>>>>>> Has bcache devices been tested and supported on multipath'd disks? I'm > >>>>>>> looking into an Ubuntu bug[1], where these 2 projects are clashing. > >>>>>>> I'm wondering if there was any consideration or support for > >>>>>>> multipathing when this project was made. > >>>>>>> > >>>>>>> Also, your new project, bcachefs, might be hitting the same scenario. > >>>>>>> I haven't had the time to test this though unfortunately. > >>>>>>> > >>>>>>> Thanks for your time, > >>>>>>> -Mitch > >>>>>>> > >>>>>>> [0] - https://bcache.evilpiepirate.org/#index4h1 > >>>>>>> [1] - https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1887558 > >>>>>>> > >>>>>> > >>>>>> From the Ubuntu bug report, I don’t see the kernel version. After parallel and asynchronous initialization was enabled, the udev rule won’t always occupy the bcache block device for long time. > >>>>>> > >>>>>> It might be a bit helpful if you may provide the kernel version and Ubuntu os version. BTW I don’t have ubuntu account and cannot access pastern.canonical.com. > >>>>>> > >>>>>> Thanks. > >>>>>> > >>>>>> Coly Li > >>>>> > >>>> > >>> > >> >
