> 2022年6月21日 21:35，Nikhil Kshirsagar <nkshirsagar@xxxxxxxxx> 写道： > > Hi Coly, > > thanks, so , I will also change uapi/linux/bcache.h SB_JOURNAL_BUCKETS > to 8 but I noted the note in the kernel header file, which said "/* > SB_JOURNAL_BUCKETS must be divisible by BITS_PER_LONG */" which I > think is 32.. Hi Nikhil, It’s fine, for me I modify it to 16, and 8 should work as well. > > As of now I have not built the backported patch into the kernel > (Ubuntu-4.15.0-183.192) which I am simply using to reproduce the > issue. Then I will build the patch into the kernel and verify that the > deadlock isn't ever seen. My problem is simply reproducing it for now > (without the fix compiled in). What I'm trying is (modify > SB_JOURNAL_BUCKETS to 8 both in bcache-tools and kernel, and then > created the bcache device and then filled it up by fio random writes > of about 250 gb data, and then run constantly.. - > https://pastebin.com/Sh3q6fXv .. not a reboot. Do you think in fact a > reboot is needed to reproduce it, and this approach of > de-register/register would not work to see the deadlock?) Reboot is not necessary. You can also try to stop all the cache and backing devices and removed the bcache kernel module, and reload bcache.ko and re-register cache and backing device. It is not easy to reproduce the deadlock, last time when I tried to do it, indeed I triggered another deadlock in journal code. This was why I stopped fixing the journal no-space deadlock because I though it would not happen in practice. This time I am not able to trigger the no-space deadlock, what I do is to observe whether there is enough space always reserved. If there is space reserved, then it won’t deadlock. > > Once I reproduce it successfully a few times, I'll compile in the > backported patch and then verify I never see the deadlock again, > that's the best I can do I guess in the circumstances. > > Thank you (and to Kent++ for helping me with the backport patch) for > all the help with this! I am very grateful for your help and > responses. > Indeed, I am not sure whether what you triggered is the journal no-space deadlock or other deadlock which was not fixed in 4.15 kernel. So I am not able to provide advice. For me, if I see there is always reserved journal space which can be used in registration time, then I believe the no-space deadlock won’t happen. Coly Li > > On Tue, 21 Jun 2022 at 18:53, Coly Li <colyli@xxxxxxx> wrote: >> >> >> >>> 2022年6月21日 13:40，Nikhil Kshirsagar <nkshirsagar@xxxxxxxxx> 写道： >>> >>> I figured later that you probably meant for me to change the >>> SB_JOURNAL_BUCKETS to 8 in bcache-tools and not the kernel? >>> >>> Regards, >>> Nikhil. >> >> >> Hi Nikhil, >> >> As I said in previous offline email, you should modify both bcache-tool and kernel code for SB_JOURNAL_BUCKETS, to 8 or 16, and recompile both. >> With the patch it is very hard to reproduce the deadlock (because it is fixed by this patch), you may observe the free journal space in run time and reboot time. If there is alway at least 1 journal bucket reserved during run time, then you won’t observe the journal no-space deadlock in boot time. >> >> But 4.15 kernel is not robust enough for bcache (5.4+ is good and 5.10+ is better), if you are stucked by other bugs during this testing, it is possible. >> >> Coly Li >> >> >>> >>> On Tue, 21 Jun 2022 at 11:06, Nikhil Kshirsagar <nkshirsagar@xxxxxxxxx> wrote: >>>> >>>> Thank you Kent, >>>> >>>> I've made this change, in include/uapi/linux/bcache.h and will build >>>> the kernel with it to attempt to reproduce the issue, and create a new >>>> bcache device. Just wondering if the note about it being divisible by >>>> BITS_PER_LONG may restrict it to a minimum value of 32? >>>> >>>> #define SB_JOURNAL_BUCKETS 8 >>>> /* SB_JOURNAL_BUCKETS must be divisible by BITS_PER_LONG */ >>>> >>>> I have a "cache" nvme disk of about 350 tb and some slow disks, each >>>> approx 300tb which I will use to create the bcache device once the >>>> kernel is installed. My bcache setup typically would look like, >>>> >>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >>>> sdb 8:16 0 279.4G 0 disk >>>> └─bcache0 252:0 0 279.4G 0 disk >>>> sdc 8:32 0 279.4G 0 disk >>>> └─bcache2 252:256 0 279.4G 0 disk >>>> sdd 8:48 0 279.4G 0 disk >>>> └─bcache1 252:128 0 279.4G 0 disk >>>> nvme0n1 259:0 0 372.6G 0 disk >>>> ├─bcache0 252:0 0 279.4G 0 disk >>>> ├─bcache1 252:128 0 279.4G 0 disk >>>> └─bcache2 252:256 0 279.4G 0 disk >>>> >>>> Regards, >>>> Nikhil. >>>> >>>> On Tue, 21 Jun 2022 at 10:05, Kent Overstreet <kent.overstreet@xxxxxxxxx> wrote: >>>>> >>>>> On Tue, Jun 21, 2022 at 09:11:10AM +0530, Nikhil Kshirsagar wrote: >>>>>> Hello all, >>>>>> >>>>>> I am trying to reproduce the problem that >>>>>> 32feee36c30ea06e38ccb8ae6e5c44c6eec790a6 fixes, but I am not sure how. >>>>>> This is to verify and test its backport >>>>>> (https://pastebin.com/fEYmPZqC) onto kernel 4.15 (Thanks Kent for the >>>>>> help with that backport!) >>>>>> >>>>>> Could this be reproduced by creating a bcache device with a smaller >>>>>> journal size? And if so, is there some way to pass the journal size >>>>>> argument during the creation of the bcache device? >>>>> >>>>> Change SB_JOURNAL_BUCKETS to 8 and make a new cache, it's used in the >>>>> initialization path. >>>>> >>>>> Bonus points would be to tweak journal reclaim so that we're slower to reclaim >>>>> to makes sure the journal stays full, and then test recovery. >>
