Re: Deadlock in wbt / rq-qos

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Just looking at blk-wbt.c...

Should...

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/blk-wbt.c?h=v5.13-rc5&id=482e302a61f1fc62b0e13be20bc7a11a91b5832d#n164

if (!inflight || diff >= rwb->wb_background / 2)

Be:

if (!inflight || diff >= limit / 2)

?

On Sun, Jun 13, 2021 at 8:49 AM Omar Kilani <omar.kilani@xxxxxxxxx> wrote:
>
> Hi there,
>
> I appear to have stumbled upon a deadlock in wbt or rq-qos.
>
> My journal of a lot of data points is over here:
>
> https://github.com/openzfs/zfs/issues/12204
>
> I initially deadlocked on RHEL 8.4's 4.18.0-305.3.1.el8_4.x86_64
> kernel, but the code in blk-wbt.c / blk-rq-qos.c is functionally
> identical to 5.13.0-rc5, so I tried that and I'm able to deadlock that
> as well. I believe the same code exists all the way back to 5.0.1.
>
> The Something Weird (tm) about this is that it possibly only happens
> on AMD EPYC CPUs. I just don't have the necessary setup to confirm
> that either way, but it's a hunch because I can't reproduce it on an
> Ice Lake VM (but the Ice Lake VM also has more storage bandwidth so
> that could be the thing, and I can't decrease that storage bandwidth,
> so I can't do a like-for-like test.)
>
> I "instrumented" wbt / rq-qos with a bunch of printk's which you can
> see with this patch:
>
> https://gist.github.com/omarkilani/2ad526c3546b40537b546450c8f685dc
>
> I then ran my repro workload to cause the deadlock, here's the dmesg
> output just before the deadlock and then the backtraces with my printk
> patch applied:
>
> https://gist.githubusercontent.com/omarkilani/ff0a96d872e09b4fb648272d104e0053/raw/d3da3974162f8aa87b7309317af80929fadf250f/dmesg.wbt.deadlock.log
>
> Happy to apply whatever / run whatever to get more data.
>
> Thanks!
>
> Regards,
> Omar



[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux