I improved the logging output: https://gist.github.com/omarkilani/2ad526c3546b40537b546450c8f685dc And deadlocked again: https://gist.githubusercontent.com/omarkilani/3d870b6dc440e04357add8c66d371d86/raw/29437a909af475b92fe4d259ff059beea65cdb8e/wbt.deadlock-002.log On Sun, Jun 13, 2021 at 10:03 AM Omar Kilani <omar.kilani@xxxxxxxxx> wrote: > > Just looking at blk-wbt.c... > > Should... > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/blk-wbt.c?h=v5.13-rc5&id=482e302a61f1fc62b0e13be20bc7a11a91b5832d#n164 > > if (!inflight || diff >= rwb->wb_background / 2) > > Be: > > if (!inflight || diff >= limit / 2) > > ? > > On Sun, Jun 13, 2021 at 8:49 AM Omar Kilani <omar.kilani@xxxxxxxxx> wrote: > > > > Hi there, > > > > I appear to have stumbled upon a deadlock in wbt or rq-qos. > > > > My journal of a lot of data points is over here: > > > > https://github.com/openzfs/zfs/issues/12204 > > > > I initially deadlocked on RHEL 8.4's 4.18.0-305.3.1.el8_4.x86_64 > > kernel, but the code in blk-wbt.c / blk-rq-qos.c is functionally > > identical to 5.13.0-rc5, so I tried that and I'm able to deadlock that > > as well. I believe the same code exists all the way back to 5.0.1. > > > > The Something Weird (tm) about this is that it possibly only happens > > on AMD EPYC CPUs. I just don't have the necessary setup to confirm > > that either way, but it's a hunch because I can't reproduce it on an > > Ice Lake VM (but the Ice Lake VM also has more storage bandwidth so > > that could be the thing, and I can't decrease that storage bandwidth, > > so I can't do a like-for-like test.) > > > > I "instrumented" wbt / rq-qos with a bunch of printk's which you can > > see with this patch: > > > > https://gist.github.com/omarkilani/2ad526c3546b40537b546450c8f685dc > > > > I then ran my repro workload to cause the deadlock, here's the dmesg > > output just before the deadlock and then the backtraces with my printk > > patch applied: > > > > https://gist.githubusercontent.com/omarkilani/ff0a96d872e09b4fb648272d104e0053/raw/d3da3974162f8aa87b7309317af80929fadf250f/dmesg.wbt.deadlock.log > > > > Happy to apply whatever / run whatever to get more data. > > > > Thanks! > > > > Regards, > > Omar
