Hi, > On 9. Aug 2024, at 03:13, Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote: > > > Yes, for sure IO are stuck in md127 and never get dispatched to nvme, > for now I'll say this is a raid5 problem. Note, that this is raid6, not raid5! Sorry, I never explicitly mentioned that and it was buried in the mdstat output. Not sure whether that code is internally the same anyway… > Can you describe in steps how do you reporduce this problem? We must > figure out where are those IO and why they're not dispatched. I'll be > easier to debug if I can reporduce it. Otherwise I'll have to give you > a debug patch. My workload is pretty simple IMHO: 1. This is on an XFS volume: meta-data=/dev/mapper/backy isize=512 agcount=112, agsize=268435328 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=1 data = bsize=4096 blocks=30001587200, imaxpct=1 = sunit=128 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 2. I’m rsyncing a large number of files using: rsync -avz “<remote-machine>:/<remote-folder>" . Specific aspects of the data that might come into play: After 5-6 tries of syncing and locking up, I now have a directory on that volume that contains 154.820 files directly and 1.828.933 files recursively. The recursive structure uses a /year/<hash prefix>/<hash>-filename structure to spread out the 1.8 million files more evenly. The total amount of data is 1.0 TiB at the moment. The smallest files are empty or just a few kilobytes. The mode of the filesizes is 180.984 bytes. There is a second workload on the machine that has been running smoothly for a few weeks which backs up virtual machine images and creates a somewhat similarly (but more consistent) hashed structure but doesn’t use rsync but a more complex diff + content hashing + compression approach. However, that other tool does emit write barriers here and there -wWhich rsync doesn’t AFAIK. I double checked, but the other workload was not active during my last lockup. Happy to apply a debug patch to help diagnosing this. Cheers, Christian -- Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick