Re: PROBLEM: repeatable lockup on RAID-6 with LUKS dm-crypt on NVMe devices when rsyncing many files

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

> On 15. Aug 2024, at 17:53, John Stoffel <john@xxxxxxxxxxx> wrote:
> 
>> I’m not making progress here. I can’t reproduce those on in-memory
>> loopback raid 6. However: i can’t fully produce the rsync. For me
>> this only triggered after around 1.5hs of progress on the NVMe which
>> resulted in the hangup. I can only create around 20 GiB worth of
>> raid 6 volume on this machine. I’ve tried running rsync until it
>> exhausts the space, deleting the content and running rsync again,
>> but I feel like this isn’t suffient to trigger the issue. :(
> 
> You're running on the older 5.13.x kernel or the newer 6.x kernel?  

6.10.3. I can reproduce reliably on the stack of dm-crypt/mdraid/xfs, but not on in memory things at the moment. From my perspective, because my test case on there isn’t big enough.

>> I’m trying to find whether any specific pattern in the files around
>> the time it locks up might be relevant here and try to run the rsync
>> over that portion.
> 
> That's a good idea.  Do you have highly compressed files which are
> maybe exploding in size when put into LUKS encrypted partitions?  Just
> a random thought, not a real idea.

Nope. No encryption in the files, also it wasn’t true that I could reproduce on specific portions. When restarting from “fresh” it kept going to different points until it crashed and it needed to churn through around 200-300GiB until it finally caved in.

>> On the plus side, I have a script now that can create the various
>> loopback settings quickly, so I can try out things as needed. Not
>> that valuable without a reproducer, yet, though.
> 
> Yay!  Please share it.

Will do next week after a bit of cleanup.

Thanks for helping so far - I’ll keep pulling this thread until it becomes loose, I hope … ;)

Christian

-- 
Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick






[Index of Archives]     [Linux RAID Wiki]     [ATA RAID]     [Linux SCSI Target Infrastructure]     [Linux Block]     [Linux IDE]     [Linux SCSI]     [Linux Hams]     [Device Mapper]     [Device Mapper Cryptographics]     [Kernel]     [Linux Admin]     [Linux Net]     [GFS]     [RPM]     [git]     [Yosemite Forum]


  Powered by Linux