W dniu 28.07.2024 o 12:30, Mateusz Jończyk pisze: > W dniu 25.07.2024 o 16:27, Paul E Luse pisze: >> On Thu, 25 Jul 2024 09:15:40 +0200 >> Mateusz Jończyk <mat.jonczyk@xxxxx> wrote: >> >>> Dnia 24 lipca 2024 23:19:06 CEST, Paul E Luse >>> <paul.e.luse@xxxxxxxxxxxxxxx> napisał/a: >>>> On Wed, 24 Jul 2024 22:35:49 +0200 >>>> Mateusz Jończyk <mat.jonczyk@xxxxx> wrote: >>>> >>>>> W dniu 22.07.2024 o 07:39, Mateusz Jończyk pisze: >>>>>> W dniu 20.07.2024 o 16:47, Mateusz Jończyk pisze: >>>>>>> Hello, >>>>>>> >>>>>>> In my laptop, I used to have two RAID1 arrays on top of NVMe and >>>>>>> SATA SSD drives: /dev/md0 for /boot (not partitioned), /dev/md1 >>>>>>> for remaining data (LUKS >>>>>>> + LVM + ext4). For performance, I have marked the RAID component >>>>>>> device for /dev/md1 on the SATA SSD drive write-mostly, which >>>>>>> "means that the 'md' driver will avoid reading from these >>>>>>> devices if at all possible" (man mdadm). >>>>>>> >>>>>>> Recently, the NVMe drive started having problems (PCI AER errors >>>>>>> and the controller disappearing), so I removed it from the >>>>>>> arrays and wiped it. However, I have reseated the drive in the >>>>>>> M.2 socket and this apparently fixed it (verified with tests). >>>>>>> >>>>>>> $ cat /proc/mdstat >>>>>>> Personalities : [raid1] [linear] [multipath] [raid0] [raid6] >>>>>>> [raid5] [raid4] [raid10] md1 : active raid1 sdb5[1](W) >>>>>>> 471727104 blocks super 1.2 [2/1] [_U] >>>>>>> bitmap: 4/4 pages [16KB], 65536KB chunk >>>>>>> >>>>>>> md2 : active (auto-read-only) raid1 sdb6[3](W) sda1[2] >>>>>>> 3142656 blocks super 1.2 [2/2] [UU] >>>>>>> bitmap: 0/1 pages [0KB], 65536KB chunk >>>>>>> >>>>>>> md0 : active raid1 sdb4[3] >>>>>>> 2094080 blocks super 1.2 [2/1] [_U] >>>>>>> >>>>>>> unused devices: <none> >>>>>>> >>>>>>> (md2 was used just for testing, ignore it). >>>>>>> >>>>>>> Today, I have tried to add the drive back to the arrays by >>>>>>> using a script that executed in quick succession: >>>>>>> >>>>>>> mdadm /dev/md0 --add --readwrite /dev/nvme0n1p2 >>>>>>> mdadm /dev/md1 --add --readwrite /dev/nvme0n1p3 >>>>>>> >>>>>>> This was on Linux 6.10.0, patched with my previous patch: >>>>>>> >>>>>>> https://lore.kernel.org/linux-raid/20240711202316.10775-1-mat.jonczyk@xxxxx/ >>>>>>> >>>>>>> (which fixed a regression in the kernel and allows it to start >>>>>>> /dev/md1 with a single drive in write-mostly mode). >>>>>>> In the background, I was running "rdiff-backup --compare" that >>>>>>> was comparing data between my array contents and a backup >>>>>>> attached via USB. >>>>>>> >>>>>>> This, however resulted in mayhem - I was unable to start any >>>>>>> program with an input-output error, etc. I used SysRQ + C to >>>>>>> save a kernel log: >>>>>>> >>>>> Hello, >>>>> >>>>> Unfortunately, hardware failure seems not to be the case. >>>>> >>>>> I did test it again on 6.10, twice, and in both cases I got >>>>> filesystem corruption (but not as severe). >>>>> >>>>> On Linux 6.1.96 it seems to be working well (also did two tries). >>>>> >>>>> Please note: in my tests, I was using a RAID component device with >>>>> a write-mostly bit set. This setup does not work on 6.9+ out of the >>>>> box and requires the following patch: >>>>> >>>>> commit 36a5c03f23271 ("md/raid1: set max_sectors during early >>>>> return from choose_slow_rdev()") >>>>> >>>>> that is in master now. >>>>> >>>>> It is also heading into stable, which I'm going to interrupt. > Hello, > > With much effort (challenging to reproduce reliably) I think have nailed down the issue to the read_balance refactoring series in 6.9: [snip] > After code analysis, I have noticed that the following check that was present in old > read_balance() is not present (in equivalent form in the new code): > > if (!test_bit(In_sync, &rdev->flags) && > rdev->recovery_offset < this_sector + sectors) > continue; > > (in choose_slow_rdev() and choose_first_rdev() and possibly other functions) > > which would cause the kernel to read from the device being synced to before > it is ready. Hello, I think have made a reliable (and safe) reproducer for this bug: Prerequisite: create an array on top of 2 devices 1GB+ large: mdadm --create /dev/md4 --level=1 --raid-devices=2 /dev/nvme0n1p5 --write-mostly /dev/sdb8 The script: -------------------------------8<------------------------ #!/bin/bash mdadm /dev/md4 --fail /dev/nvme0n1p5 sleep 1 mdadm /dev/md4 --remove failed sleep 1 # fill with random data shred -n1 -v /dev/md4 # fill with zeros shred -n0 -zv /dev/nvme0n1p5 sha256sum /dev/md4 echo 1 > /proc/sys/vm/drop_caches date # calculate a shasum while the array is being synced ( sha256sum /dev/md4; date ) & mdadm /dev/md4 --add --readwrite /dev/nvme0n1p5 date -------------------------------8<------------------------ The two shasums should be equal, but they were different in my tests on affected kernels. Also, in my tests with the script, *without* a write-mostly device in the array, the problems did not happen. Greetings, Mateusz
