Hi,
On 3/5/23 05:36, Benjamin Sonntag wrote:
Hi everyone,
I think we found a bug in the mdadm code here at Octopuce.
Probably something wrong inside md raid.
I'm reporting it here, please tell me if that's not the right place to report it, or if you need any other information.
This bug "hangs" processes in the Device-busy (D) state forever, until we reboot. It has been tested on both a debian 5.10 an 6.0 Linux kernel
Do you mean it happened on both kernel versions? Could you share
relevant stacks by "cat /proc/${pid of D state process}/stack''?
How to trigger the bug:
- create a raid5 or raid6 block device using mdadm
mdadm --create /dev/md0 -l 5 -n 3 /dev/sd{a,b,c}2
- create a partition on it and mount it USING DISCARD/TRIM (important) (the underlying device must support trim)
mkfs.ext4 /dev/md0
mount /dev/md0 /mnt -o discard
- create a few files
for i in $( seq 1 1000 ) ; do dd if=/dev/zero of=/mnt/$i bs=10M count=1 ; done
- expand the raid by adding a new drive
mdadm --add /dev/md0 /dev/sdd2
mdadm --grow /dev/md0 -n 4
- the last command will start a "reshape" operation on md0
- DURING THE RESHAPE (it's important) erase some file (it goes fine)
rm /mnt/* -rf
- THEN, still during the reshape (important) try to sync or fsync
sync
- the sync process get stuck in the D state. no way to kill it until reboot
(in fact, any process that does sync during the reshape after some files were deleted will get stuck, such as mariadbd or rsyslog...)
- If you don't mount with discard your partition, the 'sync' works properly
An easy way to circumvent this problem:
- before reshaping, remount without discard
mount /mnt -o remount,nodiscard
- after the reshaping ends, remount with discard
mount /mnt -o remount,discard
We don't really know how to start searching for a solution, since it requires knowing the internal of MD & Discard pretty well :/ (and I'm definitely not a kernel coder ;) )
thanks for your help on this issue,
Assume reshape + discard works with previous kernel version, maybe
you can try to bisect kernel tree to see which commit might caused
the bug.
Thanks,
Guoqing
