On Thu, Oct 25, 2018 at 03:47:39PM +0200, Jack Wang wrote:
Chris Dunlop <chris@xxxxxxxxxxxx> 于2018年10月25日周四 下午2:52写道：
Hi,
kernel: v4.18.16
mdadm: current HEAD, 5d518de
Hi,
Possible related to
https://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git/commit/?h=for-next&id=059421e041eb461fb2b3e81c9adaec18ef03ca3c
Can you try to apply the patch to your kernel to see if it fixes your problem?
Yep, that fixes it. Survived 50+ rounds of my reproducer which hadn't
survived 10 rounds without this patch.
Thanks,
Chris
Regards,
Jack
It looks like there's some issue between scrubbing and writing to
suspend_lo and suspend_hi, e.g. an inverted lock or missed wakeup etc.
I had an md lockup on a production machine when running a scrub and
raid6check on the same md device at the same time. I eventually had to
reset the box to recover.
At the point of the hang md1_raid6 was chewing up a lot of cpu but not
making any progress per /sys/block/md1/md/sync_completed), and the
raid6check was unkillable (kill -9 didn't work) and lsof showed it had
/sys/devices/virtual/block/md1/md/suspend_lo open for write.
The raid6check code writes to suspend_lo and suspend_hi in it's
lock_stripe() routine, to lock each stripe in turn as it works it's way
through the md device.
I'm able to reproduce the lockup on a debian9 single cpu kvm virtual
machine in 2-10 rounds of the reproducer below.
The reproducer prints dots at intervals on the order of a few seconds. If
the problem is hit, the dots stop coming. At that point the shell should
have suspend_lo or suspend_hi open for write, and will be unkillable.
Cheers,
Chris
----------------------------------------------------------------------
#
# Setup
#
# Create 6 x 11-dev raid6, wait for sync to finish
#
function test_setup
{
for md in md{1..6}; do
echo "creating ${md}"
for i in {1..11}; do
f=/var/tmp/${md}-vdev${i}
truncate -s 2G "${f}"
loop[$i]=$(losetup -f)
losetup "${loop[$i]}" "${f}"
done
mdadm --create "/dev/${md}" --level=6 --raid-disks=11 "${loop[@]}"
done
while grep resync /proc/mdstat; do sleep 2; done
cat /proc/mdstat
}
#
# Reproducer
#
# Continuous scrub of all mds, and lock successive stripes of md1 per
# raid6check:lock_stripe()
#
function test_run
{
declare -i component_size=$(($(</sys/block/md1/md/component_size) * 1024)) # KB to bytes
declare -i chunk_size=$(</sys/block/md1/md/chunk_size)
declare -i stripes=$((component_size / chunk_size))
declare -i data_disks=$(($(</sys/block/md1/md/raid_disks) - 2))
declare -i i=0 j stripe
while : ; do
i=$((i + 1))
date +"%F-%T Round $i"
#
# Start scrub on all mds
#
for md in md{1..6}; do
echo check > "/sys/block/${md}/md/sync_action"
done
sleep 2
#
# keep writing to md1 suspend_{lo,hi} as raid6check does
#
j=0
while grep -q check /proc/mdstat; do
j=$((j + 1))
echo -e " $j \c"
stripe=0
while [[ stripe -le stripes ]] ; do
[[ $((stripe % 10)) -eq 0 ]] && echo -e '.\c'
echo $((stripe * chunk_size * data_disks)) > /sys/devices/virtual/block/md1/md/suspend_lo
echo $(((stripe + 1) * chunk_size * data_disks)) > /sys/devices/virtual/block/md1/md/suspend_hi
sleep 0.2
stripe+=1
done
echo
done
done
}
----------------------------------------------------------------------
