+cc Shaohua and NeilBrown Swapnil Ingle <swapnil.ingle@xxxxxxxxxxxxxxxx> 于2018年8月9日周四 下午4:20写道： > > Hi, > > For RAID1 with 2 slave devices underneath, we see different data on the slaves. > The IO is generated by using FIO with ioengine=libaio. > > Following script sometimes leads to have different md5sum on the slave devices. > > ================== > modprobe -r brd > modprobe brd rd_nr=2 rd_size=2097152 > dd if=/dev/zero of=/dev/ram1 oflag=direct || true > dd if=/dev/zero of=/dev/ram0 oflag=direct || true > dd if=/dev/ram1 | md5sum > dd if=/dev/ram0 | md5sum > mdadm --create md127 -f -n 2 -l raid1 /dev/ram0 /dev/ram1 <<EOF > yes > EOF > sleep 10 > cat /proc/mdstat > mkfs.ext4 /dev/md127 > mount /dev/md127 /tmp > touch /tmp/bla.txt > > echo -n " > [global] > rw=write > direct=1 > size=1G > ioengine=libaio > iodepth=128 > iodepth_batch_submit=128 > iodepth_batch_complete=128 > numjobs=4 > group_reporting > [job0] > filename=/tmp/bla.txt > [job1] > filename=/tmp/bla.txt > [job2] > filename=/tmp/bla.txt > [job3] > filename=/tmp/bla.txt > " > fio.ini > > cat fio.ini > > fio fio.ini > > ls -lah /tmp/ > > umount /tmp > > mdadm --stop md127 > > dd if=/dev/ram1 obs=1M ibs=1M skip=4 | md5sum > dd if=/dev/ram0 obs=1M ibs=1M skip=4 | md5sum > ========================== > > Is this expected behavior with async IO's? > > Thanks, > -Swapnil > -- It's pretty easy to reproduce the problem. I looks quite clear, the IO to both legs get reordered, which lead to both legs has different data on it. We extends the script a bit to better demonstrate the problem. We've seen it on different kernel version 4.4 and 4.14, I feel latest upstream is the same, but haven't tested it. Could you guys give us a hint, how could we avoid/fix the problem. Thanks, Jack
Attachment:
test_raid1_md5sum.sh
Description: application/shellscript
