On Fri, Aug 10, 2018 at 01:31:18PM +0200, Jack Wang wrote: > +cc Shaohua and NeilBrown > Swapnil Ingle <swapnil.ingle@xxxxxxxxxxxxxxxx> 于2018年8月9日周四 下午4:20写道： > > > > Hi, > > > > For RAID1 with 2 slave devices underneath, we see different data on the slaves. > > The IO is generated by using FIO with ioengine=libaio. > > > > Following script sometimes leads to have different md5sum on the slave devices. > > > > ================== > > modprobe -r brd > > modprobe brd rd_nr=2 rd_size=2097152 > > dd if=/dev/zero of=/dev/ram1 oflag=direct || true > > dd if=/dev/zero of=/dev/ram0 oflag=direct || true > > dd if=/dev/ram1 | md5sum > > dd if=/dev/ram0 | md5sum > > mdadm --create md127 -f -n 2 -l raid1 /dev/ram0 /dev/ram1 <<EOF > > yes > > EOF > > sleep 10 > > cat /proc/mdstat > > mkfs.ext4 /dev/md127 > > mount /dev/md127 /tmp > > touch /tmp/bla.txt > > > > echo -n " > > [global] > > rw=write > > direct=1 > > size=1G > > ioengine=libaio > > iodepth=128 > > iodepth_batch_submit=128 > > iodepth_batch_complete=128 > > numjobs=4 > > group_reporting > > [job0] > > filename=/tmp/bla.txt > > [job1] > > filename=/tmp/bla.txt > > [job2] > > filename=/tmp/bla.txt > > [job3] > > filename=/tmp/bla.txt > > " > fio.ini > > > > cat fio.ini > > > > fio fio.ini > > > > ls -lah /tmp/ > > > > umount /tmp > > > > mdadm --stop md127 > > > > dd if=/dev/ram1 obs=1M ibs=1M skip=4 | md5sum > > dd if=/dev/ram0 obs=1M ibs=1M skip=4 | md5sum > > ========================== > > > > Is this expected behavior with async IO's? > > > > Thanks, > > -Swapnil > > -- > It's pretty easy to reproduce the problem. > I looks quite clear, the IO to both legs get reordered, which lead to > both legs has different data on it. > > We extends the script a bit to better demonstrate the problem. > We've seen it on different kernel version 4.4 and 4.14, I feel latest > upstream is the same, but haven't tested it. > > Could you guys give us a hint, how could we avoid/fix the problem. Sorry, it's not clear to me, but are you using "md5sum" on the _whole_ RAID-1 components? If this is the case, I think these will always be different. Try to check "md5sum" on the newly created array, before anything else. I guess they'll be different. The point is the "md" headers will be already different, since the UUID is different between the two components. Furthermore, depending on the filesystem, it could be inconsistencies are created, but these should be only in unused array areas. In any case, maybe I got it wrong and it is something completely different. bye, pg > > Thanks, > Jack -- piergiorgio
