Hi,
Recently a data integrity issue about skip_copy was found. We are able
to reproduce it and found the root cause. This data integrity issue
might happen if there are other layers between file system and raid5.
[How to Reproduce]
1. Create a raid5 named md0 first (with skip_copy enable), and wait md0
resync done which ensures that all data and parity are synchronized
2. Use lvm tools to create a logical volume named lv-md0 over md0
3. Format an ext4 file system on lv-md0 and mount on /mnt
4. Do some db operations (e.g. sqlite insert) to write data through /mnt
5. When those db operations finished, we do the following command
"echo check > /sys/block/md0/md/sync_action" to check mismatch_cnt,
it is very likely that we got mismatch_cnt != 0 when check finished
[Root Cause]
After tracing code and more experiments, it is more proper to say that
it's a problem about backing_dev_info (bdi) instead of a bug about
skip_copy.
We notice that:
1. skip_copy counts on BDI_CAP_STABLE_WRITES to ensure that bio's
page
will not be modified before raid5 completes I/O. Thus we can skip
copy
page from bio to stripe cache
2. The ext4 file system will call wait_for_stable_page() to ask
whether
the mapped bdi requires stable writes
Data integrity happens because:
1. When raid5 enable skip_copy, it will only set it's own bdi
required
BDI_CAP_STABLE_WRITES, but this information will not propagate to
other bdi between file system and md
2. When ext4 file system check stable writes requirement by calling
wait_for_stable_page(), it can only see the underlying bdi's
capability
and cannot see all related bdi
Thus, skip_copy works fine if we format file system directly on md.
But data integrity issue happens if there are some other block layers
(e.g. dm)
between file system and md.
[Result]
We do more tests on different storage stacks, here are the results.
The following settings can pass the test thousand times:
1. raid5 with skip_copy enable + ext4
2. raid5 with skip_copy disable + ext4
3. raid5 with skip_copy disable + lvm + ext4
The following setting will fail the test in 10 rounds:
1. raid5 with skip_copy enable + lvm + ext4
I think the solution might be let all bdi can communicate through
different block layers,
then we can pass BDI_CAP_STABLE_WRITES information if we enable
skip_copy.
But the current bdi structure is not allowed us to do that.
What would you suggest to do if we want to make skip_copy more reliable
?
Best Regards,
Alex
