On 11/11/2013 19:52, Justin Piszcz wrote:
4- Linux or SSD bug on trim, such as trimming wrong offsets killing live
data
5- MD does not lock regions during check so returns erroneous mismatches for
areas being written. This would be harmless but your mismatches number seems
to high to me for this.
I wonder if this could be it.
It's not, your reboot test confirmed it's not, when you did this:
Ran a check > sync_action and re-checked the mismatch_cnt:
cat /sys/devices/virtual/block/md1/md/mismatch_cnt
68352
this should have been zero if it was the case
I would suggest to investigate further. One idea is to find which files are
affected....
I took a slightly different approach, hopefully this will provide the
information you are looking for:
Actually no, and you "fixed" it so you cannot do any further test until
the number of mismatches grows again
Rebooted to a system rescue cd:
Did not mount the filesystem, before a check:
cat /sys/devices/virtual/block/md1/md/mismatch_cnt
256
Ran a check > sync_action and re-checked the mismatch_cnt:
cat /sys/devices/virtual/block/md1/md/mismatch_cnt
68352
Ran a repair > sync_action
68352 (expected, need to re-run check):
Ran a check > sync_action
0
It appears when there a files moving around / being written to it can
throw off the mismatch_cnt?
Maybe, and it shouldn't happen. This is a serious bug somewhere, it
corrupts data, we need to find it.
As the FS above was not mounted, it
repaired ok?
No, you just copied over one disk to the other. This does not mean
"fixed" in the filesystem sense. Data is still corrupted, just the two
legs of the RAID are now corrupted identically one to the other.
Wait so that mismatches grow again a couple of thousands, then I suggest
you really do what I wrote in my previous email.
If you can afford to bring the system offline then it's really easy
because you can find all mismatching files in one shot
- wait for mismatch_cnt reach 2000 at least (the more, the better), then
reboot machine with a livecd
- mount RAID
- mount the filesystem readonly
- (very important or it will resync) activate bitmap for raid1,
preferably with small chunksize
- fail 1 drive so to degrade raid1
- drop caches with blockdev --flushbufs on the md device such as
/dev/md2, on the two underlying partitions such as /dev/sd[ab]2, and
maybe even on the two disk holding then such as /dev/sd[ab] (I'm not
really sure what is the minimum needed) ; and also echo 3 >
/proc/sys/vm/drop_caches
- recursive md5sum for all files of the filesystem (something like find
-type f -print0 | xargs -0 md5sum (untested)) > redirect stdout to a
file on another filesystem
- reattach drive with --re-add, let it resync the differences using the
bitmap (there shouldn't be any, should complete immediately)
- fail the other drive
- drop all caches again
- again find | md5sum , redirected to another file on another filesystem
- reattach drive with --re-add
now analyze differences between md5sums. Those are the files which are
different in the two legs of the RAID, and they shouldn't be (aka
corruption).
Find preferably humanly readable text files which are sequentially
written, such as log files. It is more difficult to understand what's
wrong for files changed in the middle such as database files, or binary
files.
Copy those files out, to another filesystem.
You need to, again:
- fail 1 drive so to degrade raid1
- drop caches as described above
- copy all files out, to a directory in another filesystem
- reattach drive with --re-add
- fail the other drive
- drop all caches again
- copy all files out again to another directory of another filesystem
- reattach drive with --re-add
At this point you can restart machine to production.
Inspect the two versions of such files... If you can tell us something
about which files got corrupted and what you exactly see in the
corruption point (you can use hexdump to see binary chars), we could
make some further guesses.
Regards
J.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
