antlists <antlists@xxxxxxxxxxxxxxx> writes:
> On 02/08/2021 23:20, Stephan Böttcher wrote:
>> Wol writes:
>>
>> Yes, and it issued a few hundred messages of sector numbers with
>> mismatches in the parity, like: sectors 1460408-1460416.
>
> Bear in mind I'm now getting out of my depth, I just edit the wiki, I
> don't know the deep internals :-), but something doesn't feel right
> here. 408-416 is *nine* sectors. I'd expect it to be a multiple of 2,
> or related to the number of data drives (in your case 4). So 9 is well
> weird.
from drivers/md/raid5.c:
atomic64_add(STRIPE_SECTORS, &conf->mddev->resync_mismatches);
if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery)) {
/* don't try to repair!! */
set_bit(STRIPE_INSYNC, &sh->state);
pr_warn_ratelimited("%s: mismatch sector in range "
"%llu-%llu\n", mdname(conf->mddev),
(unsigned long long) sh->sector,
(unsigned long long) sh->sector +
STRIPE_SECTORS);
The check reads STRIPE_SECTORS=8 sectors from each drive at drive
location sh->sectors=…408, checks the parity, since one drive is missing
it cannot find out on which drive the error is and issues this warning,
reporting sectors […408, …416).
>> These are block numbers I calculated that may match those sector
>> numbers, using the awk script in my first mail.
>> The ext4 block size is 4kByte.
>> The chunk size is 512kByte = 1024 sectors = 128 blocks. Is that per
>> drive or total? I assume per drive.
>
> I would think so. You have 6 chunks per stripe, d1, d2, d3, d4, P and
> Q, so each stripe is 2MB.
>> The sector numbers are per drive. Sector 1460408 is in chunk
>> 1460408/1024 = 1426, sector offset 1460408 % 1024 = 184.
>
> ??? If it's scanning md8, then the sector numbers will be relative to
> md8.
why?
> Stuff working at this level is merely scanning a block device - it has
> no clue whether it's a drive - it may well not be!
?
>> My RAID6 with six drives has a payload of 4 × 512kByte per chunk
>> slice.
>> That is 512 blocks per chunk. The first block of the affected chunk
>> should be 1426 × 512 = 730112.
>> The sector offset 184 into the chunk is 184/8 = 23 blocks in, i.e.,
>> block
>> 730112 + 23 = 730135. And the corresponding sectors in the other drives
>> are multiples of the chunk size ahead, so I need to look for blocks
>> 730112 + 23 + i×128, i=0…3.
>> So when I ask debugfs which inode uses the block 730135, I should
>> get
>> (one of) the file(s) that is affected by the mismatch.
>
> If this is a DISK block, then debugfs will be getting VERY confused,
> because it will be looking for the filesystem block 73015, which will
> be somewhere else entirely.
- I took the sector number reported by the raid driver, and calculated
which block numbers on /dev/md8 those correspond to.
sector 1460408 -> block 730135
- I asked debugfs (ext2progs) which inodes uses those blocks.
- I asked debugfs which file paths point to those innodes.
- I asked debugfs to dump those files to a filesystem on another host.
The `debugfs dump` I repeated five times, once with each remaining drive
omitted from the --readonly assembly with four drives.
A few files were always the same. Did I make a mistake in my
calculations, did I miss some offset?
Some files were different for all six dumps. Probably because of errors
in multiple chunks.
Most files have dumps that agree for some subset. When that happened,
the "full" five drive dump was always among the set.
I put in a new drive, the array is now complete with six drives. I will
have a look how the files compare now, and for those few that may be
valuable I may be able to fix something picking from the six dumps I made.
Gruß,
--
Stephan
