Hello Neil,
from the patch it looks like for raid levels with more than a single
redundancy, like 3-way raid1 or raid6, when there is an additional
missing drive, the bits will still not be cleared, correct?
This seems to be protected by !bitmap->mddev->degraded part. Because
these bits are still needed to rebuild future drive(s)?
Thanks,
Alex.
On Fri, Dec 23, 2011 at 12:48 AM, NeilBrown <neilb@xxxxxxx> wrote:
> On Wed, 31 Aug 2011 13:23:01 -0500 (CDT) Chris Pearson
> <pearson.christopher.j@xxxxxxxxx> wrote:
>
>> I'm happy to apply a patch to whichever kernel you like, but the blocks have since cleared, so I will try and reproduce it first.
>
> I have finally identified the problem here. I was looking into a different
> but related problem and saw what was happening. I don't know what I didn't
> notice it before.
>
> You can easily reproduce the problem by writing to an array with a bitmap
> while a spare is recovering. Any bits that get set in the section that has
> already been recovered will stay set.
>
> This patch fixes it and will - with luck - be in 3.2.
>
> Thanks,
> NeilBrown
>
> From b9664495d2a884fbf7195e1abe4778cc6c3ae9b7 Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@xxxxxxx>
> Date: Fri, 23 Dec 2011 09:42:52 +1100
> Subject: [PATCH] md/bitmap: It is OK to clear bits during recovery.
>
> commit d0a4bb492772ce5c4bdfba3744a99ed6f6fb238f introduced a
> regression which is annoying but fairly harmless.
>
> When writing to an array that is undergoing recovery (a spare
> in being integrated into the array), writing to the array will
> set bits in the bitmap, but they will not be cleared when the
> write completes.
>
> For bits covering areas that have not been recovered yet this is not a
> problem as the recovery will clear the bits. However bits set in
> already-recovered region will stay set and never be cleared.
> This doesn't risk data integrity. The only negatives are:
> - next time there is a crash, more resyncing than necessary will
> be done.
> - the bitmap doesn't look clean, which is confusing.
>
> While an array is recovering we don't want to update the
> 'events_cleared' setting in the bitmap but we do still want to clear
> bits that have very recently been set - providing they were written to
> the recovering device.
>
> So split those two needs - which previously both depended on 'success'
> and always clear the bit of the write went to all devices.
>
> Signed-off-by: NeilBrown <neilb@xxxxxxx>
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index b690711..6d03774 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -1393,9 +1393,6 @@ void bitmap_endwrite(struct bitmap *bitmap, sector_t offset, unsigned long secto
> atomic_read(&bitmap->behind_writes),
> bitmap->mddev->bitmap_info.max_write_behind);
> }
> - if (bitmap->mddev->degraded)
> - /* Never clear bits or update events_cleared when degraded */
> - success = 0;
>
> while (sectors) {
> sector_t blocks;
> @@ -1409,7 +1406,7 @@ void bitmap_endwrite(struct bitmap *bitmap, sector_t offset, unsigned long secto
> return;
> }
>
> - if (success &&
> + if (success && !bitmap->mddev->degraded &&
> bitmap->events_cleared < bitmap->mddev->events) {
> bitmap->events_cleared = bitmap->mddev->events;
> bitmap->need_sync = 1;
>
>
>
>>
>> On Wed, 31 Aug 2011, NeilBrown wrote:
>>
>> >Date: Wed, 31 Aug 2011 17:38:42 +1000
>> >From: NeilBrown <neilb@xxxxxxx>
>> >To: Chris Pearson <kermit4@xxxxxxxxx>
>> >Cc: linux-raid@xxxxxxxxxxxxxxx
>> >Subject: Re: dirty chunks on bitmap not clearing (RAID1)
>> >
>> >On Mon, 29 Aug 2011 11:30:56 -0500 Chris Pearson <kermit4@xxxxxxxxx> wrote:
>> >
>> >> I have the same problem. 3 chunks are always dirty.
>> >>
>> >> I'm using 2.6.38-8-generic and mdadm - v3.1.4 - 31st August 2010
>> >>
>> >> If that's not normal, then maybe what I've done differently is that I
>> >> created the array, raid 1, with one live and one missing disk, then
>> >> added the second one later after writing a lot of data.
>> >>
>> >> Also, though probably not the cause, I continued writing data while it
>> >> was syncing, and a couple times during the syncing, both drives
>> >> stopped responding and I had to power off.
>> >>
>> >> # cat /proc/mdstat
>> >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> >> [raid4] [raid10]
>> >> md127 : active raid1 sdd1[0] sdc1[2]
>> >> 1904568184 blocks super 1.2 [2/2] [UU]
>> >> bitmap: 3/15 pages [12KB], 65536KB chunk
>> >>
>> >> unused devices: <none>
>> >>
>> >> # mdadm -X /dev/sd[dc]1
>> >> Filename : /dev/sdc1
>> >> Magic : 6d746962
>> >> Version : 4
>> >> UUID : 43761dc5:4383cf0f:41ef2dab:43e6d74e
>> >> Events : 40013
>> >> Events Cleared : 40013
>> >> State : OK
>> >> Chunksize : 64 MB
>> >> Daemon : 5s flush period
>> >> Write Mode : Allow write behind, max 256
>> >> Sync Size : 1904568184 (1816.34 GiB 1950.28 GB)
>> >> Bitmap : 29062 bits (chunks), 3 dirty (0.0%)
>> >> Filename : /dev/sdd1
>> >> Magic : 6d746962
>> >> Version : 4
>> >> UUID : 43761dc5:4383cf0f:41ef2dab:43e6d74e
>> >> Events : 40013
>> >> Events Cleared : 40013
>> >> State : OK
>> >> Chunksize : 64 MB
>> >> Daemon : 5s flush period
>> >> Write Mode : Allow write behind, max 256
>> >> Sync Size : 1904568184 (1816.34 GiB 1950.28 GB)
>> >> Bitmap : 29062 bits (chunks), 3 dirty (0.0%)
>> >
>> >I cannot see how this would be happening. If any bits are set, then they
>> >will be cleared after 5 seconds, and then 5 seconds later the block holding
>> >the bits will be written out so that they will appear on disk to be cleared.
>> >
>> >I assume that if you write to the array, the 'dirty' count increases, but
>> >always goes back to three?
>> >
>> >And if you stop the array and start it again, the '3' stays there?
>> >
>> >If I sent you a patch to add some tracing information would you be able to
>> >compile a new kernel with that patch applied and see what it says?
>> >
>> >Thanks,
>> >
>> >NeilBrown
>> >
>> >
>> >>
>> >>
>> >> Quoting NeilBrown <neilb@xxxxxxx>:
>> >>
>> >> > On Thu, October 15, 2009 9:39 am, aristizb@xxxxxxxxxxx wrote:
>> >> >> Hello,
>> >> >>
>> >> >> I have a RAID1 with 2 LVM disks and I am running into a strange
>> >> >> situation where having the 2 disks connected to the array the bitmap
>> >> >> never clears the dirty chunks.
>> >> >
>> >> > That shouldn't happen...
>> >> > What versions of mdadm and the Linux kernel are you using?
>> >> >
>> >> > NeilBrown
>> >> >
>> >> >>
>> >> >> I am assuming also that when a RAID1 is in write-through mode, the
>> >> >> bitmap indicates that all the data has made it to all the disks if
>> >> >> there are no dirty chunks using mdadm --examine-bitmap.
>> >> >>
>> >> >> The output of cat /proc/mdstat is:
>> >> >>
>> >> >> md2060 : active raid1 dm-5[1] dm-6[0]
>> >> >> 2252736 blocks [2/2] [UU]
>> >> >> bitmap: 1/275 pages [12KB], 4KB chunk, file: /tmp/md2060bm
>> >> >>
>> >> >>
>> >> >> The output of mdadm --examine-bitmap /tmp/md2060bm is:
>> >> >>
>> >> >> Filename : md2060bm
>> >> >> Magic : 6d746962
>> >> >> Version : 4
>> >> >> UUID : ad5fb74c:bb1c654a:087b2595:8a5d04a9
>> >> >> Events : 12
>> >> >> Events Cleared : 12
>> >> >> State : OK
>> >> >> Chunksize : 4 KB
>> >> >> Daemon : 5s flush period
>> >> >> Write Mode : Normal
>> >> >> Sync Size : 2252736 (2.15 GiB 2.31 GB)
>> >> >> Bitmap : 563184 bits (chunks), 3 dirty (0.0%)
>> >> >>
>> >> >>
>> >> >> Having the array under no IO, I waited 30 minutes but the dirty data
>> >> >> never gets clear from the bitmap, so I presume the disks are not in
>> >> >> sync; but after I ran a block by block comparison of the two devices I
>> >> >> found that they are equal.
>> >> >>
>> >> >> The superblocks and the external bitmap tell me that all the events
>> >> >> are cleared, so I am confused on why the bitmap never goes to 0 dirty
>> >> >> chunks.
>> >> >>
>> >> >> How can I tell if the disks are in sync?
>> >> >>
>> >> >>
>> >> >> Thank you in advance for any help
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >
>> >
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
