Hello, so it worked as we expected and it was most probably caused by that bug. Thanks for the assistance. Patrik 2014-09-22 12:17 GMT+02:00 NeilBrown <neilb@xxxxxxx>: > On Mon, 22 Sep 2014 09:20:35 +0200 Patrik Horník <patrik@xxxxxx> wrote: > >> Well I browsed through the sources of latest mdadm version at night >> instead of sleeping :) and was searching how it got clean flag set to >> 0. And was not sure exactly about that line there and from which >> device it gets clean state. So the bug was that it can get it from not >> current device? It makes sense because first device identified by >> mdadm is old md101 device. > > Correct. > >> >> So will it work then if I use 3.3 and somehow dont give it md101 >> device? By stopping it before -A call or by manually specifying other >> drives? Or you really recommend to build latest version of mdadm? > > mdadm -S /dev/mdXX list-of-devices-that-are-working > > should start the array for you using 3.3. > Then > mdadm /dev/mdXX --re-add /dev/md101 > > will re-add the device and hopefully do a quick bitmap-based rebuild. > >> >> What is expected behaviour with 3.3.1+? Can it be started with all >> devices and it should automatically start to recover md101? If so what >> is best way, to start it without md101 and then use -re-add to add it, >> start it without md101 and use -add or start it also with md101? > > --add should have the same effect as --re-add. > > I think mdadm 3.3.1 will just assemble the array without md101 and you then > have to --re-add that yourself. To get it re-added automatically you needed > to have "policy action=re-add" or similar in mdadm.conf, and then use > mdadm -I /dev/devicename > on each device. That should put the whole array together and re-add anything > that needs it. > > I think. > > NeilBrown > > >> >> Thank you very much. >> >> 2014-09-22 8:56 GMT+02:00 NeilBrown <neilb@xxxxxxx>: >> > On Mon, 22 Sep 2014 08:34:21 +0200 Patrik Horník <patrik@xxxxxx> wrote: >> > >> >> - Well what is exact meaning of --no-degraded then? Because I am using >> >> it also on RAID6 arrays that are missing one drive and mdadm starts >> >> them. I thought until today that it is against assembling for example >> >> RAID6 array with missing more than two drives or to be more precise >> >> array with number of drives it used last time. (I did not look at the >> >> code what does it exactly. It is mdadm 3.3 on Debian.) >> > >> > Sorry, I confused myself. >> > "--no-degraded" means "Only start the array if all expected devices are >> > present". >> > So if the array "knows" that one device is missing, it will start if all >> > other devices are present. But if it "thinks" that all devices are working, >> > then it will only start if all the devices ar there. >> > >> >> >> >> - Well array was shutdown cleanly manually by mdadm -S. Cant the not >> >> clean classification be result of md101 device between find devices or >> >> result of first two assemble tries? >> > >> > If the state still says "Clean" (which it does, thanks), the mdadm should >> > treat it as 'clean'. >> > >> > I think you are probably hitting the bug fixed by >> > >> > http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=56bbc588f7f0f3bdd3ec23f02109b427c1d3b8f1 >> > >> > which is in 3.3.1. >> > >> > So a new version of mdadm should fix it. >> > >> > NeilBrown >> > >> > >> > >> >> >> >> - Anyway as I mentioned superblock on all five devices has clean state. Example: >> >> /dev/sdk1: >> >> Magic : XXXXXXX >> >> Version : 1.2 >> >> Feature Map : 0x1 >> >> Array UUID : XXXXXXXXXXXXXXXXXXXXX >> >> Name : >> >> Creation Time : Thu Aug XXXXXXXX >> >> Raid Level : raid6 >> >> Raid Devices : 6 >> >> >> >> Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB) >> >> Array Size : 11720536064 (11177.57 GiB 12001.83 GB) >> >> Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB) >> >> Data Offset : 262144 sectors >> >> Super Offset : 8 sectors >> >> Unused Space : before=262056 sectors, after=911 sectors >> >> State : clean >> >> Device UUID : YYYYYYYYYYYYYYYYYYYY >> >> >> >> Internal Bitmap : 8 sectors from superblock >> >> Update Time : Mon Sep 22 02:23:45 2014 >> >> Bad Block Log : 512 entries available at offset 72 sectors >> >> Checksum : ZZZZZZZZ - correct >> >> Events : EEEEEE >> >> >> >> Layout : left-symmetric >> >> Chunk Size : 512K >> >> >> >> Device Role : Active device 4 >> >> Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing) >> >> >> >> - md101 has Events count lower by 16 than others devices. >> >> >> >> - Please I need little more assurance what is exact state of array and >> >> explain why it is behaving as it is behaving, so I can be sure what >> >> steps are needed and what happens. The data on array is important. >> >> Patrik Horník >> >> šéfredaktor www.DSL.sk >> >> Tel.: +421 905 385 666 >> >> Email: patrik@xxxxxx >> >> >> >> >> >> 2014-09-22 5:19 GMT+02:00 NeilBrown <neilb@xxxxxxx>: >> >> > On Mon, 22 Sep 2014 04:11:20 +0200 Patrik Horník <patrik@xxxxxx> wrote: >> >> > >> >> >> Hello Neil, >> >> >> >> >> >> I've got this situation unfamiliar to me on RAID6 array md1 with important data. >> >> >> >> >> >> - It is RAID6 with 6 devices, 5 are partitions and 1 is another RAID0 >> >> >> array md101 from two smaller drives. One of the smaller drives froze, >> >> >> so md101 got kicked out from md1 and marked as faulty in md1. After >> >> >> while I've stopped md1 without removing md101 from it first. Then I >> >> >> rebooted and assembled md101. >> >> >> >> >> >> - First I tried mdadm -A --no-degraded -u UUID /dev/md1 but got >> >> >> "mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started." >> >> >> so I stopped the md1. >> >> >> >> >> >> - Second time I started it with -v and got: >> >> >> >> >> >> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5. >> >> >> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4. >> >> >> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1. >> >> >> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2. >> >> >> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0. >> >> >> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3. >> >> >> mdadm: added /dev/sdi1 to /dev/md1 as 1 >> >> >> mdadm: added /dev/sdh1 to /dev/md1 as 2 >> >> >> mdadm: added /dev/sde1 to /dev/md1 as 3 >> >> >> mdadm: added /dev/sdk1 to /dev/md1 as 4 >> >> >> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date) >> >> >> mdadm: added /dev/sdg1 to /dev/md1 as 0 >> >> >> mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started. >> >> >> >> >> >> - On third time I tried without --nodegraded with mdadm -A -v -u UUID >> >> >> /dev/md1. This is what I've got: >> >> >> >> >> >> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5. >> >> >> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4. >> >> >> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1. >> >> >> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2. >> >> >> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0. >> >> >> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3. >> >> >> mdadm: added /dev/sdi1 to /dev/md1 as 1 >> >> >> mdadm: added /dev/sdh1 to /dev/md1 as 2 >> >> >> mdadm: added /dev/sde1 to /dev/md1 as 3 >> >> >> mdadm: added /dev/sdk1 to /dev/md1 as 4 >> >> >> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date) >> >> >> mdadm: added /dev/sdg1 to /dev/md1 as 0 >> >> >> mdadm: /dev/md1 assembled from 5 drives - not enough to start the >> >> >> array while not clean - consider --force. >> >> >> >> >> >> Array md1 has bitmap. All drive devices have all same Events, their >> >> >> state is clean and Device Role is Active device. md101 has active >> >> >> state and lower Events. >> >> >> >> >> >> Is this expected behavior? My theory is that it is caused by md101 and >> >> >> I should start array md1 without it (by for example stopping md101) >> >> >> and then re-add it. Is that a case or is it something else? >> >> >> >> >> >> Thanks. >> >> >> >> >> >> Best regards, >> >> >> >> >> >> Patrik >> >> > >> >> > >> >> > The array is clearly degraded as one of the devices failed and hasn't been >> >> > recovered yet, so using --nodegraded is counter productive, as you >> >> > discovered. >> >> > >> >> > It appears that the array is also marked as 'dirty'. That suggests that it >> >> > wasn't shut down cleanly. >> >> > What does "mdadm --examine" of some device show? >> >> > >> >> > You probably need to re-assemble the array with --force like it suggests, >> >> > then add the failed device and let it recover. >> >> > >> >> > NeilBrown >> >> > >> >> -- >> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
