On Tue, 25 Oct 2011 18:34:57 +1100 linbloke <linbloke@xxxxxxxxxxx> wrote:
> On 16/08/11 9:55 AM, NeilBrown wrote:
> > On Mon, 15 Aug 2011 19:32:04 -0400 Jérôme Poulin<jeromepoulin@xxxxxxxxx>
> > wrote:
> >
> >> On Mon, Aug 15, 2011 at 6:42 PM, NeilBrown<neilb@xxxxxxx> wrote:
> >>> So if there are 3 drives A, X, Y where A is permanent and X and Y are rotated
> >>> off-site, then I create two RAID1s like this:
> >>>
> >>>
> >>> mdadm -C /dev/md0 -l1 -n2 --bitmap=internal /dev/A /dev/X
> >>> mdadm -C /dev/md1 -l1 -n2 --bitmap=internal /dev/md0 /dev/Y
> >> That seems nice for 2 disks, but adding another one later would be a
> >> mess. Is there any way to play with slots number manually to make it
> >> appear as an always degraded RAID ? I can't plug all the disks at once
> >> because of the maximum of 2 ports.
> > Yes, add another one later would be difficult. But if you know up-front that
> > you will want three off-site devices it is easy.
> >
> > You could
> >
> > mdadm -C /dev/md0 -l1 -n2 -b internal /dev/A missing
> > mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 missing
> > mdadm -C /dev/md2 -l1 -n2 -b internal /dev/md1 missing
> > mdadm -C /dev/md3 -l1 -n2 -b internal /dev/md2 missing
> >
> > mkfs /dev/md3 ; mount ..
> >
> > So you now have 4 "missing" devices. Each time you plug in a device that
> > hasn't been in an array before, explicitly add it to the array that you want
> > it to be a part of and let it recover.
> > When you plug in a device that was previously plugged in, just "mdadm
> > -I /dev/XX" and it will automatically be added and recover based on the
> > bitmap.
> >
> > You can have as many or as few of the transient drives plugged in at any
> > time as you like.
> >
> > There is a cost here of course. Every write potentially needs to update
> > every bitmap, so the more bitmaps, the more overhead in updating them. So
> > don't create more than you need.
> >
> > Also, it doesn't have to be a linear stack. It could be a binary tree
> > though that might take a little more care to construct. Then when an
> > adjacent pair of leafs are both off-site, their bitmap would not need
> > updating.
> >
> > NeilBrown
>
> Hi Neil, Jérôme and Pavel,
>
> I'm in the process of testing the solution described above and have been
> successful at those steps (I now have sync'd devices that I have failed
> and removed from their respective arrays - the "backups"). I can add new
> devices and also incrementally re-add the devices back to their
> respective arrays and all my tests show this process works well. The
> point which I'm now trying to resolve is how to create a new array from
> one of the off-site components - ie, the restore from backup test.
> Below are the steps I've taken to implement and verify each step, you
> can skip to the bottom section "Restore from off-site backup" to get to
> the point if you like. When the wiki is back up, I'll post this process
> there for others who are looking for mdadm based offline backups. Any
> corrections gratefully appreciated.
>
> Based on the example above, for a target setup of 7 off-site devices
> synced to a two device RAID1, my test setup for a is:
>
> RAID Array Online Device Off-site device
> md100 sdc sdd
> md101 md100 sde
> md102 md101 sdf
> md103 md102 sdg
> md104 md103 sdh
> md105 md104 sdi
> md106 md105 sdj
>
> root@deb6dev:~# uname -a
> Linux deb6dev 2.6.32-5-686 #1 SMP Tue Mar 8 21:36:00 UTC 2011 i686 GNU/Linux
> root@deb6dev:~# mdadm -V
> mdadm - v3.1.4 - 31st August 2010
> root@deb6dev:~# cat /etc/debian_version
> 6.0.1
>
> Create the nested arrays
> ---------------------
> root@deb6dev:~# mdadm -C /dev/md100 -l1 -n2 -b internal -e 1.2 /dev/sdc
> missing
> mdadm: array /dev/md100 started.
> root@deb6dev:~# mdadm -C /dev/md101 -l1 -n2 -b internal -e 1.2
> /dev/md100 missing
> mdadm: array /dev/md101 started.
> root@deb6dev:~# mdadm -C /dev/md102 -l1 -n2 -b internal -e 1.2
> /dev/md101 missing
> mdadm: array /dev/md102 started.
> root@deb6dev:~# mdadm -C /dev/md103 -l1 -n2 -b internal -e 1.2
> /dev/md102 missing
> mdadm: array /dev/md103 started.
> root@deb6dev:~# mdadm -C /dev/md104 -l1 -n2 -b internal -e 1.2
> /dev/md103 missing
> mdadm: array /dev/md104 started.
> root@deb6dev:~# mdadm -C /dev/md105 -l1 -n2 -b internal -e 1.2
> /dev/md104 missing
> mdadm: array /dev/md105 started.
> root@deb6dev:~# mdadm -C /dev/md106 -l1 -n2 -b internal -e 1.2
> /dev/md105 missing
> mdadm: array /dev/md106 started.
>
> root@deb6dev:~# cat /proc/mdstat
> Personalities : [raid1]
> md106 : active (auto-read-only) raid1 md105[0]
> 51116 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md105 : active raid1 md104[0]
> 51128 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md104 : active raid1 md103[0]
> 51140 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md103 : active raid1 md102[0]
> 51152 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md102 : active raid1 md101[0]
> 51164 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md101 : active raid1 md100[0]
> 51176 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md100 : active raid1 sdc[0]
> 51188 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> unused devices: <none>
>
> Create and mount a filesystem
> --------------------------
> root@deb6dev:~# mkfs.ext3 /dev/md106
> <<successful mkfs output snipped>>
> root@deb6dev:~# mount -t ext3 /dev/md106 /mnt/backup
> root@deb6dev:~# df | grep backup
> /dev/md106 49490 4923 42012 11% /mnt/backup
>
> Plug in a device that hasn't been in an array before
> -------------------------------------------
> root@deb6dev:~# mdadm -vv /dev/md100 --add /dev/sdd
> mdadm: added /dev/sdd
> root@deb6dev:~# cat /proc/mdstat
> Personalities : [raid1]
> md106 : active raid1 md105[0]
> 51116 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md105 : active raid1 md104[0]
> 51128 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md104 : active raid1 md103[0]
> 51140 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md103 : active raid1 md102[0]
> 51152 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md102 : active raid1 md101[0]
> 51164 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md101 : active raid1 md100[0]
> 51176 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> md100 : active raid1 sdd[2] sdc[0]
> 51188 blocks super 1.2 [2/2] [UU]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
>
> Write to the array
> ---------------
> root@deb6dev:~# dd if=/dev/urandom of=a.blob bs=1M count=20
> 20+0 records in
> 20+0 records out
> 20971520 bytes (21 MB) copied, 5.05528 s, 4.1 MB/s
> root@deb6dev:~# dd if=/dev/urandom of=b.blob bs=1M count=10
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 2.59361 s, 4.0 MB/s
> root@deb6dev:~# dd if=/dev/urandom of=c.blob bs=1M count=5
> 5+0 records in
> 5+0 records out
> 5242880 bytes (5.2 MB) copied, 1.35619 s, 3.9 MB/s
> root@deb6dev:~# md5sum *blob > md5sums.txt
> root@deb6dev:~# ls -l
> total 35844
> -rw-r--r-- 1 root root 20971520 Oct 25 15:57 a.blob
> -rw-r--r-- 1 root root 10485760 Oct 25 15:57 b.blob
> -rw-r--r-- 1 root root 5242880 Oct 25 15:57 c.blob
> -rw-r--r-- 1 root root 123 Oct 25 15:57 md5sums.txt
> root@deb6dev:~# cp *blob /mnt/backup
> root@deb6dev:~# ls -l /mnt/backup
> total 35995
> -rw-r--r-- 1 root root 20971520 Oct 25 15:58 a.blob
> -rw-r--r-- 1 root root 10485760 Oct 25 15:58 b.blob
> -rw-r--r-- 1 root root 5242880 Oct 25 15:58 c.blob
> drwx------ 2 root root 12288 Oct 25 15:27 lost+found
> root@deb6dev:~# df | grep backup
> /dev/md106 49490 40906 6029 88% /mnt/backup
> root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
> md100 : active raid1 sdd[2] sdc[0]
> 51188 blocks super 1.2 [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> Data written and array devices in sync (bitmap 0/1)
>
> Fail and remove device
> -------------------
> root@deb6dev:~# sync
> root@deb6dev:~# mdadm -vv /dev/md100 --fail /dev/sdd --remove /dev/sdd
> mdadm: set /dev/sdd faulty in /dev/md100
> mdadm: hot removed /dev/sdd from /dev/md100
> root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
> md100 : active raid1 sdc[0]
> 51188 blocks super 1.2 [2/1] [U_]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> Device may now be unplugged
>
>
> Write to the array again
> --------------------
> root@deb6dev:~# rm /mnt/backup/b.blob
> root@deb6dev:~# ls -l /mnt/backup
> total 25714
> -rw-r--r-- 1 root root 20971520 Oct 25 15:58 a.blob
> -rw-r--r-- 1 root root 5242880 Oct 25 15:58 c.blob
> drwx------ 2 root root 12288 Oct 25 15:27 lost+found
> root@deb6dev:~# df | grep backup
> /dev/md106 49490 30625 16310 66% /mnt/backup
> root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
> md100 : active raid1 sdc[0]
> 51188 blocks super 1.2 [2/1] [U_]
> bitmap: 1/1 pages [4KB], 65536KB chunk
>
> bitmap 1/1 shows array is not in sync (we know it's due to the writes
> pending for the device we previously failed)
>
> Plug in a device that was previously plugged in
> ----------------------------------------
> root@deb6dev:~# mdadm -vv -I /dev/sdd --run
> mdadm: UUID differs from /dev/md/0.
> mdadm: UUID differs from /dev/md/1.
> mdadm: /dev/sdd attached to /dev/md100 which is already active.
> root@deb6dev:~# sync
> root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
> md100 : active raid1 sdd[2] sdc[0]
> 51188 blocks super 1.2 [2/2] [UU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> Device reconnected [UU] and in sync (bitmap 0/1)
>
> Restore from off-site device
> ------------------------
> Remove device from array
> root@deb6dev:~# mdadm -vv /dev/md100 --fail /dev/sdd --remove /dev/sdd
> mdadm: set /dev/sdd faulty in /dev/md100
> mdadm: hot removed /dev/sdd from /dev/md100
> root@deb6dev:~# mdadm -Ev /dev/sdd
> /dev/sdd:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 4c957fac:d7dbc792:b642daf0:d22e313e
> Name : deb6dev:100 (local to host deb6dev)
> Creation Time : Tue Oct 25 15:22:19 2011
> Raid Level : raid1
> Raid Devices : 2
>
> Avail Dev Size : 102376 (50.00 MiB 52.42 MB)
> Array Size : 102376 (50.00 MiB 52.42 MB)
> Data Offset : 24 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 381f453f:5a97f1f6:bb5098bb:8c071a95
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Tue Oct 25 17:27:53 2011
> Checksum : acbcee5f - correct
> Events : 250
>
>
> Device Role : Active device 1
> Array State : AA ('A' == active, '.' == missing)
>
> Assemble a new array from off-site component:
> root@deb6dev:~# mdadm -vv -A /dev/md200 --run /dev/sdd
> mdadm: looking for devices for /dev/md200
> mdadm: /dev/sdd is identified as a member of /dev/md200, slot 1.
> mdadm: no uptodate device for slot 0 of /dev/md200
> mdadm: added /dev/sdd to /dev/md200 as 1
> mdadm: /dev/md200 has been started with 1 drive (out of 2).
> root@deb6dev:~#
>
> Check file-system on new array
> root@deb6dev:~# fsck.ext3 -f -n /dev/md200
> e2fsck 1.41.12 (17-May-2010)
> fsck.ext3: Superblock invalid, trying backup blocks...
> fsck.ext3: Bad magic number in super-block while trying to open /dev/md200
>
> The superblock could not be read or does not describe a correct ext2
> filesystem. If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
> e2fsck -b 8193 <device>
>
>
> How do I use these devices in a new array?
>
You need to also assemble md201 md202 md203 md204 md205 md206
and the fsck/mount md206
Each of these is made by assembling the single previous md20X array.
mdadm -A /dev/md201 --run /dev/md200
mdadm -A /dev/md202 --run /dev/md201
....
mdadm -A /dev/md206 --run /dev/md205
All the rest of your description looks good!
Thanks,
NeilBrown
Attachment:
signature.asc
Description: PGP signature
