On 16/08/11 9:55 AM, NeilBrown wrote:
On Mon, 15 Aug 2011 19:32:04 -0400 Jérôme Poulin<jeromepoulin@xxxxxxxxx>
wrote:
On Mon, Aug 15, 2011 at 6:42 PM, NeilBrown<neilb@xxxxxxx> wrote:
So if there are 3 drives A, X, Y where A is permanent and X and Y are rotated
off-site, then I create two RAID1s like this:
mdadm -C /dev/md0 -l1 -n2 --bitmap=internal /dev/A /dev/X
mdadm -C /dev/md1 -l1 -n2 --bitmap=internal /dev/md0 /dev/Y
That seems nice for 2 disks, but adding another one later would be a
mess. Is there any way to play with slots number manually to make it
appear as an always degraded RAID ? I can't plug all the disks at once
because of the maximum of 2 ports.
Yes, add another one later would be difficult. But if you know up-front that
you will want three off-site devices it is easy.
You could
mdadm -C /dev/md0 -l1 -n2 -b internal /dev/A missing
mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 missing
mdadm -C /dev/md2 -l1 -n2 -b internal /dev/md1 missing
mdadm -C /dev/md3 -l1 -n2 -b internal /dev/md2 missing
mkfs /dev/md3 ; mount ..
So you now have 4 "missing" devices. Each time you plug in a device that
hasn't been in an array before, explicitly add it to the array that you want
it to be a part of and let it recover.
When you plug in a device that was previously plugged in, just "mdadm
-I /dev/XX" and it will automatically be added and recover based on the
bitmap.
You can have as many or as few of the transient drives plugged in at any
time as you like.
There is a cost here of course. Every write potentially needs to update
every bitmap, so the more bitmaps, the more overhead in updating them. So
don't create more than you need.
Also, it doesn't have to be a linear stack. It could be a binary tree
though that might take a little more care to construct. Then when an
adjacent pair of leafs are both off-site, their bitmap would not need
updating.
NeilBrown
Hi Neil, Jérôme and Pavel,
I'm in the process of testing the solution described above and have been
successful at those steps (I now have sync'd devices that I have failed
and removed from their respective arrays - the "backups"). I can add new
devices and also incrementally re-add the devices back to their
respective arrays and all my tests show this process works well. The
point which I'm now trying to resolve is how to create a new array from
one of the off-site components - ie, the restore from backup test.
Below are the steps I've taken to implement and verify each step, you
can skip to the bottom section "Restore from off-site backup" to get to
the point if you like. When the wiki is back up, I'll post this process
there for others who are looking for mdadm based offline backups. Any
corrections gratefully appreciated.
Based on the example above, for a target setup of 7 off-site devices
synced to a two device RAID1, my test setup for a is:
RAID Array Online Device Off-site device
md100 sdc sdd
md101 md100 sde
md102 md101 sdf
md103 md102 sdg
md104 md103 sdh
md105 md104 sdi
md106 md105 sdj
root@deb6dev:~# uname -a
Linux deb6dev 2.6.32-5-686 #1 SMP Tue Mar 8 21:36:00 UTC 2011 i686 GNU/Linux
root@deb6dev:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010
root@deb6dev:~# cat /etc/debian_version
6.0.1
Create the nested arrays
---------------------
root@deb6dev:~# mdadm -C /dev/md100 -l1 -n2 -b internal -e 1.2 /dev/sdc
missing
mdadm: array /dev/md100 started.
root@deb6dev:~# mdadm -C /dev/md101 -l1 -n2 -b internal -e 1.2
/dev/md100 missing
mdadm: array /dev/md101 started.
root@deb6dev:~# mdadm -C /dev/md102 -l1 -n2 -b internal -e 1.2
/dev/md101 missing
mdadm: array /dev/md102 started.
root@deb6dev:~# mdadm -C /dev/md103 -l1 -n2 -b internal -e 1.2
/dev/md102 missing
mdadm: array /dev/md103 started.
root@deb6dev:~# mdadm -C /dev/md104 -l1 -n2 -b internal -e 1.2
/dev/md103 missing
mdadm: array /dev/md104 started.
root@deb6dev:~# mdadm -C /dev/md105 -l1 -n2 -b internal -e 1.2
/dev/md104 missing
mdadm: array /dev/md105 started.
root@deb6dev:~# mdadm -C /dev/md106 -l1 -n2 -b internal -e 1.2
/dev/md105 missing
mdadm: array /dev/md106 started.
root@deb6dev:~# cat /proc/mdstat
Personalities : [raid1]
md106 : active (auto-read-only) raid1 md105[0]
51116 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md105 : active raid1 md104[0]
51128 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md104 : active raid1 md103[0]
51140 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md103 : active raid1 md102[0]
51152 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md102 : active raid1 md101[0]
51164 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md101 : active raid1 md100[0]
51176 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md100 : active raid1 sdc[0]
51188 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>
Create and mount a filesystem
--------------------------
root@deb6dev:~# mkfs.ext3 /dev/md106
<<successful mkfs output snipped>>
root@deb6dev:~# mount -t ext3 /dev/md106 /mnt/backup
root@deb6dev:~# df | grep backup
/dev/md106 49490 4923 42012 11% /mnt/backup
Plug in a device that hasn't been in an array before
-------------------------------------------
root@deb6dev:~# mdadm -vv /dev/md100 --add /dev/sdd
mdadm: added /dev/sdd
root@deb6dev:~# cat /proc/mdstat
Personalities : [raid1]
md106 : active raid1 md105[0]
51116 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md105 : active raid1 md104[0]
51128 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md104 : active raid1 md103[0]
51140 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md103 : active raid1 md102[0]
51152 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md102 : active raid1 md101[0]
51164 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md101 : active raid1 md100[0]
51176 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md100 : active raid1 sdd[2] sdc[0]
51188 blocks super 1.2 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk
Write to the array
---------------
root@deb6dev:~# dd if=/dev/urandom of=a.blob bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 5.05528 s, 4.1 MB/s
root@deb6dev:~# dd if=/dev/urandom of=b.blob bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 2.59361 s, 4.0 MB/s
root@deb6dev:~# dd if=/dev/urandom of=c.blob bs=1M count=5
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 1.35619 s, 3.9 MB/s
root@deb6dev:~# md5sum *blob > md5sums.txt
root@deb6dev:~# ls -l
total 35844
-rw-r--r-- 1 root root 20971520 Oct 25 15:57 a.blob
-rw-r--r-- 1 root root 10485760 Oct 25 15:57 b.blob
-rw-r--r-- 1 root root 5242880 Oct 25 15:57 c.blob
-rw-r--r-- 1 root root 123 Oct 25 15:57 md5sums.txt
root@deb6dev:~# cp *blob /mnt/backup
root@deb6dev:~# ls -l /mnt/backup
total 35995
-rw-r--r-- 1 root root 20971520 Oct 25 15:58 a.blob
-rw-r--r-- 1 root root 10485760 Oct 25 15:58 b.blob
-rw-r--r-- 1 root root 5242880 Oct 25 15:58 c.blob
drwx------ 2 root root 12288 Oct 25 15:27 lost+found
root@deb6dev:~# df | grep backup
/dev/md106 49490 40906 6029 88% /mnt/backup
root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
md100 : active raid1 sdd[2] sdc[0]
51188 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
Data written and array devices in sync (bitmap 0/1)
Fail and remove device
-------------------
root@deb6dev:~# sync
root@deb6dev:~# mdadm -vv /dev/md100 --fail /dev/sdd --remove /dev/sdd
mdadm: set /dev/sdd faulty in /dev/md100
mdadm: hot removed /dev/sdd from /dev/md100
root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
md100 : active raid1 sdc[0]
51188 blocks super 1.2 [2/1] [U_]
bitmap: 0/1 pages [0KB], 65536KB chunk
Device may now be unplugged
Write to the array again
--------------------
root@deb6dev:~# rm /mnt/backup/b.blob
root@deb6dev:~# ls -l /mnt/backup
total 25714
-rw-r--r-- 1 root root 20971520 Oct 25 15:58 a.blob
-rw-r--r-- 1 root root 5242880 Oct 25 15:58 c.blob
drwx------ 2 root root 12288 Oct 25 15:27 lost+found
root@deb6dev:~# df | grep backup
/dev/md106 49490 30625 16310 66% /mnt/backup
root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
md100 : active raid1 sdc[0]
51188 blocks super 1.2 [2/1] [U_]
bitmap: 1/1 pages [4KB], 65536KB chunk
bitmap 1/1 shows array is not in sync (we know it's due to the writes
pending for the device we previously failed)
Plug in a device that was previously plugged in
----------------------------------------
root@deb6dev:~# mdadm -vv -I /dev/sdd --run
mdadm: UUID differs from /dev/md/0.
mdadm: UUID differs from /dev/md/1.
mdadm: /dev/sdd attached to /dev/md100 which is already active.
root@deb6dev:~# sync
root@deb6dev:~# cat /proc/mdstat | grep -A 3 '^md100'
md100 : active raid1 sdd[2] sdc[0]
51188 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
Device reconnected [UU] and in sync (bitmap 0/1)
Restore from off-site device
------------------------
Remove device from array
root@deb6dev:~# mdadm -vv /dev/md100 --fail /dev/sdd --remove /dev/sdd
mdadm: set /dev/sdd faulty in /dev/md100
mdadm: hot removed /dev/sdd from /dev/md100
root@deb6dev:~# mdadm -Ev /dev/sdd
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4c957fac:d7dbc792:b642daf0:d22e313e
Name : deb6dev:100 (local to host deb6dev)
Creation Time : Tue Oct 25 15:22:19 2011
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 102376 (50.00 MiB 52.42 MB)
Array Size : 102376 (50.00 MiB 52.42 MB)
Data Offset : 24 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 381f453f:5a97f1f6:bb5098bb:8c071a95
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Oct 25 17:27:53 2011
Checksum : acbcee5f - correct
Events : 250
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing)
Assemble a new array from off-site component:
root@deb6dev:~# mdadm -vv -A /dev/md200 --run /dev/sdd
mdadm: looking for devices for /dev/md200
mdadm: /dev/sdd is identified as a member of /dev/md200, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md200
mdadm: added /dev/sdd to /dev/md200 as 1
mdadm: /dev/md200 has been started with 1 drive (out of 2).
root@deb6dev:~#
Check file-system on new array
root@deb6dev:~# fsck.ext3 -f -n /dev/md200
e2fsck 1.41.12 (17-May-2010)
fsck.ext3: Superblock invalid, trying backup blocks...
fsck.ext3: Bad magic number in super-block while trying to open /dev/md200
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
How do I use these devices in a new array?
Kind regards and thanks for your help,
Josh
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
