Hello
Exactly this is why I started the topic:
"Say the mirrors are incosistent due to an unnoted read error, the RAID
layer can not decide which of the two legs has faulty data. It can
whatsoever reread both legs in hope the faulty read is corrected on reread
and rewrite afterwards. I fear such actions are only taken during a forced
rebuild though."
Back in 2005 when I was working a lot with servers built of commodity hardware (crappy asus motherboards with their *fake* raid controllers on board) I saw lots of interesting things. That was about the time when I lost my faith in RAID technologies forever. I rather make backups to tapes, cds, other drives periodically and stacking them up somewhere.
Some of the worst failures I saw were corrupted RAID5 arrays with ext3, reiserfs, xfs. These corruptions mostly happened because of regular power outages and whenever I had to deal with them I know the chances to get any data back is less than 20% and then we don't even talk about any encryption just regular filesystems in raid.
How does LUKS handles if part of the encrypted disk (not the header) or container gets corrupted?
With some encryption technologies even if 1 bit gets damaged in a container the data lost forever or becomes partially corrupted.
So this was back then in the time of slow ATA drives, linux kernel 2.4, raid-utils. Recovery on a 100GB drive took over a day.
Today still bunch of low end servers have those fake software raid controllers where you cant even swap a drive without shutting the machine down. Even tho if something goes wrong with an mdadm based raid array you still have more tools, community support and chance to recover data then from a 3ware or hp array.
From: "Sven Eschenberg" <sven@xxxxxxxxxxxxxxxxxxxxx>
To: dm-crypt@xxxxxxxx
Subject: Re: LUKS safety on RAID 1 mirror
As you stated, backups are mandatory and RAID's purpose is extended
availability (and speed).
Regarding the concerns of the OP:
When a device fails and gets marked as failed there's no difference to
single drive operation. With TLER drives the drive will probably not get
marked faulty and the broken sector can be rewritten with the data of the
other leg, if that's implemented apropriately.
What is problematic in a RAID is failure and unreported errors during
read(). Say a sector including the LUKS header is instable, gets read and
the retrieved data is faulty then broken data might get written to the
mirror during manipulation operations including a following write. (Can be
compensated by backups though)
With two disks the probability of such a specific error increases, on the
other hand a RAID1 implementation *should* level reads which in turn
decreases the prob. to hit such a specific read error.
The question that remains is: How probable is an unnoted (or unreported)
read error and how does the RAID implementation handle specific error
scenarios? (Unfortunately there's firmware bugs ...)
Say the mirrors are incosistent due to an unnoted read error, the RAID
layer can not decide which of the two legs has faulty data. It can
whatsoever reread both legs in hope the faulty read is corrected on reread
and rewrite afterwards. I fear such actions are only taken during a forced
rebuild though.
Reagrds
-Sven
On Tue, November 25, 2014 15:24, Arno Wagner wrote:
> On Tue, Nov 25, 2014 at 11:28:47 CET, Fabrice Bongartz wrote:
>> Hi Mark,
>>
>> I currently employ the following setup:
>> I have multiple md software raid 1 arrays and luks on top of that. For
>> example, /dev/sda1 and /dev/sdb1 are two identifcal disks which are in a
>> raid1 using md raid as /dev/md0. The luks encrypted device is /dev/md0.
>> So far, I have had two discs fail in two different arrays and I have had
>> no problem restoring them. The array continued in degrated mode and I
>> could safely replace the two drives and add the new disks to the arrays
>> using the mdadm command.
>>
>> I am also curious as to what the devs have to say about this.
>
> RAID and LUKS are in separate layers and do not influence
> each other. See FAQ Items 2.2 ad 2.8. 2.8 also has a picture.
>
> If you place LUKS atop RAID, you get pretty much
> the same change as with a normal filesystem atop RAID. Of
> course, the LUKS header is critical, which is why you should
> always have a header backup, just the same as without RAID.
>
> If you place LUKS below RAID (not that good an idea), you
> will have to unlock the raw devices before the RAID can
> be assembled. You should have header backups for as much
> devices as are neded to assemble the RAID, but better for
> all.
>
> Really, these are separate issuses, LUKS and RAID do not
> magically interact behind your back.
>
> Gr"usse,
> Arno
>
>> BTW: I always make a complete backup on a third external disk, I don't
>> want to take any chances.
>>
>> Cheers,
>>
>> Fabrice Bongartz
>>
>>
>> Von: "Mark Connor" <markc44@xxxxxxx>
>> An: "dm-crypt" <dm-crypt@xxxxxxxx>
>> Gesendet: Dienstag, 25. November 2014 11:03:17
>> Betreff: LUKS safety on RAID 1 mirror
>>
>> Hello
>>
>> I currently have a deployment with luks (aes-cbc-256) on different 1TB,
>> 500GB, 300GB etc. drives. All the drives use different keys and XFS
>> filesystem on the top of luks.
>> I'm planning to replace this setup with 2X4TB disks in software raid1
>> (with mdraid) but I have my concerns.
>>
>> 1, If a sector goes bad on disk1 that normally shouldn't be replicated
>> to disk2 but in case of luks I don't know what happens then.
>>
>> 2, I think it is more practical -when one is dealing with encryption- to
>> keep many smaller partitions encrypted with separate keys, in case of
>> partial disk failure (other parts of the disk can still be accessed).
>> Also all the partitions have their own separate luks headers...
>>
>> Unlike if I don't even create partition just put sda (4TB) sdb(4TB) into
>> and md0 array and make luks on that one, if anything goes wrong with the
>> header I lose all my data or if any part of the disks breaks.
>>
>> I know that ultimately raid is only protect against drive failures (not
>> if files get corrupted or deleted) so have to have a separated
>> snapshotted backup next to it. But would implementing raid1 in case of
>> luks be an advantage or a disadvantage?
>>
>> Thanks
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@xxxxxxxx
>> http://www.saout.de/mailman/listinfo/dm-crypt
>
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@xxxxxxxx
>> http://www.saout.de/mailman/listinfo/dm-crypt
>
>
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: arno@xxxxxxxxxxx
> GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D
> 9718
> ----
> A good decision is based on knowledge and not on numbers. -- Plato
>
> If it's in the news, don't worry about it. The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@xxxxxxxx
> http://www.saout.de/mailman/listinfo/dm-crypt
>
_______________________________________________
dm-crypt mailing list
dm-crypt@xxxxxxxx
http://www.saout.de/mailman/listinfo/dm-crypt
_______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
