Re: Ceph newbee questions

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




>> 
>> You can do that for a PoC, but that's a bad idea for any production workload.  You'd want at least three nodes with OSDs to use the default RF=3 replication.  You can do RF=2, but at the peril of your mortal data.
> 
> I'm not sure I agree - I think size=2, min_size=2 is no worse than
> RAID1 for data security.

size=2, min_size=2 *is* RAID1.  Except that you become unavailable if a single drive is unavailable.

>> That isn't even the main risk as I understand it.  Of course a double
> failure is going to be a problem with size=2, or traditional RAID1,
> and I think anybody choosing this configuration accepts this risk.

We see people often enough who don’t know that.  I’ve seen double failures.  ymmv.


>  As I understand it, the reason min_size=1 is a trap has nothing to do
> with double failures per se.

It’s one of the concerns.

> 
> The issue is that Ceph OSDs are somewhat prone to flapping during
> recovery (OOM, etc).  So even if the disk is fine, an OSD can go down
> for a short time.  If you have size=2, min=1 configured, then when
> this happens the PG will become degraded and will continue operating
> on the other OSD, and the flapping OSD becomes stale.  Then when it
> comes back up it recovers.  The problem is that if the other OSD has a
> permanent failure (disk crash/etc) while the first OSD is flapping,
> now you have no good OSDs, because when the flapping OSD comes back up
> it is stale, and its PGs have no peer.  

Indeed, arguably that’s an overlapping failure.  I’ve seen this too, and have a pg query to demonstrate it.


> I suspect there are ways to re-activate it, though this will result in potential data
> inconsistency since writes were allowed to the cluster and will then
> get rolled back.

Yep.

> With only two OSDs I'm guessing that would be the
> main impact (well, depending on journaling behavior/etc), but if you
> have more OSDs than that then you could have situations where one file
> is getting rolled back, and some other file isn't, and so on.

But you’d have a voting majority.

> 
> With min_size=2 you're fairly safe from flapping because there will
> always be two replicas that have the most recent version of every PG,
> and so you can still tolerate a permanent failure of one of them.

Exactly.

> 
> size=2, min=2 doesn't suffer this failure mode, because anytime there
> is flapping the PG goes inactive and no writes can be made, so when
> the other OSD comes back up there is nothing to recover.  Of course
> this results in IO blocks and downtime, which is obviously
> undesirable, but it is likely a more recoverable state than
> inconsistent writes.

Agreed, the difference between availability and durability.  Depends what’s important to you.


> 
> Apologies if I've gotten any of that wrong, but my understanding is
> that it is these sorts of failure modes that cause min_size=1 to be a
> trap.  This isn't the sort of thing that typically happens in a RAID1
> config, or at least that admins don't think about.

It’s both.


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux