Re: Anybody using 4x (size=4) replication?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 7/25/19 9:56 AM, Xiaoxi Chen wrote:
> The real impact of changing min_size to 1 , is not about the possibility
> of losing data ,but how much data it will lost.. in both case you will
> lost some data , just how much.
> 
> Let PG X -> (osd A, B, C), min_size = 2, size =3
> In your description, 
> 
> T1,   OSD A goes down due to upgrade, now the PG is in degraded mode
> with   (B,C),  note that the PG is still active so that there is data
> only written to B and C.  
> 
> T2 , B goes down to due to disk failure.  C is the only one holding the
> portion of data between [T1, T2].
> The failure rate of C, in this situation  , is independent to whether we
> continue writing to C. 
> 
> if C failed in T3,  
>     w/o changing min_size, you  lost data from [T1, T2] together with
> data unavailable from [T2,T3]
>     changing min_size = 1 , you lost data from [T1, T3] 
> 
> But agree, it is a tradeoff,  depending on how you believe you wont have
> two drive failure in a row within 15 mins upgrade window...
> 

Yes. So with min_size=1 you would need a single disk failure (B in your
example) to loose data.

If min_size=2 you would need both B and C to fail within that window or
when they are recovering A.

That is an even smaller chance than a single disk failure.

Wido

> Wido den Hollander <wido@xxxxxxxx <mailto:wido@xxxxxxxx>> 于2019年7月25
> 日周四 下午3:39写道:
> 
> 
> 
>     On 7/25/19 9:19 AM, Xiaoxi Chen wrote:
>     > We had hit this case in production but my solution will be change
>     > min_size = 1 immediately so that PG back to active right after.
>     >
>     > It somewhat tradeoff reliability(durability) with availability during
>     > that window of 15 mins but if you are certain one out of two "failure"
>     > is due to recoverable issue, it worth to do so.
>     >
> 
>     That's actually dangerous imho.
> 
>     Because while you set min_size=1 you will be mutating data on that
>     single disk/OSD.
> 
>     If the other two OSDs come back recovery will start. Now IF that single
>     disk/OSD now dies while performing the recovery you have lost data.
> 
>     The PG (or PGs) becomes inactive and you either need to perform data
>     recovery on the failed disk or revert back to the last state.
> 
>     I can't take that risk in this situation.
> 
>     Wido
> 
>     > My 0.02
>     >
>     > Wido den Hollander <wido@xxxxxxxx <mailto:wido@xxxxxxxx>
>     <mailto:wido@xxxxxxxx <mailto:wido@xxxxxxxx>>> 于2019年7月25
>     > 日周四 上午3:48写道:
>     >
>     >
>     >
>     >     On 7/24/19 9:35 PM, Mark Schouten wrote:
>     >     > I’d say the cure is worse than the issue you’re trying to
>     fix, but
>     >     that’s my two cents.
>     >     >
>     >
>     >     I'm not completely happy with it either. Yes, the price goes
>     up and
>     >     latency increases as well.
>     >
>     >     Right now I'm just trying to find a clever solution to this.
>     It's a 2k
>     >     OSD cluster and the likelihood of an host or OSD crashing is
>     reasonable
>     >     while you are performing maintenance on a different host.
>     >
>     >     All kinds of things have crossed my mind where using size=4 is one
>     >     of them.
>     >
>     >     Wido
>     >
>     >     > Mark Schouten
>     >     >
>     >     >> Op 24 jul. 2019 om 21:22 heeft Wido den Hollander
>     <wido@xxxxxxxx <mailto:wido@xxxxxxxx>
>     >     <mailto:wido@xxxxxxxx <mailto:wido@xxxxxxxx>>> het volgende
>     geschreven:
>     >     >>
>     >     >> Hi,
>     >     >>
>     >     >> Is anybody using 4x (size=4, min_size=2) replication with Ceph?
>     >     >>
>     >     >> The reason I'm asking is that a customer of mine asked me for a
>     >     solution
>     >     >> to prevent a situation which occurred:
>     >     >>
>     >     >> A cluster running with size=3 and replication over different
>     >     racks was
>     >     >> being upgraded from 13.2.5 to 13.2.6.
>     >     >>
>     >     >> During the upgrade, which involved patching the OS as well,
>     they
>     >     >> rebooted one of the nodes. During that reboot suddenly a
>     node in a
>     >     >> different rack rebooted. It was unclear why this happened, but
>     >     the node
>     >     >> was gone.
>     >     >>
>     >     >> While the upgraded node was rebooting and the other node
>     crashed
>     >     about
>     >     >> 120 PGs were inactive due to min_size=2
>     >     >>
>     >     >> Waiting for the nodes to come back, recovery to finish it took
>     >     about 15
>     >     >> minutes before all VMs running inside OpenStack were back
>     again.
>     >     >>
>     >     >> As you are upgraded or performing any maintenance with
>     size=3 you
>     >     can't
>     >     >> tolerate a failure of a node as that will cause PGs to go
>     inactive.
>     >     >>
>     >     >> This made me think about using size=4 and min_size=2 to
>     prevent this
>     >     >> situation.
>     >     >>
>     >     >> This obviously has implications on write latency and cost,
>     but it
>     >     would
>     >     >> prevent such a situation.
>     >     >>
>     >     >> Is anybody here running a Ceph cluster with size=4 and
>     min_size=2 for
>     >     >> this reason?
>     >     >>
>     >     >> Thank you,
>     >     >>
>     >     >> Wido
>     >     >> _______________________________________________
>     >     >> ceph-users mailing list
>     >     >> ceph-users@xxxxxxxxxxxxxx
>     <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx
>     <mailto:ceph-users@xxxxxxxxxxxxxx>>
>     >     >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>     >     _______________________________________________
>     >     ceph-users mailing list
>     >     ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
>     <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>
>     >     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>     >
> 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux