Hello,
What is your current setup, 1 server pet data center with 12 osd
each? What is your current crush rule and LRC crush rule?
On Fri, Apr 28, 2023, 12:29 Michel Jouvin
<michel.jouvin@xxxxxxxxxxxxxxx> wrote:
Hi,
I think I found a possible cause of my PG down but still
understand why.
As explained in a previous mail, I setup a 15-chunk/OSD EC pool
(k=9,
m=6) but I have only 12 OSD servers in the cluster. To workaround
the
problem I defined the failure domain as 'osd' with the reasoning
that as
I was using the LRC plugin, I had the warranty that I could loose
a site
without impact, thus the possibility to loose 1 OSD server. Am I
wrong?
Best regards,
Michel
Le 24/04/2023 à 13:24, Michel Jouvin a écrit :
> Hi,
>
> I'm still interesting by getting feedback from those using the LRC
> plugin about the right way to configure it... Last week I upgraded
> from Pacific to Quincy (17.2.6) with cephadm which is doing the
> upgrade host by host, checking if an OSD is ok to stop before
actually
> upgrading it. I had the surprise to see 1 or 2 PGs down at some
points
> in the upgrade (happened not for all OSDs but for every
> site/datacenter). Looking at the details with "ceph health
detail", I
> saw that for these PGs there was 3 OSDs down but I was expecting
the
> pool to be resilient to 6 OSDs down (5 for R/W access) so I'm
> wondering if there is something wrong in our pool configuration
(k=9,
> m=6, l=5).
>
> Cheers,
>
> Michel
>
> Le 06/04/2023 à 08:51, Michel Jouvin a écrit :
>> Hi,
>>
>> Is somebody using LRC plugin ?
>>
>> I came to the conclusion that LRC k=9, m=3, l=4 is not the
same as
>> jerasure k=9, m=6 in terms of protection against failures and
that I
>> should use k=9, m=6, l=5 to get a level of resilience >= jerasure
>> k=9, m=6. The example in the documentation (k=4, m=2, l=3)
suggests
>> that this LRC configuration gives something better than
jerasure k=4,
>> m=2 as it is resilient to 3 drive failures (but not 4 if I
understood
>> properly). So how many drives can fail in the k=9, m=6, l=5
>> configuration first without loosing RW access and second without
>> loosing data?
>>
>> Another thing that I don't quite understand is that a pool
created
>> with this configuration (and failure domain=osd,
locality=datacenter)
>> has a min_size=3 (max_size=18 as expected). It seems wrong to
me, I'd
>> expected something ~10 (depending on answer to the previous
question)...
>>
>> Thanks in advance if somebody could provide some sort of
>> authoritative answer on these 2 questions. Best regards,
>>
>> Michel
>>
>> Le 04/04/2023 à 15:53, Michel Jouvin a écrit :
>>> Answering to myself, I found the reason for 2147483647: it's
>>> documented as a failure to find enough OSD (missing OSDs). And
it is
>>> normal as I selected different hosts for the 15 OSDs but I
have only
>>> 12 hosts!
>>>
>>> I'm still interested by an "expert" to confirm that LRC k=9,
m=3,
>>> l=4 configuration is equivalent, in terms of redundancy, to a
>>> jerasure configuration with k=9, m=6.
>>>
>>> Michel
>>>
>>> Le 04/04/2023 à 15:26, Michel Jouvin a écrit :
>>>> Hi,
>>>>
>>>> As discussed in another thread (Crushmap rule for
multi-datacenter
>>>> erasure coding), I'm trying to create an EC pool spanning 3
>>>> datacenters (datacenters are present in the crushmap), with the
>>>> objective to be resilient to 1 DC down, at least keeping the
>>>> readonly access to the pool and if possible the read-write
access,
>>>> and have a storage efficiency better than 3 replica (let say a
>>>> storage overhead <= 2).
>>>>
>>>> In the discussion, somebody mentioned LRC plugin as a possible
>>>> jerasure alternative to implement this without tweaking the
>>>> crushmap rule to implement the 2-step OSD allocation. I
looked at
>>>> the documentation
>>>>
(https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/)
>>>> but I have some questions if someone has experience/expertise
with
>>>> this LRC plugin.
>>>>
>>>> I tried to create a rule for using 5 OSDs per datacenter (15 in
>>>> total), with 3 (9 in total) being data chunks and others being
>>>> coding chunks. For this, based of my understanding of
examples, I
>>>> used k=9, m=3, l=4. Is it right? Is this configuration
equivalent,
>>>> in terms of redundancy, to a jerasure configuration with k=9,
m=6?
>>>>
>>>> The resulting rule, which looks correct to me, is:
>>>>
>>>> --------
>>>>
>>>> {
>>>> "rule_id": 6,
>>>> "rule_name": "test_lrc_2",
>>>> "ruleset": 6,
>>>> "type": 3,
>>>> "min_size": 3,
>>>> "max_size": 15,
>>>> "steps": [
>>>> {
>>>> "op": "set_chooseleaf_tries",
>>>> "num": 5
>>>> },
>>>> {
>>>> "op": "set_choose_tries",
>>>> "num": 100
>>>> },
>>>> {
>>>> "op": "take",
>>>> "item": -4,
>>>> "item_name": "default~hdd"
>>>> },
>>>> {
>>>> "op": "choose_indep",
>>>> "num": 3,
>>>> "type": "datacenter"
>>>> },
>>>> {
>>>> "op": "chooseleaf_indep",
>>>> "num": 5,
>>>> "type": "host"
>>>> },
>>>> {
>>>> "op": "emit"
>>>> }
>>>> ]
>>>> }
>>>>
>>>> ------------
>>>>
>>>> Unfortunately, it doesn't work as expected: a pool created with
>>>> this rule ends up with its pages active+undersize, which is
>>>> unexpected for me. Looking at 'ceph health detail` output, I
see
>>>> for each page something like:
>>>>
>>>> pg 52.14 is stuck undersized for 27m, current state
>>>> active+undersized, last acting
>>>>
[90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647]
>>>>
>>>> For each PG, there is 3 '2147483647' entries and I guess it
is the
>>>> reason of the problem. What are these entries about? Clearly
it is
>>>> not OSD entries... Looks like a negative number, -1, which in
terms
>>>> of crushmap ID is the crushmap root (named "default" in our
>>>> configuration). Any trivial mistake I would have made?
>>>>
>>>> Thanks in advance for any help or for sharing any successful
>>>> configuration?
>>>>
>>>> Best regards,
>>>>
>>>> Michel
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
