Thanks very much for the detailed explanation. We made some decisions based on budget in order to get the cluster size we needed without understanding this issue. We’ll have to try and increase our hardware to increase our replication size to 3. > On Oct 14, 2021, at 2:32 PM, Eugen Block <eblock@xxxxxx> wrote: > > Hi, > > there you have the explanation, as I suspected it's the min_size = size of the pool(s). > To prevent data loss in case of OSD failure you set reasonable replication size and a min_size for a pool. The usual is size = 3 and min_size = 2 so you can still read/write in case one host or OSD fails. There have been plenty of explanations for this, I recommend to search this mailing list for some more examples. However, if your pool size is only 2 (which is not recommended for various reasons!) you should keep min_size = 2. This prevents possible data loss in case several OSDs fail storing PGs for the same pool. > So the decision is to either reduce storage overhead by having only two replicas (size = 2) instead of three but then having IO paused while OSDs are down. Or you could increase the pool size to 3 (keep min_size = 2) with a higher storage overhead but more resiliency and availability in case of failures. > > Regards, > Eugen > > > Zitat von Jeffrey Turmelle <jefft@xxxxxxxxxxxxxxxx>: > >> Thanks for the reply Eugen, >> >> We have a 6-node nautilus (14.2.22) cluster, each with 30 12TB OSDs. I’ve attached the osd tree, if useful. I’m happy to provide more information, but to be honest I’m a relatively new cluster administrator and not sure exactly what information is useful in debugging this. I don’t know what to look for. >> >> Its a replicated (replica size 2) cephfs pool, with min-size=2, 8192 PGs. >> >> The clients that mount cephfs remain connected without issue, but the NFS-Ganesha clients hang until the pg degraded status is cleared. In this case, the clients are CentOS6 hosts mounting the Ceph FS as NFS4.1, ReadOnly. We have NFS-Ganesha configured as a cluster, and the status of the pcs NFS cluster says its fine. If I restart the NFS cluster, it will clear the client hang for a few seconds, but the client will hang again shortly (seconds) after. >> >> >> >>> On Oct 14, 2021, at 2:44 AM, Eugen Block <eblock@xxxxxx> wrote: >>> >>> Hi, >>> >>> what does your 'ceph osd tree' look like and which rules are in place for the affected pools? Can you provide more details about those pools like size, min_size, replicated or erasure-coded? >>> The first thing coming to mind is min_size. For example, if you have six hosts and an erasure-coded pool with size = 6 and failure domain is host then the pool is degraded and won't recover because there are not enough hosts. Usually, min_size would be lower (k + 1) so IO would still be served even if a host goes down, but since we don't know anything about your cluster yet we can't really tell what's going on there. >>> >>> Regards, >>> Eugen >>> >>> >>> Zitat von Jeff Turmelle <jefft@xxxxxxxxxxxxxxxx>: >>> >>>> We are using NFS-Ganesha to serve data from our Nautilus cluster to older clients. We recently had an OSD fail and the NFS server will not respond while we have degraded data redundancy. This also happens on the rare occasion when we have some lost objects on a PG. Is this a known issue and is there a workaround? >>>> >>>> — >>>> Jeff Turmelle, Lead Systems Analyst >>>> International Research Institute for Climate and Society <http://iri.columbia.edu/> >>>> Columbia Climate School <https://climate.columbia.edu/> >>>> cell: (845) 652-3461 >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > >
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
