Hello Michel,
It might be worth mentioning that the next releases of Reef and Quincy
should increase the default value of osd_max_scrubs from 1 to 3. See
the Reef pull request: https://github.com/ceph/ceph/pull/55173
You could try increasing this configuration setting if you
haven't already, but note that it can impact client I/O performance.
Also, if the delays appear to be related to a single OSD, have you
checked the health and performance of this device?
On Fri, 22 Mar 2024 at 09:29, Michel Jouvin
<michel.jouvin@xxxxxxxxxxxxxxx> wrote:
Hi,
As I said in my initial message, I'd in mind to do exactly the
same as I
identified in my initial analysis that all the PGs with this problem
where sharing one OSD (but only 20 PGs had the problem over ~200
hosted
by the OSD). But as I don't feel I'm in an urgent situation, I was
wondering if collecting more information on the problem may have some
value and which one... If it helps, I add below the `pg dump` for
the 17
PGs still with a "stucked scrub".
I observed the "stucked scrubs" is lowering very slowly. In the
last 12
hours, 1 more PG was successfully scrubbed/deep scrubbed. In case
it was
not clear in my initial message, the lists of PGs with a too old
scrub
and too old deep scrub are the same.
Without an answer, next week i may consider doing what you did:
remove
the suspect OSD (instead of just restarting it) and see it
unblocks the
stucked scrubs.
Best regards,
Michel
--------------------------------- "ceph pg dump pgs" for the 17
PGs with
a too old scrub and deep scrub (same list)
------------------------------------------------------------
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND
BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE
STATE_STAMP VERSION REPORTED
UP UP_PRIMARY ACTING ACTING_PRIMARY
LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB
DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION
SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED
29.7e3 260 0 0 0 0
1090519040 0 0 1978 500
1978 active+clean 2024-03-21T18:28:53.369789+0000
39202'2478 83812:97136 [29,141,64,194] 29
[29,141,64,194] 29 39202'2478
2024-02-17T19:56:34.413412+0000 39202'2478
2024-02-17T19:56:34.413412+0000 0 3 queued for deep
scrub
0 0
25.7cc 0 0 0 0 0
0 0 0 0 1076 0
active+clean 2024-03-21T18:09:48.104279+0000 46253'548
83812:89843 [29,50,173] 29 [29,50,173]
29 39159'536 2024-02-17T18:14:54.950401+0000 39159'536
2024-02-17T18:14:54.950401+0000 0 1 queued for deep
scrub
0 0
25.70c 0 0 0 0 0
0 0 0 0 918 0
active+clean 2024-03-21T18:00:57.942902+0000 46253'514
83812:95212 [29,195,185] 29 [29,195,185] 29
39159'530 2024-02-18T03:56:17.559531+0000 39159'530
2024-02-16T17:39:03.281785+0000 0 1 queued for deep
scrub
0 0
29.70c 249 0 0 0 0
1044381696 0 0 1987 600
1987 active+clean 2024-03-21T18:35:36.848189+0000
39202'2587 83812:99628 [29,138,63,12] 29
[29,138,63,12] 29 39202'2587
2024-02-17T21:34:22.042560+0000 39202'2587
2024-02-17T21:34:22.042560+0000 0 1 queued for deep
scrub
0 0
29.705 231 0 0 0 0
968884224 0 0 1959 500 1959
active+clean 2024-03-21T18:18:22.028551+0000 39202'2459
83812:91258 [29,147,173,61] 29 [29,147,173,61]
29 39202'2459 2024-02-17T16:41:40.421763+0000 39202'2459
2024-02-17T16:41:40.421763+0000 0 1 queued for deep
scrub
0 0
29.6b9 236 0 0 0 0
989855744 0 0 1956 500 1956
active+clean 2024-03-21T18:11:29.912132+0000 39202'2456
83812:95607 [29,199,74,16] 29 [29,199,74,16]
29 39202'2456 2024-02-17T11:46:06.706625+0000 39202'2456
2024-02-17T11:46:06.706625+0000 0 1 queued for deep
scrub
0 0
25.56e 0 0 0 0 0
0 0 0 0 1158 0
active+clean+scrubbing+deep 2024-03-22T08:09:38.840145+0000
46253'514 83812:637482 [111,29,128] 111
[111,29,128] 111 39159'579
2024-03-06T17:57:53.158936+0000 39159'579
2024-03-06T17:57:53.158936+0000 0 1 queued for deep
scrub
0 0
25.56a 0 0 0 0 0
0 0 0 0 1055 0
active+clean 2024-03-21T18:00:57.940851+0000 46253'545
83812:93475 [29,19,211] 29 [29,19,211]
29 46253'545 2024-03-07T11:12:45.881545+0000 46253'545
2024-03-07T11:12:45.881545+0000 0 28 queued for deep
scrub
0 0
25.55a 0 0 0 0 0
0 0 0 0 1022 0
active+clean 2024-03-21T18:10:24.124914+0000 46253'565
83812:89876 [29,58,195] 29 [29,58,195]
29 46253'561 2024-02-17T06:54:35.320454+0000 46253'561
2024-02-17T06:54:35.320454+0000 0 28 queued for deep
scrub
0 0
29.c0 256 0 0 0 0
1073741824 0 0 1986 600 1986
active+clean+scrubbing+deep 2024-03-22T08:09:12.849868+0000
39202'2586 83812:603625 [22,150,29,56] 22
[22,150,29,56] 22 39202'2586
2024-03-07T18:53:22.952868+0000 39202'2586
2024-03-07T18:53:22.952868+0000 0 1 queued for deep
scrub
0 0
18.6 15501 0 0 0 0
63959444676 0 0 2068 3000 2068
active+clean+scrubbing+deep 2024-03-22T02:29:24.508889+0000
81688'663900 83812:1272160 [187,29,211] 187
[187,29,211] 187 52735'663878
2024-03-06T16:36:32.080259+0000 52735'663878
2024-03-06T16:36:32.080259+0000 0 684445 deep scrubbing
for 20373s 449 0
16.15 0 0 0 0 0
0 0 0 0 0 0
active+clean 2024-03-21T18:20:29.632554+0000 0'0
83812:104893 [29,165,85] 29 [29,165,85]
29 0'0 2024-02-17T06:54:06.370647+0000 0'0
2024-02-17T06:54:06.370647+0000 0 28 queued for deep
scrub
0 0
25.45 0 0 0 0 0
0 0 0 0 1036 0
active+clean 2024-03-21T18:10:24.125134+0000 39159'561
83812:93649 [29,13,58] 29 [29,13,58]
29 39159'512 2024-02-27T12:27:35.728176+0000 39159'512
2024-02-27T12:27:35.728176+0000 0 1 queued for deep
scrub
0 0
29.249 260 0 0 0 0
1090519040 0 0 1970 500
1970 active+clean 2024-03-21T18:29:22.588805+0000
39202'2470 83812:96016 [29,191,18,143] 29
[29,191,18,143] 29 39202'2470
2024-02-17T13:32:42.910335+0000 39202'2470
2024-02-17T13:32:42.910335+0000 0 1 queued for deep
scrub
0 0
29.25a 248 0 0 0 0
1040187392 0 0 1952 600
1952 active+clean 2024-03-21T18:20:29.623422+0000
39202'2552 83812:99157 [29,200,85,164] 29
[29,200,85,164] 29 39202'2552
2024-02-17T08:33:14.326087+0000 39202'2552
2024-02-17T08:33:14.326087+0000 0 1 queued for deep
scrub
0 0
25.3cf 0 0 0 0 0
0 0 0 0 1343 0
active+clean 2024-03-21T18:16:00.933375+0000 46253'598
83812:91659 [29,75,175] 29 [29,75,175]
29 46253'598 2024-02-17T11:48:51.840600+0000 46253'598
2024-02-17T11:48:51.840600+0000 0 28 queued for deep
scrub
0 0
29.4ec 243 0 0 0 0
1019215872 0 0 1933 500
1933 active+clean 2024-03-21T18:15:35.389598+0000
39202'2433 83812:101501 [29,206,63,17] 29
[29,206,63,17] 29 39202'2433
2024-02-17T15:10:41.027755+0000 39202'2433
2024-02-17T15:10:41.027755+0000 0 3 queued for deep
scrub
0 0
Le 22/03/2024 à 08:16, Bandelow, Gunnar a écrit :
> Hi Michael,
>
> i think yesterday i found the culprit in my case.
>
> After inspecting "ceph pg dump" and especially the column
> "last_scrub_duration". I found, that any PG without proper
scrubbing
> was located on one of three OSDs (and all these OSDs share the same
> SSD for their DB). I put them on "out" and now after backfill and
> remapping everything seems to be fine.
>
> Only the log is still flooded with "scrub starts" and i have no
clue
> why these OSDs are causing the problems.
> Will investigate further.
>
> Best regards,
> Gunnar
>
> ===================================
>
> Gunnar Bandelow
> Universitätsrechenzentrum (URZ)
> Universität Greifswald
> Felix-Hausdorff-Straße 18
> 17489 Greifswald
> Germany
>
> Tel.: +49 3834 420 1450
>
>
> --- Original Nachricht ---
> *Betreff: * Re: Reef (18.2): Some PG not scrubbed/deep
> scrubbed for 1 month
> *Von: *"Michel Jouvin" <michel.jouvin@xxxxxxxxxxxxxxx
> <mailto:michel.jouvin@xxxxxxxxxxxxxxx>>
> *An: *ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> *Datum: *21-03-2024 23:40
>
>
>
> Hi,
>
> Today we decided to upgrade from 18.2.0 to 18.2.2. No real
hope of a
> direct impact (nothing in the change log related to something
> similar)
> but at least all daemons were restarted so we thought that
may be
> this
> will clear the problem at least temporarily. Unfortunately
it has not
> been the case. The same pages are still stuck, despite
continuous
> activity of scrubbing/deep scrubbing in the cluster...
>
> I'm happy to provide more information if somebody tells me
what to
> look
> at...
>
> Cheers,
>
> Michel
>
> Le 21/03/2024 à 14:40, Bernhard Krieger a écrit :
> > Hi,
> >
> > i have the same issues.
> > Deep scrub havent finished the jobs on some PGs.
> >
> > Using ceph 18.2.2.
> > Initial installed version was 18.0.0
> >
> >
> > In the logs i see a lot of scrub/deep-scrub starts
> >
> > Mar 21 14:21:09 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.b deep-scrubstarts
> > Mar 21 14:21:10 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1a deep-scrubstarts
> > Mar 21 14:21:17 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1c deep-scrubstarts
> > Mar 21 14:21:19 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 11.1 scrubstarts
> > Mar 21 14:21:27 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 14.6 scrubstarts
> > Mar 21 14:21:30 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 10.c deep-scrubstarts
> > Mar 21 14:21:35 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 12.3 deep-scrubstarts
> > Mar 21 14:21:41 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 6.0 scrubstarts
> > Mar 21 14:21:44 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 8.5 deep-scrubstarts
> > Mar 21 14:21:45 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 5.66 deep-scrubstarts
> > Mar 21 14:21:49 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 5.30 deep-scrubstarts
> > Mar 21 14:21:50 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.b deep-scrubstarts
> > Mar 21 14:21:52 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1a deep-scrubstarts
> > Mar 21 14:21:54 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1c deep-scrubstarts
> > Mar 21 14:21:55 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 11.1 scrubstarts
> > Mar 21 14:21:58 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 14.6 scrubstarts
> > Mar 21 14:22:01 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 10.c deep-scrubstarts
> > Mar 21 14:22:04 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 12.3 scrubstarts
> > Mar 21 14:22:13 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 6.0 scrubstarts
> > Mar 21 14:22:15 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 8.5 deep-scrubstarts
> > Mar 21 14:22:20 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 5.66 deep-scrubstarts
> > Mar 21 14:22:27 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 5.30 scrubstarts
> > Mar 21 14:22:30 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.b deep-scrubstarts
> > Mar 21 14:22:32 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1a deep-scrubstarts
> > Mar 21 14:22:33 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1c deep-scrubstarts
> > Mar 21 14:22:35 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 11.1 deep-scrubstarts
> > Mar 21 14:22:37 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 14.6 scrubstarts
> > Mar 21 14:22:38 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 10.c scrubstarts
> > Mar 21 14:22:39 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 12.3 scrubstarts
> > Mar 21 14:22:41 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 6.0 deep-scrubstarts
> > Mar 21 14:22:43 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 8.5 deep-scrubstarts
> > Mar 21 14:22:46 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 5.66 deep-scrubstarts
> > Mar 21 14:22:49 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 5.30 scrubstarts
> > Mar 21 14:22:55 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.b deep-scrubstarts
> > Mar 21 14:22:57 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1a deep-scrubstarts
> > Mar 21 14:22:58 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 13.1c deep-scrubstarts
> > Mar 21 14:23:03 ceph-node10 ceph-osd[3804193]:
log_channel(cluster)
> > log [DBG] : 11.1 deep-scrubstarts
> >
> >
> >
> > *
> > *The amount of scrubbed/deep-scrubbed pgs changes every
few seconds.
> >
> > [root@ceph-node10 ~]# ceph -s | grep active+clean
> > pgs: 214 active+clean
> > 50 active+clean+scrubbing+deep
> > 25 active+clean+scrubbing
> > [root@ceph-node10 ~]# ceph -s | grep active+clean
> > pgs: 208 active+clean
> > 53 active+clean+scrubbing+deep
> > 28 active+clean+scrubbing
> > [root@ceph-node10 ~]# ceph -s | grep active+clean
> > pgs: 208 active+clean
> > 53 active+clean+scrubbing+deep
> > 28 active+clean+scrubbing
> > [root@ceph-node10 ~]# ceph -s | grep active+clean
> > pgs: 207 active+clean
> > 54 active+clean+scrubbing+deep
> > 28 active+clean+scrubbing
> > [root@ceph-node10 ~]# ceph -s | grep active+clean
> > pgs: 202 active+clean
> > 56 active+clean+scrubbing+deep
> > 31 active+clean+scrubbing
> > [root@ceph-node10 ~]# ceph -s | grep active+clean
> > pgs: 213 active+clean
> > 45 active+clean+scrubbing+deep
> > 31 active+clean+scrubbing
> >
> > ceph pg dump showing PGs which are not deep scrubbed since
january.
> > Some PGs deep scrubbing over 700000 seconds.
> >
> > *[ceph: root@ceph-node10 /]# ceph pg dump pgs | grep -e
> 'scrubbing f'
> > 5.6e 221223 0 0 0
0
> > 927795290112 0 0 4073 3000
4073
> > active+clean+scrubbing+deep 2024-03-20T01:07:21.196293+
> > 0000 128383'15766927 128383:20517419 [2,4,18,16,14,21]
> 2
> > [2,4,18,16,14,21] 2 125519'12328877
> > 2024-01-23T11:25:35.503811+0000 124844'11873951
2024-01-21T22:
> > 24:12.620693+0000 0 5 deep
> scrubbing
> > for 270790s 53772
> > 0
> > 5.6c 221317 0 0 0
0
> > 928173256704 0 0 6332 0
6332
> > active+clean+scrubbing+deep 2024-03-18T09:29:29.233084+
> > 0000 128382'15788196 128383:20727318 [6,9,12,14,1,4]
> 6
> > [6,9,12,14,1,4] 6 127180'14709746
> > 2024-03-06T12:47:57.741921+0000 124817'11821502
2024-01-20T20:
> > 59:40.566384+0000 0 13452 deep
> scrubbing
> > for 273519s 122803
> > 0
> > 5.6a 221325 0 0 0
0
> > 928184565760 0 0 4649 3000
4649
> > active+clean+scrubbing+deep 2024-03-13T03:48:54.065125+
> > 0000 128382'16031499 128383:21221685 [13,11,1,2,9,8]
> 13
> > [13,11,1,2,9,8] 13 127181'14915404
> > 2024-03-06T13:16:58.635982+0000 125967'12517899
2024-01-28T09:
> > 13:08.276930+0000 0 10078 deep
> scrubbing
> > for 726001s 184819
> > 0
> > 5.54 221050 0 0 0
0
> > 927036203008 0 0 4864 3000
4864
> > active+clean+scrubbing+deep 2024-03-18T00:17:48.086231+
> > 0000 128383'15584012 128383:20293678 [0,20,18,19,11,12]
> 0
> > [0,20,18,19,11,12] 0 127195'14651908
> > 2024-03-07T09:22:31.078448+0000 124816'11813857
2024-01-20T16:
> > 43:15.755200+0000 0 9808 deep
> scrubbing
> > for 306667s 142126
> > 0
> > 5.47 220849 0 0 0
0
> > 926233448448 0 0 5592 0
5592
> > active+clean+scrubbing+deep 2024-03-12T08:10:39.413186+
> > 0000 128382'15653864 128383:20403071 [16,15,20,0,13,21]
> 16
> > [16,15,20,0,13,21] 16 127183'14600433
> > 2024-03-06T18:21:03.057165+0000 124809'11792397
2024-01-20T05:
> > 27:07.617799+0000 0 13066 deep
> scrubbing
> > for 796697s 209193
> > 0
> > dumped pgs
> >
> >
> > *
> >
> >
> > regards
> > Bernhard
> >
> >
> >
> >
> >
> >
> > On 20/03/2024 21:12, Bandelow, Gunnar wrote:
> >> Hi,
> >>
> >> i just wanted to mention, that i am running a cluster
with reef
> >> 18.2.1 with the same issue.
> >>
> >> 4 PGs start to deepscrub but dont finish since mid
february. In
> the
> >> pg dump they are shown as scheduled for deep scrub. They
sometimes
> >> change their status from active+clean to
> active+clean+scrubbing+deep
> >> and back.
> >>
> >> Best regards,
> >> Gunnar
> >>
> >> =======================================================
> >>
> >> Gunnar Bandelow
> >> Universitätsrechenzentrum (URZ)
> >> Universität Greifswald
> >> Felix-Hausdorff-Straße 18
> >> 17489 Greifswald
> >> Germany
> >>
> >> Tel.: +49 3834 420 1450
> >>
> >>
> >>
> >>
> >> --- Original Nachricht ---
> >> *Betreff: * Re: Reef (18.2): Some PG not
scrubbed/deep
> >> scrubbed for 1 month
> >> *Von: *"Michel Jouvin" <michel.jouvin@xxxxxxxxxxxxxxx
> <mailto:michel.jouvin@xxxxxxxxxxxxxxx>
> >> <michel.jouvin@xxxxxxxxxxxxxxx
> <mailto:michel.jouvin@xxxxxxxxxxxxxxx>>>
> >> *An: *ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> <ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>>
> >> *Datum: *20-03-2024 20:00
> >>
> >>
> >>
> >> Hi Rafael,
> >>
> >> Good to know I am not alone!
> >>
> >> Additional information ~6h after the OSD restart:
over the
> 20 PGs
> >> impacted, 2 have been processed successfully... I don't
> have a clear
> >> picture on how Ceph prioritize the scrub of one PG over
> another, I
> >> had
> >> thought that the oldest/expired scrubs are taken
first but
> it may
> >> not be
> >> the case. Anyway, I have seen a very significant
decrese of
> the
> >> scrub
> >> activity this afternoon and the cluster is not loaded
at all
> >> (almost no
> >> users yet)...
> >>
> >> Michel
> >>
> >> Le 20/03/2024 à 17:55, quaglio@xxxxxxxxxx
> <mailto:quaglio@xxxxxxxxxx>
> >> <quaglio@xxxxxxxxxx <mailto:quaglio@xxxxxxxxxx>> a
écrit :
> >> > Hi,
> >> > I upgraded a cluster 2 weeks ago here. The
situation
> is the
> >> same
> >> > as Michel.
> >> > A lot of PGs no scrubbed/deep-scrubed.
> >> >
> >> > Rafael.
> >> >
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> <mailto:ceph-users@xxxxxxx>
> >> <ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>>
> >> > To unsubscribe send an email to
ceph-users-leave@xxxxxxx
> <mailto:ceph-users-leave@xxxxxxx>
> >> <ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@xxxxxxx
> <mailto:ceph-users@xxxxxxx>
> >> <ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>>
> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> <mailto:ceph-users-leave@xxxxxxx>
> >> <ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>>
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list --ceph-users@xxxxxxx
> <mailto:ceph-users@xxxxxxx>
> >> To unsubscribe send an email toceph-users-leave@xxxxxxx
> <mailto:toceph-users-leave@xxxxxxx>
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> <mailto:ceph-users@xxxxxxx>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> <mailto:ceph-users-leave@xxxxxxx>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> <mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> <mailto:ceph-users-leave@xxxxxxx>
>
>
> _______________________________________________
> ceph-users mailing list --ceph-users@xxxxxxx
> To unsubscribe send an email toceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
