Re: CephFS metadata pool size

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



ok. Thank you for your help.
We will try this and report back in a few days.


[image: ariadne.ai Logo] Lars Köppel
Developer
Email: lars.koeppel@xxxxxxxxxx
Phone: +49 6221 5993580 <+4962215993580>
ariadne.ai (Germany) GmbH
Häusserstraße 3, 69115 Heidelberg
Amtsgericht Mannheim, HRB 744040
Geschäftsführer: Dr. Fabian Svara
https://ariadne.ai


On Thu, Jun 13, 2024 at 3:01 PM Eugen Block <eblock@xxxxxx> wrote:

> I'm quite sure that this could result in the impact you're seeing. To
> confirm that suspicion you could stop deleting and wait a couple of
> days to see if the usage stabilizes. And if it does, maybe delete less
> files at once or so to see how far you can tweak it. That would be my
> approach.
>
> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>
> > We have been using snapshots for a long time.
> > The only change in usage is that we are currently deleting many small
> files
> > from the system. Because this is slow (~150 requests/s) this is running
> for
> > the last few weeks. Could such a load result in a problem with the MDS?
> >
> > I have to ask for permission to order more drives. This could take some
> > time.
> >
> > [image: ariadne.ai Logo] Lars Köppel
> > Developer
> > Email: lars.koeppel@xxxxxxxxxx
> > Phone: +49 6221 5993580 <+4962215993580>
> > ariadne.ai (Germany) GmbH
> > Häusserstraße 3, 69115 Heidelberg
> > Amtsgericht Mannheim, HRB 744040
> > Geschäftsführer: Dr. Fabian Svara
> > https://ariadne.ai
> >
> >
> > On Thu, Jun 13, 2024 at 12:55 PM Eugen Block <eblock@xxxxxx> wrote:
> >
> >> Downgrading isn't supported, I don't think that would be a good idea.
> >> I also don't see anything obvious standing out in the pg output. Any
> >> chance you can add more OSDs to the metadata pool to see if it stops
> >> at some point? Did the cluster usage change in any way? For example
> >> cephfs snapshots which haven't been used before or something like that?
> >>
> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >>
> >> > I updated from 17.2.6 to 17.2.7 and a few hours later to 18.2.2.
> >> > Would it be an option to go back to 17.2.6?
> >> >
> >> >
> >> > [image: ariadne.ai Logo] Lars Köppel
> >> > Developer
> >> > Email: lars.koeppel@xxxxxxxxxx
> >> > Phone: +49 6221 5993580 <+4962215993580>
> >> > ariadne.ai (Germany) GmbH
> >> > Häusserstraße 3, 69115 Heidelberg
> >> > Amtsgericht Mannheim, HRB 744040
> >> > Geschäftsführer: Dr. Fabian Svara
> >> > https://ariadne.ai
> >> >
> >> >
> >> > On Wed, Jun 12, 2024 at 5:30 PM Eugen Block <eblock@xxxxxx> wrote:
> >> >
> >> >> Which version did you upgrade from to 18.2.2?
> >> >> I can’t pin it down to a specific issue, but somewhere in the back of
> >> >> my mind is something related to a new omap format or something. But
> >> >> I’m really not sure at all.
> >> >>
> >> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >> >>
> >> >> > I am happy to help you with as much information as possible. I
> >> probably
> >> >> > just don't know where to look for it.
> >> >> > Below are the requested information. The cluster is rebuilding the
> >> >> > zapped OSD at the moment. This will probably take the next few
> days.
> >> >> >
> >> >> >
> >> >> > sudo ceph pg ls-by-pool metadata
> >> >> > PG     OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES     OMAP_BYTES*
> >> >> >  OMAP_KEYS*  LOG   LOG_DUPS  STATE
> >> >> >      SINCE  VERSION          REPORTED         UP         ACTING
> >> >> > SCRUB_STAMP                      DEEP_SCRUB_STAMP
> >> >> > LAST_SCRUB_DURATION  SCRUB_SCHEDULING
> >> >> > 10.0   5217325   4994695          0        0   4194304   5880891340
> >> >> > 9393865  1885      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'180849582  79875:391519635  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T09:08:09.829362+0000  2024-05-28T05:52:59.321589+0000
> >> >> >        627  periodic scrub scheduled @
> 2024-06-17T08:21:31.808348+0000
> >> >> > 10.1   5214785   5193424          0        0         0   5843682713
> >> >> > 9410150  1912      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180914288  79875:342746928  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-01T15:56:28.927288+0000  2024-05-27T03:31:37.682966+0000
> >> >> >        966  queued for scrub
> >> >> > 10.2   5218432   5187168          0        0         0   6402011266
> >> >> > 9812513  1874      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180970531  79875:341340204  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T13:40:58.994256+0000  2024-06-11T13:40:58.994256+0000
> >> >> >       1942  periodic scrub scheduled @
> 2024-06-17T06:07:15.329675+0000
> >> >> > 10.3   5217413   5217413          0        0   8388788   5766005023
> >> >> > 9271787  1923      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181012233  79875:388295881  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T00:35:56.965547+0000  2024-05-23T19:54:56.121729+0000
> >> >> >        492  periodic scrub scheduled @
> 2024-06-18T06:39:31.103864+0000
> >> >> > 10.4   5220069   5220069          0        0  12583466   6027548724
> >> >> > 9537290  1959      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181576075  79875:405295868  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T17:47:22.923514+0000  2024-05-31T02:06:55.339574+0000
> >> >> >        581  periodic scrub scheduled @
> 2024-06-17T00:59:37.214420+0000
> >> >> > 10.5   5216162   5211999          0        0   4194304   5941347251
> >> >> > 9542764  1930      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180455793  79875:338418517  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-11T22:50:16.170708+0000  2024-05-30T23:49:54.316379+0000
> >> >> >        528  periodic scrub scheduled @
> 2024-06-17T04:39:25.905185+0000
> >> >> > 10.6   5216100   4980459          0        0   4521984   6428088514
> >> >> > 9850762  1911      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'184045876  79875:396809795  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T22:24:05.102716+0000  2024-06-11T22:24:05.102716+0000
> >> >> >       1082  periodic scrub scheduled @
> 2024-06-17T07:58:44.289885+0000
> >> >> > 10.7   5218232   5218232          0        0   4194304   6377065363
> >> >> > 9849360  1919      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182672562  79875:342449062  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-11T06:22:15.689422+0000  2024-06-11T06:22:15.689422+0000
> >> >> >       8225  periodic scrub scheduled @
> 2024-06-17T13:05:59.225052+0000
> >> >> > 10.8   5219620   5182816          0        0         0   6167304290
> >> >> > 9691796  1896      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179628377  79875:378022884  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-11T22:06:01.386763+0000  2024-06-11T22:06:01.386763+0000
> >> >> >       1286  periodic scrub scheduled @
> 2024-06-17T07:54:54.133093+0000
> >> >> > 10.9   5219448   5164591          0        0   8388698   5796048346
> >> >> > 9338312  1868      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181739392  79875:387412389  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T05:21:00.586747+0000  2024-05-26T11:10:59.780673+0000
> >> >> >        539  periodic scrub scheduled @
> 2024-06-18T15:32:59.155092+0000
> >> >> > 10.a   5219861   5163635          0        0  12582912   5841839055
> >> >> > 9387200  1916      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180205688  79875:379381294  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T12:35:05.571200+0000  2024-05-22T11:07:16.041773+0000
> >> >> >       1093  periodic deep scrub scheduled @
> >> >> 2024-06-17T05:21:40.136463+0000
> >> >> > 10.b   5217949   5217949          0        0  16777216   5935863260
> >> >> > 9462127  1881      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181655745  79875:343806807  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T22:41:28.976920+0000  2024-05-26T08:43:29.217457+0000
> >> >> >        520  periodic scrub scheduled @
> 2024-06-17T17:44:32.764093+0000
> >> >> > 10.c   5221697   5217118          0        0   4194304   6015217841
> >> >> > 9574445  1928      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180892826  79875:341490398  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-11T09:20:58.443473+0000  2024-05-30T00:13:50.306507+0000
> >> >> >        768  periodic scrub scheduled @
> 2024-06-16T19:41:21.977436+0000
> >> >> > 10.d   5217727   4908764          0        0         0   5825598519
> >> >> > 9349877  1930      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'179050347  79875:387455993  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-02T07:27:27.873631+0000  2024-05-20T03:58:30.225170+0000
> >> >> >        952  queued for deep scrub
> >> >> > 10.e   5215040   5210572          0        0   4194304   5790634469
> >> >> > 9327651  1905      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180049044  79875:377196119  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T05:03:08.521423+0000  2024-05-22T13:47:46.753131+0000
> >> >> >        527  periodic scrub scheduled @
> 2024-06-17T17:30:36.079456+0000
> >> >> > 10.f   5217202   4885952          0        0   8388630   6005274626
> >> >> > 9589178  1917      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'181953069  79875:396402453  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T12:20:49.661741+0000  2024-06-11T12:20:49.661741+0000
> >> >> >       3949  periodic scrub scheduled @
> 2024-06-17T18:37:10.974112+0000
> >> >> > 10.10  5214985   5157391          0        0   4194394   6275024632
> >> >> > 9721505  1922      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181801708  79875:395055895  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T01:49:47.831191+0000  2024-06-11T01:49:47.831191+0000
> >> >> >       9579  periodic scrub scheduled @
> 2024-06-16T11:40:28.305905+0000
> >> >> > 10.11  5219236   4983134          0        0   4194304   5787580073
> >> >> > 9316893  1901      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'181011816  79875:343488767  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-03T00:06:14.862209+0000  2024-05-27T00:05:56.468361+0000
> >> >> >        960  queued for scrub
> >> >> > 10.12  5214103   5182770          0        0        90   5970750188
> >> >> > 9505613  1891      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180210613  79875:341042616  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-12T06:39:48.252419+0000  2024-05-28T01:32:03.381577+0000
> >> >> >        527  periodic scrub scheduled @
> 2024-06-18T11:23:33.994438+0000
> >> >> > 10.13  5214580   5214580          0        0   8388642   5793432968
> >> >> > 9335940  1957      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182394614  79875:393423214  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-10T17:52:04.690203+0000  2024-05-22T03:39:02.453336+0000
> >> >> >       1151  periodic deep scrub scheduled @
> >> >> 2024-06-16T00:28:37.682765+0000
> >> >> > 10.14  5218591   5218591          0        0   5053325   6046268958
> >> >> > 9582481  1915      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'184576050  79875:425213800  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-03T05:36:07.656965+0000  2024-05-28T22:46:16.814318+0000
> >> >> >        985  queued for scrub
> >> >> > 10.15  5215919   4907184          0        0  11939335   5752284246
> >> >> > 9285870  1889      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'179713036  79875:337938973  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T14:06:50.138992+0000  2024-05-24T07:52:22.522455+0000
> >> >> >        786  periodic scrub scheduled @
> 2024-06-17T13:00:48.869790+0000
> >> >> > 10.16  5221091   5221091          0        0   8388608   5966308565
> >> >> > 9487816  1916      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'183445984  79875:399413009  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-03T02:01:53.649417+0000  2024-06-03T02:01:53.649417+0000
> >> >> >       6935  queued for scrub
> >> >> > 10.17  5218069   5194732          0        0   4194304   6231638001
> >> >> > 9735915  1890      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182390128  79875:340719915  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-12T06:31:01.523916+0000  2024-06-12T06:31:01.523916+0000
> >> >> >       1272  periodic scrub scheduled @
> 2024-06-17T16:33:45.723673+0000
> >> >> > 10.18  5214253   5159897          0        0         0   6128506469
> >> >> > 9673261  1886      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179754208  79875:389150273  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T02:53:31.027154+0000  2024-06-12T02:53:31.027154+0000
> >> >> >       1308  periodic scrub scheduled @
> 2024-06-17T22:19:41.359070+0000
> >> >> > 10.19  5220131   5183342          0        0         0   5869775360
> >> >> > 9460583  1953      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181750923  79875:393162989  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T19:21:24.106504+0000  2024-05-22T15:42:01.721091+0000
> >> >> >        522  periodic scrub scheduled @
> 2024-06-17T20:09:55.173174+0000
> >> >> > 10.1a  5217968   5186700          0        0   4194304   6183652625
> >> >> > 9733847  1896      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181056240  79875:394241128  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-02T15:37:49.730965+0000  2024-06-02T15:37:49.730965+0000
> >> >> >       7347  queued for scrub
> >> >> > 10.1b  5221259   5161879          0        0         0   5989030303
> >> >> > 9561035  1875      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181052305  79875:343144823  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-12T06:01:07.213894+0000  2024-06-02T00:23:51.063901+0000
> >> >> >        543  periodic scrub scheduled @
> 2024-06-17T15:51:21.700275+0000
> >> >> > 10.1c  5217034   5217034          0        0       554   5979861913
> >> >> > 9504442  1875      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180913313  79875:341328275  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-01T20:02:47.025129+0000  2024-06-01T20:02:47.025129+0000
> >> >> >       6568  queued for scrub
> >> >> > 10.1d  5219691   4997027          0        0  12582912   6081321658
> >> >> > 9599164  1918      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'179638837  79875:391808225  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T20:27:02.506715+0000  2024-05-31T22:58:06.578167+0000
> >> >> >        517  periodic scrub scheduled @
> 2024-06-16T21:05:59.442200+0000
> >> >> > 10.1e  5213369   5208776          0        0   4194304   6295629499
> >> >> > 9765317  1941      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181145210  79875:391326487  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T07:49:28.432084+0000  2024-06-12T07:49:28.432084+0000
> >> >> >       1990  periodic scrub scheduled @
> 2024-06-18T01:04:07.677468+0000
> >> >> > 10.1f  5218399   5000094          0        0   8388608   5821063844
> >> >> > 9421222  1896      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'180960296  79875:400635456  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T13:53:40.278050+0000  2024-05-22T19:39:02.345663+0000
> >> >> >        761  periodic scrub scheduled @
> 2024-06-16T22:24:39.094932+0000
> >> >> > 10.20  5216437   5185379          0        0         0   6280994122
> >> >> > 9719574  1945      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181610117  79875:391874431  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T12:58:57.754648+0000  2024-06-11T12:58:57.754648+0000
> >> >> >       2288  periodic scrub scheduled @
> 2024-06-17T23:33:35.937406+0000
> >> >> > 10.21  5217349   5217349          0        0   8388698   6348931429
> >> >> > 9822197  1867      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181724357  79875:343883762  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T18:28:34.044546+0000  2024-06-11T18:28:34.044546+0000
> >> >> >       1817  periodic scrub scheduled @
> 2024-06-17T12:18:47.312213+0000
> >> >> > 10.22  5215544   4943475          0        0   4194304   5982815232
> >> >> > 9482159  1899      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'180230331  79875:340839853  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T15:21:21.994674+0000  2024-05-30T08:01:03.301650+0000
> >> >> >        649  periodic scrub scheduled @
> 2024-06-17T01:33:20.228420+0000
> >> >> > 10.23  5217758   5217758          0        0       146   5804086623
> >> >> > 9336298  1920      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180948432  79875:388158699  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-03T05:19:32.311846+0000  2024-05-27T08:13:15.250058+0000
> >> >> >        971  queued for scrub
> >> >> > 10.24  5221505   5221505          0        0         0   6360995644
> >> >> > 9816333  1925      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182198758  79875:406847098  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-12T04:00:44.315682+0000  2024-06-12T04:00:44.315682+0000
> >> >> >       2046  periodic scrub scheduled @
> 2024-06-18T01:00:22.708069+0000
> >> >> > 10.25  5214911   5214911          0        0  12583002   5726320657
> >> >> > 9264352  1910      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180132024  79875:338167995  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T14:43:47.076906+0000  2024-05-23T12:16:34.271856+0000
> >> >> >        715  periodic scrub scheduled @
> 2024-06-17T14:42:04.951046+0000
> >> >> > 10.26  5217588   4973667          0        0         0   6040775867
> >> >> > 9598406  1927      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'188144536  79875:400277134  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-02T06:55:09.016528+0000  2024-06-02T06:55:09.016528+0000
> >> >> >       7047  queued for scrub
> >> >> > 10.27  5218490   5218490          0        0   8388630   6265942186
> >> >> > 9818746  1941      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182400328  79875:342052414  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T01:01:06.674323+0000  2024-06-12T01:01:06.674323+0000
> >> >> >       1500  periodic scrub scheduled @
> 2024-06-18T09:04:18.289464+0000
> >> >> > 10.28  5221153   5221153          0        0         0   6049095371
> >> >> > 9597182  1913      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179438777  79875:378265375  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T04:09:58.896192+0000  2024-06-01T04:32:46.011974+0000
> >> >> >        553  periodic scrub scheduled @
> 2024-06-18T00:53:21.846375+0000
> >> >> > 10.29  5214066   5214066          0        0   4194304   5767555623
> >> >> > 9316899  1937      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181568997  79875:387683159  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-10T22:48:57.757493+0000  2024-05-21T07:14:53.348341+0000
> >> >> >        938  periodic deep scrub scheduled @
> >> >> 2024-06-16T06:24:55.108695+0000
> >> >> > 10.2a  5216314   4977811          0        0   4194304   6012567810
> >> >> > 9563804  1951      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'181140734  79875:381017643  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-11T09:57:29.097708+0000  2024-05-30T03:47:22.798044+0000
> >> >> >       1169  periodic scrub scheduled @
> 2024-06-16T10:08:48.053531+0000
> >> >> > 10.2b  5219620   5219620          0        0   8388608   5829917912
> >> >> > 9412426  1889      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180465782  79875:342263856  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T08:30:08.941782+0000  2024-05-24T22:00:29.535205+0000
> >> >> >        636  periodic scrub scheduled @
> 2024-06-17T09:00:09.155675+0000
> >> >> > 10.2c  5214087   5214087          0        0         0   5874182922
> >> >> > 9375708  1871      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180616041  79875:341244736  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-10T23:10:08.184410+0000  2024-05-27T17:41:07.583255+0000
> >> >> >       1268  periodic scrub scheduled @
> 2024-06-16T00:28:52.710385+0000
> >> >> > 10.2d  5213968   4984822          0        0   4337824   5835754026
> >> >> > 9417033  1947      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'178824607  79875:386427308  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-12T09:27:43.774331+0000  2024-05-27T05:33:45.293266+0000
> >> >> >        494  periodic scrub scheduled @
> 2024-06-18T10:19:18.219366+0000
> >> >> > 10.2e  5216263   4974456          0        0  12582912   6025714323
> >> >> > 9553880  1940      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'180639631  79875:378284872  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T01:19:42.477063+0000  2024-05-31T08:42:41.293335+0000
> >> >> >        526  periodic scrub scheduled @
> 2024-06-17T09:53:24.981953+0000
> >> >> > 10.2f  5220057   5220057          0        0  13839831   5804027574
> >> >> > 9370862  1882      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180961915  79875:394513378  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-11T19:58:05.145147+0000  2024-05-24T20:18:32.301831+0000
> >> >> >        535  periodic scrub scheduled @
> 2024-06-18T03:16:19.773648+0000
> >> >> > 10.30  5217090   4955469          0        0  12582934   6438521431
> >> >> > 9924825  1869      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     2h  79875'181488488  79875:404940886  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T07:16:18.095439+0000  2024-06-12T07:16:18.095439+0000
> >> >> >       2190  periodic scrub scheduled @
> 2024-06-18T10:42:55.574570+0000
> >> >> > 10.31  5218176   5218176          0        0   4194304   6215016361
> >> >> > 9752698  1897      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181622879  79875:343554151  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T09:00:15.941065+0000  2024-06-12T09:00:15.941065+0000
> >> >> >       1806  periodic scrub scheduled @
> 2024-06-18T02:46:00.756777+0000
> >> >> > 10.32  5223171   4704732          0        0   8388608   6205669256
> >> >> > 9800320  1920      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180911263  79875:342754128  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T19:49:09.010583+0000  2024-06-11T19:49:09.010583+0000
> >> >> >       1664  periodic scrub scheduled @
> 2024-06-17T07:07:21.062789+0000
> >> >> > 10.33  5215341   4525338          0        0   4194304   6066965588
> >> >> > 9635403  1888      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181467497  79875:402958597  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T01:59:23.933458+0000  2024-05-31T17:01:38.620377+0000
> >> >> >        512  periodic scrub scheduled @
> 2024-06-17T22:07:45.026120+0000
> >> >> > 10.34  5217342   5217342          0        0   8388608   5999920961
> >> >> > 9570527  1956      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182013407  79875:419219483  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T09:10:17.390128+0000  2024-06-01T15:40:23.241769+0000
> >> >> >        601  periodic scrub scheduled @
> 2024-06-18T09:56:03.061859+0000
> >> >> > 10.35  5222114   5222114          0        0   8388608   6021420075
> >> >> > 9571300  1952      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180948342  79875:338598678  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T22:59:17.017317+0000  2024-05-30T15:59:15.553266+0000
> >> >> >        539  periodic scrub scheduled @
> 2024-06-17T02:41:31.718321+0000
> >> >> > 10.36  5223657   5223657          0        0   4331271   5844578100
> >> >> > 9347405  1903      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'183363403  79875:408484683  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T18:37:46.653097+0000  2024-05-25T14:13:51.648712+0000
> >> >> >        553  periodic scrub scheduled @
> 2024-06-17T12:58:00.134657+0000
> >> >> > 10.37  5219248   5219248          0        0   4194304   6072826085
> >> >> > 9579049  1936      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182091890  79875:341545087  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-11T14:19:40.744743+0000  2024-05-29T18:40:12.602405+0000
> >> >> >        771  periodic scrub scheduled @
> 2024-06-16T18:24:26.104867+0000
> >> >> > 10.38  5214022   5214022          0        0   4194858   5780805541
> >> >> > 9299038  1951      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'178809936  79875:390113817  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-02T00:55:22.046433+0000  2024-05-25T16:49:10.279003+0000
> >> >> >        931  queued for scrub
> >> >> > 10.39  5220692   4542094          0        0         0   6401799077
> >> >> > 9864164  1872      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'182881827  79875:402437400  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-12T01:42:35.239155+0000  2024-06-12T01:42:35.239155+0000
> >> >> >       1370  periodic scrub scheduled @
> 2024-06-17T10:51:19.580502+0000
> >> >> > 10.3a  5215648   4552651          0        0         0   6194465524
> >> >> > 9713558  1898      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181458295  79875:394916652  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-11T21:19:29.020889+0000  2024-06-11T21:19:29.020889+0000
> >> >> >       1227  periodic scrub scheduled @
> 2024-06-17T07:19:46.503819+0000
> >> >> > 10.3b  5218220   5218220          0        0   8388608   5951035110
> >> >> > 9490698  1878      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180815984  79875:342438023  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-03T05:03:19.483049+0000  2024-05-28T18:50:55.369009+0000
> >> >> >       1004  queued for scrub
> >> >> > 10.3c  5221615   5221615          0        0  12582912   6260894058
> >> >> > 9727358  1938      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181561627  79875:342153262  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T19:12:06.206008+0000  2024-06-11T19:12:06.206008+0000
> >> >> >       1471  periodic scrub scheduled @
> 2024-06-17T02:09:19.786893+0000
> >> >> > 10.3d  5213771   5213771          0        0         0   6215975311
> >> >> > 9699615  1946      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179354493  79875:399858588  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T04:05:10.877703+0000  2024-06-11T04:05:10.877703+0000
> >> >> >       8123  periodic scrub scheduled @
> 2024-06-17T08:03:22.770035+0000
> >> >> > 10.3e  5219446   4659577          0        0   4194304   5852588566
> >> >> > 9370152  1933      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180035882  79875:391595513  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T20:35:43.624903+0000  2024-05-22T21:40:55.606650+0000
> >> >> >        520  periodic scrub scheduled @
> 2024-06-17T00:54:06.347642+0000
> >> >> > 10.3f  5219729   4889443          0        0  16777216   5925457842
> >> >> > 9484458  1948      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'181330738  79875:409113809  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-12T05:12:01.852109+0000  2024-05-27T20:03:22.704122+0000
> >> >> >        533  periodic scrub scheduled @
> 2024-06-17T13:07:19.370647+0000
> >> >> > 10.40  5222241   4445164          0        0         0   6312545191
> >> >> > 9773697  1873      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181341418  79875:393186178  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-10T20:00:37.561045+0000  2024-06-10T20:00:37.561045+0000
> >> >> >       7712  periodic scrub scheduled @
> 2024-06-16T05:59:45.317316+0000
> >> >> > 10.41  5215195   5183508          0        0   4194326   6000770355
> >> >> > 9529537  1938      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181494334  79875:343712275  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-12T04:18:58.693597+0000  2024-06-01T11:05:59.370690+0000
> >> >> >        539  periodic scrub scheduled @
> 2024-06-17T15:34:34.997281+0000
> >> >> > 10.42  5219564   4681323          0        0  12582912   5822745109
> >> >> > 9331625  1916      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'179811520  79875:340958258  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-02T19:58:06.439770+0000  2024-05-21T18:28:28.794738+0000
> >> >> >      11044  queued for deep scrub
> >> >> > 10.43  5219412   4599458          0        0       594   5737440613
> >> >> > 9239869  1950      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180634536  79875:388879991  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-01T18:13:17.068416+0000  2024-05-20T16:32:20.302230+0000
> >> >> >       1002  queued for deep scrub
> >> >> > 10.44  5216008   5216008          0        0   4194304   6270195298
> >> >> > 9786806  1944      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182088681  79875:406175480  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T00:19:01.240405+0000  2024-06-12T00:19:01.240405+0000
> >> >> >       1593  periodic scrub scheduled @
> 2024-06-17T12:19:53.888923+0000
> >> >> > 10.45  5216238   4694313          0        0         0   5714807184
> >> >> > 9223666  1944      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'179709878  79875:338307727  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-02T00:39:50.657116+0000  2024-05-21T02:52:14.376959+0000
> >> >> >        958  queued for deep scrub
> >> >> > 10.46  5217950   5217950          0        0   9729074   5945415128
> >> >> > 9499020  1915      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'183726716  79875:395024265  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T14:56:28.875949+0000  2024-05-29T04:19:53.409918+0000
> >> >> >        724  periodic scrub scheduled @
> 2024-06-16T17:42:09.428674+0000
> >> >> > 10.47  5219125   5219125          0        0   4194304   5775128028
> >> >> > 9299099  1864      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181328922  79875:340510799  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-11T17:58:16.753327+0000  2024-05-23T14:24:25.876796+0000
> >> >> >        652  periodic scrub scheduled @
> 2024-06-17T20:23:31.713805+0000
> >> >> > 10.48  5216471   5185269          0        0   4194326   6352053972
> >> >> > 9843580  1929      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179547710  79875:378125137  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T23:22:17.435614+0000  2024-06-11T23:22:17.435614+0000
> >> >> >       1379  periodic scrub scheduled @
> 2024-06-17T19:10:56.456654+0000
> >> >> > 10.49  5217119   4580870          0        0   4194304   5971558944
> >> >> > 9526047  1879      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'182489297  79875:389298429  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T08:57:42.791265+0000  2024-05-29T07:13:57.867295+0000
> >> >> >        959  periodic scrub scheduled @
> 2024-06-17T06:44:34.852894+0000
> >> >> > 10.4a  5217577   4909174          0        0   4194304   5960086599
> >> >> > 9511657  1910      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'180982438  79875:381432369  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-12T02:31:42.576663+0000  2024-06-01T09:08:07.860775+0000
> >> >> >        480  periodic scrub scheduled @
> 2024-06-18T01:57:37.976185+0000
> >> >> > 10.4b  5220270   5220270          0        0    159808   6313986186
> >> >> > 9763672  1934      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181279214  79875:342509944  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-11T11:03:38.997373+0000  2024-06-11T11:03:38.997373+0000
> >> >> >       3970  periodic scrub scheduled @
> 2024-06-17T17:04:16.317699+0000
> >> >> > 10.4c  5217864   5217864          0        0   4194304   6104537460
> >> >> > 9679170  1948      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181081348  79875:341937760  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T06:09:49.667680+0000  2024-06-01T13:06:12.938513+0000
> >> >> >        520  periodic scrub scheduled @
> 2024-06-18T11:39:03.277402+0000
> >> >> > 10.4d  5217604   4905278          0        0   8388608   6250536027
> >> >> > 9765571  1917      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'179562016  79875:387768172  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T13:38:28.479053+0000  2024-06-11T13:38:28.479053+0000
> >> >> >       2004  periodic scrub scheduled @
> 2024-06-17T15:12:47.615309+0000
> >> >> > 10.4e  5220202   5220202          0        0  12215022   5781813383
> >> >> > 9291138  1940      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180360470  79875:377712114  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-01T22:06:12.347760+0000  2024-05-25T19:34:49.805104+0000
> >> >> >        982  queued for scrub
> >> >> > 10.4f  5215243   4561466          0        0         0   6099447266
> >> >> > 9605810  1930      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181794065  79875:396660094  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-02T13:35:21.451128+0000  2024-06-02T13:35:21.451128+0000
> >> >> >       7316  queued for scrub
> >> >> > 10.50  5215219   5184003          0        0         0   6151789627
> >> >> > 9674779  1871      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181695151  79875:405060298  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T16:42:00.221013+0000  2024-06-11T16:42:00.221013+0000
> >> >> >       2385  periodic scrub scheduled @
> 2024-06-17T10:54:13.580253+0000
> >> >> > 10.51  5218666   5218666          0        0   8388608   5774655330
> >> >> > 9331373  1920      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180711664  79875:342978462  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T14:31:51.402844+0000  2024-05-24T11:26:15.539677+0000
> >> >> >        730  periodic scrub scheduled @
> 2024-06-17T20:48:06.862863+0000
> >> >> > 10.52  5218435   5187050          0        0   4194304   6375954134
> >> >> > 9843553  1935      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180834874  79875:341502004  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T08:41:43.586833+0000  2024-06-11T08:41:43.586833+0000
> >> >> >       8368  periodic scrub scheduled @
> 2024-06-16T08:59:08.248156+0000
> >> >> > 10.53  5216822   5216822          0        0  10001995   5786896551
> >> >> > 9368323  1928      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181337168  79875:402047940  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-12T04:54:21.147742+0000  2024-05-23T23:02:36.154347+0000
> >> >> >        557  periodic scrub scheduled @
> 2024-06-17T11:39:03.144049+0000
> >> >> > 10.54  5213949   4567626          0        0   8388608   6095478377
> >> >> > 9664814  1944      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'182179986  79875:421118572  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T09:19:30.470319+0000  2024-06-01T17:56:32.089348+0000
> >> >> >        551  periodic scrub scheduled @
> 2024-06-18T09:51:50.591138+0000
> >> >> > 10.55  5217389   5217389          0        0   4194948   5905845521
> >> >> > 9403344  1950      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'183481821  79875:341223864  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T15:31:05.524698+0000  2024-05-29T11:50:10.322128+0000
> >> >> >        584  periodic scrub scheduled @
> 2024-06-17T04:28:55.809357+0000
> >> >> > 10.56  5219142   5219142          0        0         0   6311549101
> >> >> > 9773102  1897      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'184062267  79875:408792597  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-10T22:17:28.523159+0000  2024-06-10T22:17:28.523159+0000
> >> >> >       8211  periodic scrub scheduled @
> 2024-06-17T03:16:39.913869+0000
> >> >> > 10.57  5219035   4894148          0        0       180   5862209193
> >> >> > 9396761  1895      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'181886155  79875:340148291  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-10T22:33:19.364284+0000  2024-05-28T12:05:26.762682+0000
> >> >> >        951  periodic scrub scheduled @
> 2024-06-16T00:55:55.641233+0000
> >> >> > 10.58  5218318   5185905          0        0   9878587   5911568798
> >> >> > 9459707  1875      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179109213  79875:390200207  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T13:08:37.380203+0000  2024-05-30T10:19:35.091370+0000
> >> >> >        578  periodic scrub scheduled @
> 2024-06-17T15:53:56.734391+0000
> >> >> > 10.59  5218315   5185838          0        0   4194304   5727052522
> >> >> > 9214424  1875      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181700141  79875:400297310  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T11:15:00.221765+0000  2024-05-21T11:28:39.056382+0000
> >> >> >        682  periodic deep scrub scheduled @
> >> >> 2024-06-16T13:00:43.750185+0000
> >> >> > 10.5a  5218301   5218301          0        0   4194304   5833097049
> >> >> > 9365354  1942      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180699909  79875:393344045  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T01:50:51.954645+0000  2024-05-25T00:11:49.917556+0000
> >> >> >        497  periodic scrub scheduled @
> 2024-06-17T12:05:07.871109+0000
> >> >> > 10.5b  5215134   5215134          0        0   4194802   6070043467
> >> >> > 9585134  1892      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181149500  79875:342539489  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-12T08:19:32.265112+0000  2024-06-02T11:33:23.094569+0000
> >> >> >        714  periodic scrub scheduled @
> 2024-06-17T23:10:14.905316+0000
> >> >> > 10.5c  5221503   5221503          0        0   4194304   5989312430
> >> >> > 9524556  1939      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180748682  79875:340750417  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T09:37:59.483197+0000  2024-05-28T03:41:58.475567+0000
> >> >> >       1021  periodic scrub scheduled @
> 2024-06-17T16:20:33.624394+0000
> >> >> > 10.5d  5222202   5222202          0        0   8388608   6031557641
> >> >> > 9520790  1928      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'179508607  79875:399919359  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-11T17:10:33.165278+0000  2024-05-30T05:53:10.772736+0000
> >> >> >        575  periodic scrub scheduled @
> 2024-06-17T11:34:40.016966+0000
> >> >> > 10.5e  5212770   4506643          0        0         0   5830165136
> >> >> > 9388228  1938      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180002484  79875:391844220  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T22:32:49.026280+0000  2024-05-24T15:31:50.362016+0000
> >> >> >        523  periodic scrub scheduled @
> 2024-06-18T06:19:36.998416+0000
> >> >> > 10.5f  5221504   5221504          0        0   4194394   5807975316
> >> >> > 9387383  1879      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181161493  79875:408530090  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T15:10:33.261745+0000  2024-05-25T06:37:01.629711+0000
> >> >> >        845  periodic scrub scheduled @
> 2024-06-18T01:25:00.005508+0000
> >> >> > 10.60  5217784   5186615          0        0   4194304   6208553838
> >> >> > 9749044  1962      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181569072  79875:392211608  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T21:44:35.372861+0000  2024-06-11T21:44:35.372861+0000
> >> >> >       1505  periodic scrub scheduled @
> 2024-06-17T19:37:54.401372+0000
> >> >> > 10.61  5221292   4586636          0        0   8388608   5766320671
> >> >> > 9280184  1910      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180945714  79875:344272511  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T16:51:25.519891+0000  2024-05-24T17:37:21.814566+0000
> >> >> >        566  periodic scrub scheduled @
> 2024-06-18T02:56:43.473028+0000
> >> >> > 10.62  5218498   4699002          0        0   4194466   6027128219
> >> >> > 9555111  1884      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180387699  79875:341929273  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T03:10:08.547987+0000  2024-06-01T02:47:43.109035+0000
> >> >> >        509  periodic scrub scheduled @
> 2024-06-17T23:58:36.771468+0000
> >> >> > 10.63  5219101   5219101          0        0         0   6040382110
> >> >> > 9500040  1885      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181623402  79875:389756489  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-02T23:50:14.947543+0000  2024-06-02T23:50:14.947543+0000
> >> >> >      13925  queued for scrub
> >> >> > 10.64  5217025   5217025          0        0   2277792   6222337345
> >> >> > 9701950  1866      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181853168  79875:406174284  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-02T04:57:41.872723+0000  2024-06-02T04:57:41.872723+0000
> >> >> >       6394  queued for scrub
> >> >> > 10.65  5217908   4590550          0        0   4194326   6223151288
> >> >> > 9709345  1916      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181196382  79875:340107600  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T15:53:28.725001+0000  2024-06-11T15:53:28.725001+0000
> >> >> >       1343  periodic scrub scheduled @
> 2024-06-17T11:59:39.917825+0000
> >> >> > 10.66  5218911   4893654          0        0   4194304   5815703055
> >> >> > 9338987  1939      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'183062531  79875:395193073  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-12T01:10:56.919233+0000  2024-05-25T04:50:04.300293+0000
> >> >> >        590  periodic scrub scheduled @
> 2024-06-18T11:39:55.201903+0000
> >> >> > 10.67  5217539   4587208          0        0   4194304   6095526883
> >> >> > 9640323  1955      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181875315  79875:342878178  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T07:58:41.522730+0000  2024-06-01T07:20:59.951966+0000
> >> >> >        553  periodic scrub scheduled @
> 2024-06-18T05:40:51.996147+0000
> >> >> > 10.68  5215852   4505902          0        0   8388608   6093858586
> >> >> > 9596421  1926      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'179490643  79875:379632203  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T16:02:14.995195+0000  2024-05-30T22:02:42.335910+0000
> >> >> >        526  periodic scrub scheduled @
> 2024-06-17T02:53:33.980471+0000
> >> >> > 10.69  5216596   5216596          0        0   4194304   5833268044
> >> >> > 9400414  1899      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181876551  79875:387059449  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T17:00:57.111851+0000  2024-05-23T07:05:05.989427+0000
> >> >> >        572  periodic scrub scheduled @
> 2024-06-17T18:52:42.850336+0000
> >> >> > 10.6a  5214106   4497370          0        0         0   5745504618
> >> >> > 9296938  1869      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180109669  79875:381871149  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T00:27:45.155477+0000  2024-05-25T09:22:19.570011+0000
> >> >> >        494  periodic scrub scheduled @
> 2024-06-18T07:00:30.017009+0000
> >> >> > 10.6b  5219339   5219339          0        0         0   5801390847
> >> >> > 9370084  1957      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180041524  79875:342270608  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-12T03:01:39.358464+0000  2024-05-23T01:04:22.522035+0000
> >> >> >        488  periodic scrub scheduled @
> 2024-06-17T19:03:41.161095+0000
> >> >> > 10.6c  5217917   5217917          0        0       554   6100979305
> >> >> > 9612250  1955      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180968918  79875:341488318  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T18:47:35.969345+0000  2024-05-31T11:54:59.559316+0000
> >> >> >        588  periodic scrub scheduled @
> 2024-06-18T06:02:38.982765+0000
> >> >> > 10.6d  5221048   4550388          0        0   4194304   6024067862
> >> >> > 9621947  1927      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'179373269  79875:388095690  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T03:26:37.931310+0000  2024-05-31T04:10:41.991457+0000
> >> >> >        502  periodic scrub scheduled @
> 2024-06-18T08:52:34.478975+0000
> >> >> > 10.6e  5215550   4888179          0        0   8388608   6110450001
> >> >> > 9666892  1906      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'180885974  79875:380260263  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-11T20:18:23.657558+0000  2024-06-11T20:18:23.657558+0000
> >> >> >       1218  periodic scrub scheduled @
> 2024-06-18T07:12:57.858451+0000
> >> >> > 10.6f  5215850   4566820          0        0         0   6107146788
> >> >> > 9634911  1923      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181752773  79875:396708021  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-10T17:32:51.735004+0000  2024-06-10T17:32:51.735004+0000
> >> >> >      10119  periodic scrub scheduled @
> 2024-06-17T02:47:23.675814+0000
> >> >> > 10.70  5218211   4672319          0        0   4194304   5793441438
> >> >> > 9371563  1881      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180405012  79875:404578822  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-03T04:46:35.130293+0000  2024-05-22T07:50:20.898441+0000
> >> >> >        969  queued for deep scrub
> >> >> > 10.71  5218299   4599312          0        0   4194304   5807950891
> >> >> > 9349720  1906      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180690682  79875:344108361  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-12T03:18:16.645927+0000  2024-05-23T09:32:56.055703+0000
> >> >> >        487  periodic scrub scheduled @
> 2024-06-17T13:47:03.236787+0000
> >> >> > 10.72  5223470   4922347          0        0   8388608   6416468325
> >> >> > 9922777  1958      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'180797342  79875:342356916  [0,1,2]p0  [1,2]p1
> >> >> >  2024-06-12T05:42:47.545762+0000  2024-06-12T05:42:47.545762+0000
> >> >> >       1307  periodic scrub scheduled @
> 2024-06-18T13:20:41.826465+0000
> >> >> > 10.73  5218301   4520651          0        0         0   6293832301
> >> >> > 9738943  1953      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'182246986  79875:403792984  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-11T20:59:02.079157+0000  2024-06-11T20:59:02.079157+0000
> >> >> >       1399  periodic scrub scheduled @
> 2024-06-17T01:00:32.819334+0000
> >> >> > 10.74  5219061   4595112          0        0         0   5974775781
> >> >> > 9469653  1943      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181947609  79875:420571221  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-03T02:17:19.687482+0000  2024-05-28T08:43:38.825954+0000
> >> >> >        927  queued for scrub
> >> >> > 10.75  5220994   4596634          0        0  25165824   5822315216
> >> >> > 9362548  1877      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'180376305  79875:338847231  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-03T05:52:28.743255+0000  2024-05-24T05:11:00.211501+0000
> >> >> >        981  queued for scrub
> >> >> > 10.76  5222111   4892295          0        0         0   6067998823
> >> >> > 9574254  1935      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     2h  79875'183871180  79875:409218100  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-03T04:14:04.513004+0000  2024-06-03T04:14:04.513004+0000
> >> >> >       6987  queued for scrub
> >> >> > 10.77  5219020   4574683          0        0         0   5889074277
> >> >> > 9460824  1896      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181614825  79875:342558162  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-12T08:07:38.169523+0000  2024-05-27T15:15:35.587972+0000
> >> >> >        536  periodic scrub scheduled @
> 2024-06-17T18:29:44.366900+0000
> >> >> > 10.78  5220442   4693235          0        0   8388608   5879566566
> >> >> > 9395501  1923      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'179031456  79875:391462415  [1,0,2]p1  [1,2]p1
> >> >> >  2024-06-02T09:32:42.834344+0000  2024-05-27T10:41:23.822124+0000
> >> >> >        994  queued for scrub
> >> >> > 10.79  5219734   4702117          0        0   4194304   6129933545
> >> >> > 9646669  1926      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'182574416  79875:402249812  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-12T05:52:02.959814+0000  2024-06-01T00:53:49.548055+0000
> >> >> >        555  periodic scrub scheduled @
> 2024-06-17T17:48:00.189475+0000
> >> >> > 10.7a  5215303   4526846          0        0         0   6316171107
> >> >> > 9817029  1932      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181765231  79875:395877057  [2,0,1]p2  [2,1]p2
> >> >> >  2024-06-12T02:23:42.692096+0000  2024-06-12T02:23:42.692096+0000
> >> >> >       1457  periodic scrub scheduled @
> 2024-06-17T22:13:46.037000+0000
> >> >> > 10.7b  5217213   4606816          0        0   4194304   6226859283
> >> >> > 9718621  1952      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'181413541  79875:343968658  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T23:52:28.414752+0000  2024-06-11T23:52:28.414752+0000
> >> >> >       1308  periodic scrub scheduled @
> 2024-06-17T18:52:39.473049+0000
> >> >> > 10.7c  5215075   5215075          0        0   4194304   6013877208
> >> >> > 9602345  1867      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'180991860  79875:341775615  [0,2,1]p0  [1,2]p1
> >> >> >  2024-06-11T23:30:40.614412+0000  2024-05-31T14:10:30.542603+0000
> >> >> >        503  periodic scrub scheduled @
> 2024-06-17T02:30:15.591036+0000
> >> >> > 10.7d  5220700   4569720          0        0   5837609   6258736025
> >> >> > 9764836  1897      3000
> >> >> active+undersized+degraded+remapped+backfilling
> >> >> >     3h  79875'179741885  79875:401448508  [2,1,0]p2  [2,1]p2
> >> >> >  2024-06-12T04:45:05.030905+0000  2024-06-12T04:45:05.030905+0000
> >> >> >       1565  periodic scrub scheduled @
> 2024-06-17T04:54:42.039122+0000
> >> >> > 10.7e  5217113   5217113          0        0  16777216   6072081676
> >> >> > 9598390  1902      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'181040361  79875:391032348  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T17:37:26.589968+0000  2024-06-11T17:37:26.589968+0000
> >> >> >       1613  periodic scrub scheduled @
> 2024-06-17T05:06:05.391340+0000
> >> >> > 10.7f  5218719   5218719          0        0   9909061   6130469539
> >> >> > 9657824  1943      3000
> >> >> active+undersized+degraded+remapped+backfill_wait
> >> >> >     3h  79875'182283514  79875:408550888  [1,2,0]p1  [1,2]p1
> >> >> >  2024-06-11T13:05:03.552526+0000  2024-06-11T13:05:03.552526+0000
> >> >> >       1796  periodic scrub scheduled @
> 2024-06-17T09:57:04.545523+0000
> >> >> >
> >> >> > * NOTE: Omap statistics are gathered during deep scrub and may be
> >> >> > inaccurate soon afterwards depending on utilization. See
> >> >> >
> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics
> >> for
> >> >> > further details.
> >> >> >
> >> >> >
> >> >> >
> >> >> > sudo ceph osd df
> >> >> > ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP
> >> META
> >> >> >   AVAIL    %USE   VAR   PGS  STATUS
> >> >> >  0    ssd   3.63869   1.00000  3.6 TiB   82 GiB   50 MiB    40 GiB
> >>  42
> >> >> GiB
> >> >> >  3.6 TiB   2.21  0.03    0      up
> >> >> >  1    ssd   3.63869   1.00000  3.6 TiB  2.0 TiB   16 GiB  1021 GiB
> >> 1.0
> >> >> TiB
> >> >> >  1.6 TiB  55.41  0.75  161      up
> >> >> >  2    ssd   3.63869   1.00000  3.6 TiB  3.0 TiB   16 GiB   1.1 TiB
> >> 1.9
> >> >> TiB
> >> >> >  676 GiB  81.86  1.11  161      up
> >> >> >
> >> >> >
> >> >> > sudo ceph df detail
> >> >> > --- RAW STORAGE ---
> >> >> > CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
> >> >> > hdd    1.2 PiB  308 TiB  871 TiB   871 TiB      73.86
> >> >> > ssd     11 TiB  5.8 TiB  5.1 TiB   5.1 TiB      46.51
> >> >> > TOTAL  1.2 PiB  314 TiB  876 TiB   876 TiB      73.61
> >> >> >
> >> >> > --- POOLS ---
> >> >> > POOL      ID   PGS   STORED   (DATA)   (OMAP)  OBJECTS     USED
> >>  (DATA)
> >> >> > (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED
> >> COMPR
> >> >> >  UNDER COMPR
> >> >> > .mgr       1     1  827 MiB  827 MiB      0 B      208  1.6 GiB
> 1.6
> >> GiB
> >> >> >    0 B   0.11    734 GiB            N/A       50 GiB    N/A
>  0
> >> B
> >> >> >      0 B
> >> >> > metadata  10   128  1.0 TiB  647 MiB  1.0 TiB  667.90M  2.1 TiB
> 1.3
> >> GiB
> >> >> >  2.1 TiB  59.45    719 GiB            N/A      3.5 TiB    N/A
> >>  0 B
> >> >> >        0 B
> >> >> > data      11  2048  288 TiB  288 TiB      0 B  566.97M  863 TiB
> 863
> >> TiB
> >> >> >    0 B  81.10     67 TiB            N/A          N/A    N/A     3.4
> >> TiB
> >> >> >  6.8 TiB
> >> >> > ssd-data  18    32   14 GiB   14 GiB      0 B  265.17k   29 GiB
>  29
> >> GiB
> >> >> >    0 B   1.90    734 GiB            N/A          N/A    N/A
>  0
> >> B
> >> >> >      0 B
> >> >> >
> >> >> >
> >> >> > sudo ceph balancer status
> >> >> > {
> >> >> >     "active": true,
> >> >> >     "last_optimize_duration": "0:00:00.000170",
> >> >> >     "last_optimize_started": "Wed Jun 12 13:14:49 2024",
> >> >> >     "mode": "upmap",
> >> >> >     "no_optimization_needed": true,
> >> >> >     "optimize_result": "Some objects (0.172816) are degraded; try
> >> again
> >> >> > later",
> >> >> >     "plans": []
> >> >> > }
> >> >> >
> >> >> >
> >> >> >
> >> >> > [image: ariadne.ai Logo] Lars Köppel
> >> >> > Developer
> >> >> > Email: lars.koeppel@xxxxxxxxxx
> >> >> > Phone: +49 6221 5993580 <+4962215993580>
> >> >> > ariadne.ai (Germany) GmbH
> >> >> > Häusserstraße 3, 69115 Heidelberg
> >> >> > Amtsgericht Mannheim, HRB 744040
> >> >> > Geschäftsführer: Dr. Fabian Svara
> >> >> > https://ariadne.ai
> >> >> >
> >> >> >
> >> >> > On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri <
> >> anthony.datri@xxxxxxxxx>
> >> >> > wrote:
> >> >> >
> >> >> >> If you have:
> >> >> >>
> >> >> >> * pg_num too low (defaults are too low)
> >> >> >> * pg_num not a power of 2
> >> >> >> * pg_num != number of OSDs in the pool
> >> >> >> * balancer not enabled
> >> >> >>
> >> >> >> any of those might result in imbalance.
> >> >> >>
> >> >> >> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote:
> >> >> >> >
> >> >> >> > I don't have any good explanation at this point. Can you share
> some
> >> >> more
> >> >> >> information like:
> >> >> >> >
> >> >> >> > ceph pg ls-by-pool <cephfs_metadata>
> >> >> >> > ceph osd df (for the relevant OSDs)
> >> >> >> > ceph df
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Eugen
> >> >> >> >
> >> >> >> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >> >> >> >
> >> >> >> >> Since my last update the size of the largest OSD increased by
> 0.4
> >> TiB
> >> >> >> while
> >> >> >> >> the smallest one only increased by 0.1 TiB. How is this
> possible?
> >> >> >> >>
> >> >> >> >> Because the metadata pool reported to have only 900MB space
> left,
> >> I
> >> >> >> stopped
> >> >> >> >> the hot-standby MDS. This gave me 8GB back but these filled up
> in
> >> the
> >> >> >> last
> >> >> >> >> 2h.
> >> >> >> >> I think I have to zap the next OSD because the filesystem is
> >> getting
> >> >> >> read
> >> >> >> >> only...
> >> >> >> >>
> >> >> >> >> How is it possible that an OSD has over 1 TiB less data on it
> >> after a
> >> >> >> >> rebuild? And how is it possible to have so different sizes of
> >> OSDs?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> [image: ariadne.ai Logo] Lars Köppel
> >> >> >> >> Developer
> >> >> >> >> Email: lars.koeppel@xxxxxxxxxx
> >> >> >> >> Phone: +49 6221 5993580 <+4962215993580>
> >> >> >> >> ariadne.ai (Germany) GmbH
> >> >> >> >> Häusserstraße 3, 69115 Heidelberg
> >> >> >> >> Amtsgericht Mannheim, HRB 744040
> >> >> >> >> Geschäftsführer: Dr. Fabian Svara
> >> >> >> >> https://ariadne.ai
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel <
> >> lars.koeppel@xxxxxxxxxx
> >> >> >
> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >>> Only in warning mode. And there were no PG splits or merges in
> >> the
> >> >> >> last 2
> >> >> >> >>> month.
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> [image: ariadne.ai Logo] Lars Köppel
> >> >> >> >>> Developer
> >> >> >> >>> Email: lars.koeppel@xxxxxxxxxx
> >> >> >> >>> Phone: +49 6221 5993580 <+4962215993580>
> >> >> >> >>> ariadne.ai (Germany) GmbH
> >> >> >> >>> Häusserstraße 3, 69115 Heidelberg
> >> >> >> >>> Amtsgericht Mannheim, HRB 744040
> >> >> >> >>> Geschäftsführer: Dr. Fabian Svara
> >> >> >> >>> https://ariadne.ai
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx>
> >> wrote:
> >> >> >> >>>
> >> >> >> >>>> I don't think scrubs can cause this. Do you have autoscaler
> >> >> enabled?
> >> >> >> >>>>
> >> >> >> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >> >> >> >>>>
> >> >> >> >>>> > Hi,
> >> >> >> >>>> >
> >> >> >> >>>> > thank you for your response.
> >> >> >> >>>> >
> >> >> >> >>>> > I don't think this thread covers my problem, because the
> OSDs
> >> for
> >> >> >> the
> >> >> >> >>>> > metadata pool fill up at different rates. So I would think
> >> this
> >> >> is
> >> >> >> no
> >> >> >> >>>> > direct problem with the journal.
> >> >> >> >>>> > Because we had earlier problems with the journal I changed
> >> some
> >> >> >> >>>> > settings(see below). I already restarted all MDS multiple
> >> times
> >> >> but
> >> >> >> no
> >> >> >> >>>> > change here.
> >> >> >> >>>> >
> >> >> >> >>>> > The health warnings regarding cache pressure resolve
> normally
> >> >> after
> >> >> >> a
> >> >> >> >>>> > short period of time, when the heavy load on the client
> ends.
> >> >> >> Sometimes
> >> >> >> >>>> it
> >> >> >> >>>> > stays a bit longer because an rsync is running and copying
> >> data
> >> >> on
> >> >> >> the
> >> >> >> >>>> > cluster(rsync is not good at releasing the caps).
> >> >> >> >>>> >
> >> >> >> >>>> > Could it be a problem if scrubs run most of the time in the
> >> >> >> background?
> >> >> >> >>>> Can
> >> >> >> >>>> > this block any other tasks or generate new data itself?
> >> >> >> >>>> >
> >> >> >> >>>> > Best regards,
> >> >> >> >>>> > Lars
> >> >> >> >>>> >
> >> >> >> >>>> >
> >> >> >> >>>> > global                      basic
>  mds_cache_memory_limit
> >> >> >> >>>> >     17179869184
> >> >> >> >>>> > global                      advanced
> mds_max_caps_per_client
> >> >> >> >>>> >    16384
> >> >> >> >>>> > global                      advanced
> >> >> >> >>>> mds_recall_global_max_decay_threshold
> >> >> >> >>>> >    262144
> >> >> >> >>>> > global                      advanced
> >> mds_recall_max_decay_rate
> >> >> >> >>>> >    1.000000
> >> >> >> >>>> > global                      advanced
> >> >> mds_recall_max_decay_threshold
> >> >> >> >>>> >     262144
> >> >> >> >>>> > mds                         advanced
> mds_cache_trim_threshold
> >> >> >> >>>> >     131072
> >> >> >> >>>> > mds                         advanced  mds_heartbeat_grace
> >> >> >> >>>> >    120.000000
> >> >> >> >>>> > mds                         advanced
> >> mds_heartbeat_reset_grace
> >> >> >> >>>> >    7400
> >> >> >> >>>> > mds                         advanced  mds_tick_interval
> >> >> >> >>>> >    3.000000
> >> >> >> >>>> >
> >> >> >> >>>> >
> >> >> >> >>>> > [image: ariadne.ai Logo] Lars Köppel
> >> >> >> >>>> > Developer
> >> >> >> >>>> > Email: lars.koeppel@xxxxxxxxxx
> >> >> >> >>>> > Phone: +49 6221 5993580 <+4962215993580>
> >> >> >> >>>> > ariadne.ai (Germany) GmbH
> >> >> >> >>>> > Häusserstraße 3, 69115 Heidelberg
> >> >> >> >>>> > Amtsgericht Mannheim, HRB 744040
> >> >> >> >>>> > Geschäftsführer: Dr. Fabian Svara
> >> >> >> >>>> > https://ariadne.ai
> >> >> >> >>>> >
> >> >> >> >>>> >
> >> >> >> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx
> >
> >> >> wrote:
> >> >> >> >>>> >
> >> >> >> >>>> >> Hi,
> >> >> >> >>>> >>
> >> >> >> >>>> >> can you check if this thread [1] applies to your
> situation?
> >> You
> >> >> >> don't
> >> >> >> >>>> >> have multi-active MDS enabled, but maybe it's still some
> >> journal
> >> >> >> >>>> >> trimming, or maybe misbehaving clients? In your first post
> >> there
> >> >> >> were
> >> >> >> >>>> >> health warnings regarding cache pressure and cache size.
> Are
> >> >> those
> >> >> >> >>>> >> resolved?
> >> >> >> >>>> >>
> >> >> >> >>>> >> [1]
> >> >> >> >>>> >>
> >> >> >> >>>> >>
> >> >> >> >>>>
> >> >> >>
> >> >>
> >>
> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE
> >> >> >> >>>> >>
> >> >> >> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >> >> >> >>>> >>
> >> >> >> >>>> >> > Hello everyone,
> >> >> >> >>>> >> >
> >> >> >> >>>> >> > short update to this problem.
> >> >> >> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the
> >> expected
> >> >> >> size
> >> >> >> >>>> >> ~50%).
> >> >> >> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB.
> They
> >> >> >> jumped up
> >> >> >> >>>> and
> >> >> >> >>>> >> > down a lot but the higher one has now also reached
> >> 'nearfull'
> >> >> >> >>>> status. How
> >> >> >> >>>> >> > is this possible? What is going on?
> >> >> >> >>>> >> >
> >> >> >> >>>> >> > Does anyone have a solution how to fix this without
> zapping
> >> >> the
> >> >> >> OSD?
> >> >> >> >>>> >> >
> >> >> >> >>>> >> > Best regards,
> >> >> >> >>>> >> > Lars
> >> >> >> >>>> >> >
> >> >> >> >>>> >> >
> >> >> >> >>>> >> > [image: ariadne.ai Logo] Lars Köppel
> >> >> >> >>>> >> > Developer
> >> >> >> >>>> >> > Email: lars.koeppel@xxxxxxxxxx
> >> >> >> >>>> >> > Phone: +49 6221 5993580 <+4962215993580>
> >> >> >> >>>> >> > ariadne.ai (Germany) GmbH
> >> >> >> >>>> >> > Häusserstraße 3, 69115 Heidelberg
> >> >> >> >>>> >> > Amtsgericht Mannheim, HRB 744040
> >> >> >> >>>> >> > Geschäftsführer: Dr. Fabian Svara
> >> >> >> >>>> >> > https://ariadne.ai
> >> >> >> >>>> >> > _______________________________________________
> >> >> >> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> >> >> >> >>>> >> > To unsubscribe send an email to
> ceph-users-leave@xxxxxxx
> >> >> >> >>>> >>
> >> >> >> >>>> >>
> >> >> >> >>>> >> _______________________________________________
> >> >> >> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >> >> >> >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> >> >> >>>> >>
> >> >> >> >>>>
> >> >> >> >>>>
> >> >> >> >>>>
> >> >> >> >>>>
> >> >> >> >
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> >> >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >>
>
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux