Re: CephFS metadata pool size

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Downgrading isn't supported, I don't think that would be a good idea. I also don't see anything obvious standing out in the pg output. Any chance you can add more OSDs to the metadata pool to see if it stops at some point? Did the cluster usage change in any way? For example cephfs snapshots which haven't been used before or something like that?

Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:

I updated from 17.2.6 to 17.2.7 and a few hours later to 18.2.2.
Would it be an option to go back to 17.2.6?


[image: ariadne.ai Logo] Lars Köppel
Developer
Email: lars.koeppel@xxxxxxxxxx
Phone: +49 6221 5993580 <+4962215993580>
ariadne.ai (Germany) GmbH
Häusserstraße 3, 69115 Heidelberg
Amtsgericht Mannheim, HRB 744040
Geschäftsführer: Dr. Fabian Svara
https://ariadne.ai


On Wed, Jun 12, 2024 at 5:30 PM Eugen Block <eblock@xxxxxx> wrote:

Which version did you upgrade from to 18.2.2?
I can’t pin it down to a specific issue, but somewhere in the back of
my mind is something related to a new omap format or something. But
I’m really not sure at all.

Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:

> I am happy to help you with as much information as possible. I probably
> just don't know where to look for it.
> Below are the requested information. The cluster is rebuilding the
> zapped OSD at the moment. This will probably take the next few days.
>
>
> sudo ceph pg ls-by-pool metadata
> PG     OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES     OMAP_BYTES*
>  OMAP_KEYS*  LOG   LOG_DUPS  STATE
>      SINCE  VERSION          REPORTED         UP         ACTING
> SCRUB_STAMP                      DEEP_SCRUB_STAMP
> LAST_SCRUB_DURATION  SCRUB_SCHEDULING
> 10.0   5217325   4994695          0        0   4194304   5880891340
> 9393865  1885      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'180849582  79875:391519635  [0,1,2]p0  [1,2]p1
>  2024-06-11T09:08:09.829362+0000  2024-05-28T05:52:59.321589+0000
>        627  periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000
> 10.1   5214785   5193424          0        0         0   5843682713
> 9410150  1912      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180914288  79875:342746928  [2,1,0]p2  [2,1]p2
>  2024-06-01T15:56:28.927288+0000  2024-05-27T03:31:37.682966+0000
>        966  queued for scrub
> 10.2   5218432   5187168          0        0         0   6402011266
> 9812513  1874      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180970531  79875:341340204  [0,1,2]p0  [1,2]p1
>  2024-06-11T13:40:58.994256+0000  2024-06-11T13:40:58.994256+0000
>       1942  periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000
> 10.3   5217413   5217413          0        0   8388788   5766005023
> 9271787  1923      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181012233  79875:388295881  [1,0,2]p1  [1,2]p1
>  2024-06-12T00:35:56.965547+0000  2024-05-23T19:54:56.121729+0000
>        492  periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000
> 10.4   5220069   5220069          0        0  12583466   6027548724
> 9537290  1959      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181576075  79875:405295868  [1,2,0]p1  [1,2]p1
>  2024-06-11T17:47:22.923514+0000  2024-05-31T02:06:55.339574+0000
>        581  periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000
> 10.5   5216162   5211999          0        0   4194304   5941347251
> 9542764  1930      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180455793  79875:338418517  [2,1,0]p2  [2,1]p2
>  2024-06-11T22:50:16.170708+0000  2024-05-30T23:49:54.316379+0000
>        528  periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000
> 10.6   5216100   4980459          0        0   4521984   6428088514
> 9850762  1911      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'184045876  79875:396809795  [0,2,1]p0  [1,2]p1
>  2024-06-11T22:24:05.102716+0000  2024-06-11T22:24:05.102716+0000
>       1082  periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000
> 10.7   5218232   5218232          0        0   4194304   6377065363
> 9849360  1919      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182672562  79875:342449062  [1,0,2]p1  [1,2]p1
>  2024-06-11T06:22:15.689422+0000  2024-06-11T06:22:15.689422+0000
>       8225  periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000
> 10.8   5219620   5182816          0        0         0   6167304290
> 9691796  1896      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179628377  79875:378022884  [2,1,0]p2  [2,1]p2
>  2024-06-11T22:06:01.386763+0000  2024-06-11T22:06:01.386763+0000
>       1286  periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000
> 10.9   5219448   5164591          0        0   8388698   5796048346
> 9338312  1868      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181739392  79875:387412389  [2,1,0]p2  [2,1]p2
>  2024-06-12T05:21:00.586747+0000  2024-05-26T11:10:59.780673+0000
>        539  periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000
> 10.a   5219861   5163635          0        0  12582912   5841839055
> 9387200  1916      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180205688  79875:379381294  [1,2,0]p1  [1,2]p1
>  2024-06-11T12:35:05.571200+0000  2024-05-22T11:07:16.041773+0000
>       1093  periodic deep scrub scheduled @
2024-06-17T05:21:40.136463+0000
> 10.b   5217949   5217949          0        0  16777216   5935863260
> 9462127  1881      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181655745  79875:343806807  [0,1,2]p0  [1,2]p1
>  2024-06-11T22:41:28.976920+0000  2024-05-26T08:43:29.217457+0000
>        520  periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000
> 10.c   5221697   5217118          0        0   4194304   6015217841
> 9574445  1928      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180892826  79875:341490398  [2,1,0]p2  [2,1]p2
>  2024-06-11T09:20:58.443473+0000  2024-05-30T00:13:50.306507+0000
>        768  periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000
> 10.d   5217727   4908764          0        0         0   5825598519
> 9349877  1930      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'179050347  79875:387455993  [0,1,2]p0  [1,2]p1
>  2024-06-02T07:27:27.873631+0000  2024-05-20T03:58:30.225170+0000
>        952  queued for deep scrub
> 10.e   5215040   5210572          0        0   4194304   5790634469
> 9327651  1905      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180049044  79875:377196119  [2,1,0]p2  [2,1]p2
>  2024-06-12T05:03:08.521423+0000  2024-05-22T13:47:46.753131+0000
>        527  periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000
> 10.f   5217202   4885952          0        0   8388630   6005274626
> 9589178  1917      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'181953069  79875:396402453  [0,2,1]p0  [1,2]p1
>  2024-06-11T12:20:49.661741+0000  2024-06-11T12:20:49.661741+0000
>       3949  periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000
> 10.10  5214985   5157391          0        0   4194394   6275024632
> 9721505  1922      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181801708  79875:395055895  [2,0,1]p2  [2,1]p2
>  2024-06-11T01:49:47.831191+0000  2024-06-11T01:49:47.831191+0000
>       9579  periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000
> 10.11  5219236   4983134          0        0   4194304   5787580073
> 9316893  1901      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'181011816  79875:343488767  [1,0,2]p1  [1,2]p1
>  2024-06-03T00:06:14.862209+0000  2024-05-27T00:05:56.468361+0000
>        960  queued for scrub
> 10.12  5214103   5182770          0        0        90   5970750188
> 9505613  1891      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180210613  79875:341042616  [0,1,2]p0  [1,2]p1
>  2024-06-12T06:39:48.252419+0000  2024-05-28T01:32:03.381577+0000
>        527  periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000
> 10.13  5214580   5214580          0        0   8388642   5793432968
> 9335940  1957      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182394614  79875:393423214  [1,2,0]p1  [1,2]p1
>  2024-06-10T17:52:04.690203+0000  2024-05-22T03:39:02.453336+0000
>       1151  periodic deep scrub scheduled @
2024-06-16T00:28:37.682765+0000
> 10.14  5218591   5218591          0        0   5053325   6046268958
> 9582481  1915      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'184576050  79875:425213800  [2,0,1]p2  [2,1]p2
>  2024-06-03T05:36:07.656965+0000  2024-05-28T22:46:16.814318+0000
>        985  queued for scrub
> 10.15  5215919   4907184          0        0  11939335   5752284246
> 9285870  1889      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'179713036  79875:337938973  [0,2,1]p0  [1,2]p1
>  2024-06-11T14:06:50.138992+0000  2024-05-24T07:52:22.522455+0000
>        786  periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000
> 10.16  5221091   5221091          0        0   8388608   5966308565
> 9487816  1916      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'183445984  79875:399413009  [0,2,1]p0  [1,2]p1
>  2024-06-03T02:01:53.649417+0000  2024-06-03T02:01:53.649417+0000
>       6935  queued for scrub
> 10.17  5218069   5194732          0        0   4194304   6231638001
> 9735915  1890      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182390128  79875:340719915  [2,0,1]p2  [2,1]p2
>  2024-06-12T06:31:01.523916+0000  2024-06-12T06:31:01.523916+0000
>       1272  periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000
> 10.18  5214253   5159897          0        0         0   6128506469
> 9673261  1886      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179754208  79875:389150273  [2,1,0]p2  [2,1]p2
>  2024-06-12T02:53:31.027154+0000  2024-06-12T02:53:31.027154+0000
>       1308  periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000
> 10.19  5220131   5183342          0        0         0   5869775360
> 9460583  1953      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181750923  79875:393162989  [2,0,1]p2  [2,1]p2
>  2024-06-11T19:21:24.106504+0000  2024-05-22T15:42:01.721091+0000
>        522  periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000
> 10.1a  5217968   5186700          0        0   4194304   6183652625
> 9733847  1896      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181056240  79875:394241128  [0,2,1]p0  [1,2]p1
>  2024-06-02T15:37:49.730965+0000  2024-06-02T15:37:49.730965+0000
>       7347  queued for scrub
> 10.1b  5221259   5161879          0        0         0   5989030303
> 9561035  1875      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181052305  79875:343144823  [0,1,2]p0  [1,2]p1
>  2024-06-12T06:01:07.213894+0000  2024-06-02T00:23:51.063901+0000
>        543  periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000
> 10.1c  5217034   5217034          0        0       554   5979861913
> 9504442  1875      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180913313  79875:341328275  [1,2,0]p1  [1,2]p1
>  2024-06-01T20:02:47.025129+0000  2024-06-01T20:02:47.025129+0000
>       6568  queued for scrub
> 10.1d  5219691   4997027          0        0  12582912   6081321658
> 9599164  1918      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'179638837  79875:391808225  [0,2,1]p0  [1,2]p1
>  2024-06-11T20:27:02.506715+0000  2024-05-31T22:58:06.578167+0000
>        517  periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000
> 10.1e  5213369   5208776          0        0   4194304   6295629499
> 9765317  1941      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181145210  79875:391326487  [2,1,0]p2  [2,1]p2
>  2024-06-12T07:49:28.432084+0000  2024-06-12T07:49:28.432084+0000
>       1990  periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000
> 10.1f  5218399   5000094          0        0   8388608   5821063844
> 9421222  1896      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'180960296  79875:400635456  [0,2,1]p0  [1,2]p1
>  2024-06-11T13:53:40.278050+0000  2024-05-22T19:39:02.345663+0000
>        761  periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000
> 10.20  5216437   5185379          0        0         0   6280994122
> 9719574  1945      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181610117  79875:391874431  [0,1,2]p0  [1,2]p1
>  2024-06-11T12:58:57.754648+0000  2024-06-11T12:58:57.754648+0000
>       2288  periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000
> 10.21  5217349   5217349          0        0   8388698   6348931429
> 9822197  1867      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181724357  79875:343883762  [0,1,2]p0  [1,2]p1
>  2024-06-11T18:28:34.044546+0000  2024-06-11T18:28:34.044546+0000
>       1817  periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000
> 10.22  5215544   4943475          0        0   4194304   5982815232
> 9482159  1899      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'180230331  79875:340839853  [2,0,1]p2  [2,1]p2
>  2024-06-11T15:21:21.994674+0000  2024-05-30T08:01:03.301650+0000
>        649  periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000
> 10.23  5217758   5217758          0        0       146   5804086623
> 9336298  1920      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180948432  79875:388158699  [1,2,0]p1  [1,2]p1
>  2024-06-03T05:19:32.311846+0000  2024-05-27T08:13:15.250058+0000
>        971  queued for scrub
> 10.24  5221505   5221505          0        0         0   6360995644
> 9816333  1925      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182198758  79875:406847098  [0,2,1]p0  [1,2]p1
>  2024-06-12T04:00:44.315682+0000  2024-06-12T04:00:44.315682+0000
>       2046  periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000
> 10.25  5214911   5214911          0        0  12583002   5726320657
> 9264352  1910      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180132024  79875:338167995  [1,2,0]p1  [1,2]p1
>  2024-06-11T14:43:47.076906+0000  2024-05-23T12:16:34.271856+0000
>        715  periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000
> 10.26  5217588   4973667          0        0         0   6040775867
> 9598406  1927      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'188144536  79875:400277134  [2,0,1]p2  [2,1]p2
>  2024-06-02T06:55:09.016528+0000  2024-06-02T06:55:09.016528+0000
>       7047  queued for scrub
> 10.27  5218490   5218490          0        0   8388630   6265942186
> 9818746  1941      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182400328  79875:342052414  [1,2,0]p1  [1,2]p1
>  2024-06-12T01:01:06.674323+0000  2024-06-12T01:01:06.674323+0000
>       1500  periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000
> 10.28  5221153   5221153          0        0         0   6049095371
> 9597182  1913      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179438777  79875:378265375  [1,2,0]p1  [1,2]p1
>  2024-06-12T04:09:58.896192+0000  2024-06-01T04:32:46.011974+0000
>        553  periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000
> 10.29  5214066   5214066          0        0   4194304   5767555623
> 9316899  1937      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181568997  79875:387683159  [0,2,1]p0  [1,2]p1
>  2024-06-10T22:48:57.757493+0000  2024-05-21T07:14:53.348341+0000
>        938  periodic deep scrub scheduled @
2024-06-16T06:24:55.108695+0000
> 10.2a  5216314   4977811          0        0   4194304   6012567810
> 9563804  1951      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'181140734  79875:381017643  [2,1,0]p2  [2,1]p2
>  2024-06-11T09:57:29.097708+0000  2024-05-30T03:47:22.798044+0000
>       1169  periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000
> 10.2b  5219620   5219620          0        0   8388608   5829917912
> 9412426  1889      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180465782  79875:342263856  [1,0,2]p1  [1,2]p1
>  2024-06-12T08:30:08.941782+0000  2024-05-24T22:00:29.535205+0000
>        636  periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000
> 10.2c  5214087   5214087          0        0         0   5874182922
> 9375708  1871      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180616041  79875:341244736  [1,2,0]p1  [1,2]p1
>  2024-06-10T23:10:08.184410+0000  2024-05-27T17:41:07.583255+0000
>       1268  periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000
> 10.2d  5213968   4984822          0        0   4337824   5835754026
> 9417033  1947      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'178824607  79875:386427308  [2,0,1]p2  [2,1]p2
>  2024-06-12T09:27:43.774331+0000  2024-05-27T05:33:45.293266+0000
>        494  periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000
> 10.2e  5216263   4974456          0        0  12582912   6025714323
> 9553880  1940      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'180639631  79875:378284872  [1,0,2]p1  [1,2]p1
>  2024-06-12T01:19:42.477063+0000  2024-05-31T08:42:41.293335+0000
>        526  periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000
> 10.2f  5220057   5220057          0        0  13839831   5804027574
> 9370862  1882      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180961915  79875:394513378  [1,0,2]p1  [1,2]p1
>  2024-06-11T19:58:05.145147+0000  2024-05-24T20:18:32.301831+0000
>        535  periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000
> 10.30  5217090   4955469          0        0  12582934   6438521431
> 9924825  1869      3000
active+undersized+degraded+remapped+backfill_wait
>     2h  79875'181488488  79875:404940886  [2,1,0]p2  [2,1]p2
>  2024-06-12T07:16:18.095439+0000  2024-06-12T07:16:18.095439+0000
>       2190  periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000
> 10.31  5218176   5218176          0        0   4194304   6215016361
> 9752698  1897      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181622879  79875:343554151  [1,2,0]p1  [1,2]p1
>  2024-06-12T09:00:15.941065+0000  2024-06-12T09:00:15.941065+0000
>       1806  periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000
> 10.32  5223171   4704732          0        0   8388608   6205669256
> 9800320  1920      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180911263  79875:342754128  [0,2,1]p0  [1,2]p1
>  2024-06-11T19:49:09.010583+0000  2024-06-11T19:49:09.010583+0000
>       1664  periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000
> 10.33  5215341   4525338          0        0   4194304   6066965588
> 9635403  1888      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181467497  79875:402958597  [1,2,0]p1  [1,2]p1
>  2024-06-12T01:59:23.933458+0000  2024-05-31T17:01:38.620377+0000
>        512  periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000
> 10.34  5217342   5217342          0        0   8388608   5999920961
> 9570527  1956      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182013407  79875:419219483  [1,0,2]p1  [1,2]p1
>  2024-06-12T09:10:17.390128+0000  2024-06-01T15:40:23.241769+0000
>        601  periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000
> 10.35  5222114   5222114          0        0   8388608   6021420075
> 9571300  1952      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180948342  79875:338598678  [1,2,0]p1  [1,2]p1
>  2024-06-11T22:59:17.017317+0000  2024-05-30T15:59:15.553266+0000
>        539  periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000
> 10.36  5223657   5223657          0        0   4331271   5844578100
> 9347405  1903      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'183363403  79875:408484683  [0,2,1]p0  [1,2]p1
>  2024-06-11T18:37:46.653097+0000  2024-05-25T14:13:51.648712+0000
>        553  periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000
> 10.37  5219248   5219248          0        0   4194304   6072826085
> 9579049  1936      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182091890  79875:341545087  [1,0,2]p1  [1,2]p1
>  2024-06-11T14:19:40.744743+0000  2024-05-29T18:40:12.602405+0000
>        771  periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000
> 10.38  5214022   5214022          0        0   4194858   5780805541
> 9299038  1951      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'178809936  79875:390113817  [1,0,2]p1  [1,2]p1
>  2024-06-02T00:55:22.046433+0000  2024-05-25T16:49:10.279003+0000
>        931  queued for scrub
> 10.39  5220692   4542094          0        0         0   6401799077
> 9864164  1872      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'182881827  79875:402437400  [2,0,1]p2  [2,1]p2
>  2024-06-12T01:42:35.239155+0000  2024-06-12T01:42:35.239155+0000
>       1370  periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000
> 10.3a  5215648   4552651          0        0         0   6194465524
> 9713558  1898      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181458295  79875:394916652  [2,1,0]p2  [2,1]p2
>  2024-06-11T21:19:29.020889+0000  2024-06-11T21:19:29.020889+0000
>       1227  periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000
> 10.3b  5218220   5218220          0        0   8388608   5951035110
> 9490698  1878      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180815984  79875:342438023  [2,1,0]p2  [2,1]p2
>  2024-06-03T05:03:19.483049+0000  2024-05-28T18:50:55.369009+0000
>       1004  queued for scrub
> 10.3c  5221615   5221615          0        0  12582912   6260894058
> 9727358  1938      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181561627  79875:342153262  [1,2,0]p1  [1,2]p1
>  2024-06-11T19:12:06.206008+0000  2024-06-11T19:12:06.206008+0000
>       1471  periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000
> 10.3d  5213771   5213771          0        0         0   6215975311
> 9699615  1946      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179354493  79875:399858588  [1,2,0]p1  [1,2]p1
>  2024-06-11T04:05:10.877703+0000  2024-06-11T04:05:10.877703+0000
>       8123  periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000
> 10.3e  5219446   4659577          0        0   4194304   5852588566
> 9370152  1933      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180035882  79875:391595513  [0,1,2]p0  [1,2]p1
>  2024-06-11T20:35:43.624903+0000  2024-05-22T21:40:55.606650+0000
>        520  periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000
> 10.3f  5219729   4889443          0        0  16777216   5925457842
> 9484458  1948      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'181330738  79875:409113809  [0,2,1]p0  [1,2]p1
>  2024-06-12T05:12:01.852109+0000  2024-05-27T20:03:22.704122+0000
>        533  periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000
> 10.40  5222241   4445164          0        0         0   6312545191
> 9773697  1873      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181341418  79875:393186178  [2,1,0]p2  [2,1]p2
>  2024-06-10T20:00:37.561045+0000  2024-06-10T20:00:37.561045+0000
>       7712  periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000
> 10.41  5215195   5183508          0        0   4194326   6000770355
> 9529537  1938      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181494334  79875:343712275  [0,2,1]p0  [1,2]p1
>  2024-06-12T04:18:58.693597+0000  2024-06-01T11:05:59.370690+0000
>        539  periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000
> 10.42  5219564   4681323          0        0  12582912   5822745109
> 9331625  1916      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'179811520  79875:340958258  [1,2,0]p1  [1,2]p1
>  2024-06-02T19:58:06.439770+0000  2024-05-21T18:28:28.794738+0000
>      11044  queued for deep scrub
> 10.43  5219412   4599458          0        0       594   5737440613
> 9239869  1950      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180634536  79875:388879991  [1,0,2]p1  [1,2]p1
>  2024-06-01T18:13:17.068416+0000  2024-05-20T16:32:20.302230+0000
>       1002  queued for deep scrub
> 10.44  5216008   5216008          0        0   4194304   6270195298
> 9786806  1944      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182088681  79875:406175480  [1,0,2]p1  [1,2]p1
>  2024-06-12T00:19:01.240405+0000  2024-06-12T00:19:01.240405+0000
>       1593  periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000
> 10.45  5216238   4694313          0        0         0   5714807184
> 9223666  1944      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'179709878  79875:338307727  [1,0,2]p1  [1,2]p1
>  2024-06-02T00:39:50.657116+0000  2024-05-21T02:52:14.376959+0000
>        958  queued for deep scrub
> 10.46  5217950   5217950          0        0   9729074   5945415128
> 9499020  1915      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'183726716  79875:395024265  [1,2,0]p1  [1,2]p1
>  2024-06-11T14:56:28.875949+0000  2024-05-29T04:19:53.409918+0000
>        724  periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000
> 10.47  5219125   5219125          0        0   4194304   5775128028
> 9299099  1864      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181328922  79875:340510799  [1,0,2]p1  [1,2]p1
>  2024-06-11T17:58:16.753327+0000  2024-05-23T14:24:25.876796+0000
>        652  periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000
> 10.48  5216471   5185269          0        0   4194326   6352053972
> 9843580  1929      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179547710  79875:378125137  [0,1,2]p0  [1,2]p1
>  2024-06-11T23:22:17.435614+0000  2024-06-11T23:22:17.435614+0000
>       1379  periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000
> 10.49  5217119   4580870          0        0   4194304   5971558944
> 9526047  1879      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'182489297  79875:389298429  [2,0,1]p2  [2,1]p2
>  2024-06-11T08:57:42.791265+0000  2024-05-29T07:13:57.867295+0000
>        959  periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000
> 10.4a  5217577   4909174          0        0   4194304   5960086599
> 9511657  1910      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'180982438  79875:381432369  [0,1,2]p0  [1,2]p1
>  2024-06-12T02:31:42.576663+0000  2024-06-01T09:08:07.860775+0000
>        480  periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000
> 10.4b  5220270   5220270          0        0    159808   6313986186
> 9763672  1934      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181279214  79875:342509944  [2,1,0]p2  [2,1]p2
>  2024-06-11T11:03:38.997373+0000  2024-06-11T11:03:38.997373+0000
>       3970  periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000
> 10.4c  5217864   5217864          0        0   4194304   6104537460
> 9679170  1948      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181081348  79875:341937760  [1,0,2]p1  [1,2]p1
>  2024-06-12T06:09:49.667680+0000  2024-06-01T13:06:12.938513+0000
>        520  periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000
> 10.4d  5217604   4905278          0        0   8388608   6250536027
> 9765571  1917      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'179562016  79875:387768172  [0,1,2]p0  [1,2]p1
>  2024-06-11T13:38:28.479053+0000  2024-06-11T13:38:28.479053+0000
>       2004  periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000
> 10.4e  5220202   5220202          0        0  12215022   5781813383
> 9291138  1940      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180360470  79875:377712114  [1,0,2]p1  [1,2]p1
>  2024-06-01T22:06:12.347760+0000  2024-05-25T19:34:49.805104+0000
>        982  queued for scrub
> 10.4f  5215243   4561466          0        0         0   6099447266
> 9605810  1930      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181794065  79875:396660094  [2,1,0]p2  [2,1]p2
>  2024-06-02T13:35:21.451128+0000  2024-06-02T13:35:21.451128+0000
>       7316  queued for scrub
> 10.50  5215219   5184003          0        0         0   6151789627
> 9674779  1871      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181695151  79875:405060298  [0,2,1]p0  [1,2]p1
>  2024-06-11T16:42:00.221013+0000  2024-06-11T16:42:00.221013+0000
>       2385  periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000
> 10.51  5218666   5218666          0        0   8388608   5774655330
> 9331373  1920      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180711664  79875:342978462  [0,1,2]p0  [1,2]p1
>  2024-06-11T14:31:51.402844+0000  2024-05-24T11:26:15.539677+0000
>        730  periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000
> 10.52  5218435   5187050          0        0   4194304   6375954134
> 9843553  1935      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180834874  79875:341502004  [0,1,2]p0  [1,2]p1
>  2024-06-11T08:41:43.586833+0000  2024-06-11T08:41:43.586833+0000
>       8368  periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000
> 10.53  5216822   5216822          0        0  10001995   5786896551
> 9368323  1928      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181337168  79875:402047940  [0,2,1]p0  [1,2]p1
>  2024-06-12T04:54:21.147742+0000  2024-05-23T23:02:36.154347+0000
>        557  periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000
> 10.54  5213949   4567626          0        0   8388608   6095478377
> 9664814  1944      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'182179986  79875:421118572  [2,1,0]p2  [2,1]p2
>  2024-06-12T09:19:30.470319+0000  2024-06-01T17:56:32.089348+0000
>        551  periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000
> 10.55  5217389   5217389          0        0   4194948   5905845521
> 9403344  1950      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'183481821  79875:341223864  [1,2,0]p1  [1,2]p1
>  2024-06-11T15:31:05.524698+0000  2024-05-29T11:50:10.322128+0000
>        584  periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000
> 10.56  5219142   5219142          0        0         0   6311549101
> 9773102  1897      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'184062267  79875:408792597  [1,2,0]p1  [1,2]p1
>  2024-06-10T22:17:28.523159+0000  2024-06-10T22:17:28.523159+0000
>       8211  periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000
> 10.57  5219035   4894148          0        0       180   5862209193
> 9396761  1895      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'181886155  79875:340148291  [0,1,2]p0  [1,2]p1
>  2024-06-10T22:33:19.364284+0000  2024-05-28T12:05:26.762682+0000
>        951  periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000
> 10.58  5218318   5185905          0        0   9878587   5911568798
> 9459707  1875      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179109213  79875:390200207  [2,0,1]p2  [2,1]p2
>  2024-06-11T13:08:37.380203+0000  2024-05-30T10:19:35.091370+0000
>        578  periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000
> 10.59  5218315   5185838          0        0   4194304   5727052522
> 9214424  1875      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181700141  79875:400297310  [2,0,1]p2  [2,1]p2
>  2024-06-11T11:15:00.221765+0000  2024-05-21T11:28:39.056382+0000
>        682  periodic deep scrub scheduled @
2024-06-16T13:00:43.750185+0000
> 10.5a  5218301   5218301          0        0   4194304   5833097049
> 9365354  1942      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180699909  79875:393344045  [1,2,0]p1  [1,2]p1
>  2024-06-12T01:50:51.954645+0000  2024-05-25T00:11:49.917556+0000
>        497  periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000
> 10.5b  5215134   5215134          0        0   4194802   6070043467
> 9585134  1892      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181149500  79875:342539489  [2,0,1]p2  [2,1]p2
>  2024-06-12T08:19:32.265112+0000  2024-06-02T11:33:23.094569+0000
>        714  periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000
> 10.5c  5221503   5221503          0        0   4194304   5989312430
> 9524556  1939      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180748682  79875:340750417  [1,2,0]p1  [1,2]p1
>  2024-06-11T09:37:59.483197+0000  2024-05-28T03:41:58.475567+0000
>       1021  periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000
> 10.5d  5222202   5222202          0        0   8388608   6031557641
> 9520790  1928      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'179508607  79875:399919359  [1,0,2]p1  [1,2]p1
>  2024-06-11T17:10:33.165278+0000  2024-05-30T05:53:10.772736+0000
>        575  periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000
> 10.5e  5212770   4506643          0        0         0   5830165136
> 9388228  1938      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180002484  79875:391844220  [2,0,1]p2  [2,1]p2
>  2024-06-11T22:32:49.026280+0000  2024-05-24T15:31:50.362016+0000
>        523  periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000
> 10.5f  5221504   5221504          0        0   4194394   5807975316
> 9387383  1879      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181161493  79875:408530090  [0,2,1]p0  [1,2]p1
>  2024-06-11T15:10:33.261745+0000  2024-05-25T06:37:01.629711+0000
>        845  periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000
> 10.60  5217784   5186615          0        0   4194304   6208553838
> 9749044  1962      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181569072  79875:392211608  [0,1,2]p0  [1,2]p1
>  2024-06-11T21:44:35.372861+0000  2024-06-11T21:44:35.372861+0000
>       1505  periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000
> 10.61  5221292   4586636          0        0   8388608   5766320671
> 9280184  1910      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180945714  79875:344272511  [0,1,2]p0  [1,2]p1
>  2024-06-11T16:51:25.519891+0000  2024-05-24T17:37:21.814566+0000
>        566  periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000
> 10.62  5218498   4699002          0        0   4194466   6027128219
> 9555111  1884      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180387699  79875:341929273  [1,2,0]p1  [1,2]p1
>  2024-06-12T03:10:08.547987+0000  2024-06-01T02:47:43.109035+0000
>        509  periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000
> 10.63  5219101   5219101          0        0         0   6040382110
> 9500040  1885      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181623402  79875:389756489  [0,1,2]p0  [1,2]p1
>  2024-06-02T23:50:14.947543+0000  2024-06-02T23:50:14.947543+0000
>      13925  queued for scrub
> 10.64  5217025   5217025          0        0   2277792   6222337345
> 9701950  1866      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181853168  79875:406174284  [1,2,0]p1  [1,2]p1
>  2024-06-02T04:57:41.872723+0000  2024-06-02T04:57:41.872723+0000
>       6394  queued for scrub
> 10.65  5217908   4590550          0        0   4194326   6223151288
> 9709345  1916      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181196382  79875:340107600  [2,0,1]p2  [2,1]p2
>  2024-06-11T15:53:28.725001+0000  2024-06-11T15:53:28.725001+0000
>       1343  periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000
> 10.66  5218911   4893654          0        0   4194304   5815703055
> 9338987  1939      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'183062531  79875:395193073  [1,0,2]p1  [1,2]p1
>  2024-06-12T01:10:56.919233+0000  2024-05-25T04:50:04.300293+0000
>        590  periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000
> 10.67  5217539   4587208          0        0   4194304   6095526883
> 9640323  1955      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181875315  79875:342878178  [2,1,0]p2  [2,1]p2
>  2024-06-12T07:58:41.522730+0000  2024-06-01T07:20:59.951966+0000
>        553  periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000
> 10.68  5215852   4505902          0        0   8388608   6093858586
> 9596421  1926      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'179490643  79875:379632203  [0,1,2]p0  [1,2]p1
>  2024-06-11T16:02:14.995195+0000  2024-05-30T22:02:42.335910+0000
>        526  periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000
> 10.69  5216596   5216596          0        0   4194304   5833268044
> 9400414  1899      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181876551  79875:387059449  [1,2,0]p1  [1,2]p1
>  2024-06-11T17:00:57.111851+0000  2024-05-23T07:05:05.989427+0000
>        572  periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000
> 10.6a  5214106   4497370          0        0         0   5745504618
> 9296938  1869      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180109669  79875:381871149  [2,1,0]p2  [2,1]p2
>  2024-06-12T00:27:45.155477+0000  2024-05-25T09:22:19.570011+0000
>        494  periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000
> 10.6b  5219339   5219339          0        0         0   5801390847
> 9370084  1957      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180041524  79875:342270608  [0,1,2]p0  [1,2]p1
>  2024-06-12T03:01:39.358464+0000  2024-05-23T01:04:22.522035+0000
>        488  periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000
> 10.6c  5217917   5217917          0        0       554   6100979305
> 9612250  1955      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180968918  79875:341488318  [1,2,0]p1  [1,2]p1
>  2024-06-11T18:47:35.969345+0000  2024-05-31T11:54:59.559316+0000
>        588  periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000
> 10.6d  5221048   4550388          0        0   4194304   6024067862
> 9621947  1927      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'179373269  79875:388095690  [2,1,0]p2  [2,1]p2
>  2024-06-12T03:26:37.931310+0000  2024-05-31T04:10:41.991457+0000
>        502  periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000
> 10.6e  5215550   4888179          0        0   8388608   6110450001
> 9666892  1906      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'180885974  79875:380260263  [0,1,2]p0  [1,2]p1
>  2024-06-11T20:18:23.657558+0000  2024-06-11T20:18:23.657558+0000
>       1218  periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000
> 10.6f  5215850   4566820          0        0         0   6107146788
> 9634911  1923      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181752773  79875:396708021  [2,1,0]p2  [2,1]p2
>  2024-06-10T17:32:51.735004+0000  2024-06-10T17:32:51.735004+0000
>      10119  periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000
> 10.70  5218211   4672319          0        0   4194304   5793441438
> 9371563  1881      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180405012  79875:404578822  [1,2,0]p1  [1,2]p1
>  2024-06-03T04:46:35.130293+0000  2024-05-22T07:50:20.898441+0000
>        969  queued for deep scrub
> 10.71  5218299   4599312          0        0   4194304   5807950891
> 9349720  1906      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180690682  79875:344108361  [0,2,1]p0  [1,2]p1
>  2024-06-12T03:18:16.645927+0000  2024-05-23T09:32:56.055703+0000
>        487  periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000
> 10.72  5223470   4922347          0        0   8388608   6416468325
> 9922777  1958      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'180797342  79875:342356916  [0,1,2]p0  [1,2]p1
>  2024-06-12T05:42:47.545762+0000  2024-06-12T05:42:47.545762+0000
>       1307  periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000
> 10.73  5218301   4520651          0        0         0   6293832301
> 9738943  1953      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'182246986  79875:403792984  [2,0,1]p2  [2,1]p2
>  2024-06-11T20:59:02.079157+0000  2024-06-11T20:59:02.079157+0000
>       1399  periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000
> 10.74  5219061   4595112          0        0         0   5974775781
> 9469653  1943      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181947609  79875:420571221  [2,1,0]p2  [2,1]p2
>  2024-06-03T02:17:19.687482+0000  2024-05-28T08:43:38.825954+0000
>        927  queued for scrub
> 10.75  5220994   4596634          0        0  25165824   5822315216
> 9362548  1877      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'180376305  79875:338847231  [2,0,1]p2  [2,1]p2
>  2024-06-03T05:52:28.743255+0000  2024-05-24T05:11:00.211501+0000
>        981  queued for scrub
> 10.76  5222111   4892295          0        0         0   6067998823
> 9574254  1935      3000
active+undersized+degraded+remapped+backfilling
>     2h  79875'183871180  79875:409218100  [1,0,2]p1  [1,2]p1
>  2024-06-03T04:14:04.513004+0000  2024-06-03T04:14:04.513004+0000
>       6987  queued for scrub
> 10.77  5219020   4574683          0        0         0   5889074277
> 9460824  1896      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181614825  79875:342558162  [2,0,1]p2  [2,1]p2
>  2024-06-12T08:07:38.169523+0000  2024-05-27T15:15:35.587972+0000
>        536  periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000
> 10.78  5220442   4693235          0        0   8388608   5879566566
> 9395501  1923      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'179031456  79875:391462415  [1,0,2]p1  [1,2]p1
>  2024-06-02T09:32:42.834344+0000  2024-05-27T10:41:23.822124+0000
>        994  queued for scrub
> 10.79  5219734   4702117          0        0   4194304   6129933545
> 9646669  1926      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'182574416  79875:402249812  [1,2,0]p1  [1,2]p1
>  2024-06-12T05:52:02.959814+0000  2024-06-01T00:53:49.548055+0000
>        555  periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000
> 10.7a  5215303   4526846          0        0         0   6316171107
> 9817029  1932      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181765231  79875:395877057  [2,0,1]p2  [2,1]p2
>  2024-06-12T02:23:42.692096+0000  2024-06-12T02:23:42.692096+0000
>       1457  periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000
> 10.7b  5217213   4606816          0        0   4194304   6226859283
> 9718621  1952      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'181413541  79875:343968658  [1,2,0]p1  [1,2]p1
>  2024-06-11T23:52:28.414752+0000  2024-06-11T23:52:28.414752+0000
>       1308  periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000
> 10.7c  5215075   5215075          0        0   4194304   6013877208
> 9602345  1867      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'180991860  79875:341775615  [0,2,1]p0  [1,2]p1
>  2024-06-11T23:30:40.614412+0000  2024-05-31T14:10:30.542603+0000
>        503  periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000
> 10.7d  5220700   4569720          0        0   5837609   6258736025
> 9764836  1897      3000
active+undersized+degraded+remapped+backfilling
>     3h  79875'179741885  79875:401448508  [2,1,0]p2  [2,1]p2
>  2024-06-12T04:45:05.030905+0000  2024-06-12T04:45:05.030905+0000
>       1565  periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000
> 10.7e  5217113   5217113          0        0  16777216   6072081676
> 9598390  1902      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'181040361  79875:391032348  [1,2,0]p1  [1,2]p1
>  2024-06-11T17:37:26.589968+0000  2024-06-11T17:37:26.589968+0000
>       1613  periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000
> 10.7f  5218719   5218719          0        0   9909061   6130469539
> 9657824  1943      3000
active+undersized+degraded+remapped+backfill_wait
>     3h  79875'182283514  79875:408550888  [1,2,0]p1  [1,2]p1
>  2024-06-11T13:05:03.552526+0000  2024-06-11T13:05:03.552526+0000
>       1796  periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000
>
> * NOTE: Omap statistics are gathered during deep scrub and may be
> inaccurate soon afterwards depending on utilization. See
> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for
> further details.
>
>
>
> sudo ceph osd df
> ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP      META
>   AVAIL    %USE   VAR   PGS  STATUS
>  0    ssd   3.63869   1.00000  3.6 TiB   82 GiB   50 MiB    40 GiB   42
GiB
>  3.6 TiB   2.21  0.03    0      up
>  1    ssd   3.63869   1.00000  3.6 TiB  2.0 TiB   16 GiB  1021 GiB  1.0
TiB
>  1.6 TiB  55.41  0.75  161      up
>  2    ssd   3.63869   1.00000  3.6 TiB  3.0 TiB   16 GiB   1.1 TiB  1.9
TiB
>  676 GiB  81.86  1.11  161      up
>
>
> sudo ceph df detail
> --- RAW STORAGE ---
> CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
> hdd    1.2 PiB  308 TiB  871 TiB   871 TiB      73.86
> ssd     11 TiB  5.8 TiB  5.1 TiB   5.1 TiB      46.51
> TOTAL  1.2 PiB  314 TiB  876 TiB   876 TiB      73.61
>
> --- POOLS ---
> POOL      ID   PGS   STORED   (DATA)   (OMAP)  OBJECTS     USED   (DATA)
> (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR
>  UNDER COMPR
> .mgr       1     1  827 MiB  827 MiB      0 B      208  1.6 GiB  1.6 GiB
>    0 B   0.11    734 GiB            N/A       50 GiB    N/A         0 B
>      0 B
> metadata  10   128  1.0 TiB  647 MiB  1.0 TiB  667.90M  2.1 TiB  1.3 GiB
>  2.1 TiB  59.45    719 GiB            N/A      3.5 TiB    N/A         0 B
>        0 B
> data      11  2048  288 TiB  288 TiB      0 B  566.97M  863 TiB  863 TiB
>    0 B  81.10     67 TiB            N/A          N/A    N/A     3.4 TiB
>  6.8 TiB
> ssd-data  18    32   14 GiB   14 GiB      0 B  265.17k   29 GiB   29 GiB
>    0 B   1.90    734 GiB            N/A          N/A    N/A         0 B
>      0 B
>
>
> sudo ceph balancer status
> {
>     "active": true,
>     "last_optimize_duration": "0:00:00.000170",
>     "last_optimize_started": "Wed Jun 12 13:14:49 2024",
>     "mode": "upmap",
>     "no_optimization_needed": true,
>     "optimize_result": "Some objects (0.172816) are degraded; try again
> later",
>     "plans": []
> }
>
>
>
> [image: ariadne.ai Logo] Lars Köppel
> Developer
> Email: lars.koeppel@xxxxxxxxxx
> Phone: +49 6221 5993580 <+4962215993580>
> ariadne.ai (Germany) GmbH
> Häusserstraße 3, 69115 Heidelberg
> Amtsgericht Mannheim, HRB 744040
> Geschäftsführer: Dr. Fabian Svara
> https://ariadne.ai
>
>
> On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri <anthony.datri@xxxxxxxxx>
> wrote:
>
>> If you have:
>>
>> * pg_num too low (defaults are too low)
>> * pg_num not a power of 2
>> * pg_num != number of OSDs in the pool
>> * balancer not enabled
>>
>> any of those might result in imbalance.
>>
>> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote:
>> >
>> > I don't have any good explanation at this point. Can you share some
more
>> information like:
>> >
>> > ceph pg ls-by-pool <cephfs_metadata>
>> > ceph osd df (for the relevant OSDs)
>> > ceph df
>> >
>> > Thanks,
>> > Eugen
>> >
>> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>> >
>> >> Since my last update the size of the largest OSD increased by 0.4 TiB
>> while
>> >> the smallest one only increased by 0.1 TiB. How is this possible?
>> >>
>> >> Because the metadata pool reported to have only 900MB space left, I
>> stopped
>> >> the hot-standby MDS. This gave me 8GB back but these filled up in the
>> last
>> >> 2h.
>> >> I think I have to zap the next OSD because the filesystem is getting
>> read
>> >> only...
>> >>
>> >> How is it possible that an OSD has over 1 TiB less data on it after a
>> >> rebuild? And how is it possible to have so different sizes of OSDs?
>> >>
>> >>
>> >> [image: ariadne.ai Logo] Lars Köppel
>> >> Developer
>> >> Email: lars.koeppel@xxxxxxxxxx
>> >> Phone: +49 6221 5993580 <+4962215993580>
>> >> ariadne.ai (Germany) GmbH
>> >> Häusserstraße 3, 69115 Heidelberg
>> >> Amtsgericht Mannheim, HRB 744040
>> >> Geschäftsführer: Dr. Fabian Svara
>> >> https://ariadne.ai
>> >>
>> >>
>> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel <lars.koeppel@xxxxxxxxxx
>
>> wrote:
>> >>
>> >>> Only in warning mode. And there were no PG splits or merges in the
>> last 2
>> >>> month.
>> >>>
>> >>>
>> >>> [image: ariadne.ai Logo] Lars Köppel
>> >>> Developer
>> >>> Email: lars.koeppel@xxxxxxxxxx
>> >>> Phone: +49 6221 5993580 <+4962215993580>
>> >>> ariadne.ai (Germany) GmbH
>> >>> Häusserstraße 3, 69115 Heidelberg
>> >>> Amtsgericht Mannheim, HRB 744040
>> >>> Geschäftsführer: Dr. Fabian Svara
>> >>> https://ariadne.ai
>> >>>
>> >>>
>> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> wrote:
>> >>>
>> >>>> I don't think scrubs can cause this. Do you have autoscaler
enabled?
>> >>>>
>> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>> >>>>
>> >>>> > Hi,
>> >>>> >
>> >>>> > thank you for your response.
>> >>>> >
>> >>>> > I don't think this thread covers my problem, because the OSDs for
>> the
>> >>>> > metadata pool fill up at different rates. So I would think this
is
>> no
>> >>>> > direct problem with the journal.
>> >>>> > Because we had earlier problems with the journal I changed some
>> >>>> > settings(see below). I already restarted all MDS multiple times
but
>> no
>> >>>> > change here.
>> >>>> >
>> >>>> > The health warnings regarding cache pressure resolve normally
after
>> a
>> >>>> > short period of time, when the heavy load on the client ends.
>> Sometimes
>> >>>> it
>> >>>> > stays a bit longer because an rsync is running and copying data
on
>> the
>> >>>> > cluster(rsync is not good at releasing the caps).
>> >>>> >
>> >>>> > Could it be a problem if scrubs run most of the time in the
>> background?
>> >>>> Can
>> >>>> > this block any other tasks or generate new data itself?
>> >>>> >
>> >>>> > Best regards,
>> >>>> > Lars
>> >>>> >
>> >>>> >
>> >>>> > global                      basic     mds_cache_memory_limit
>> >>>> >     17179869184
>> >>>> > global                      advanced  mds_max_caps_per_client
>> >>>> >    16384
>> >>>> > global                      advanced
>> >>>> mds_recall_global_max_decay_threshold
>> >>>> >    262144
>> >>>> > global                      advanced  mds_recall_max_decay_rate
>> >>>> >    1.000000
>> >>>> > global                      advanced
mds_recall_max_decay_threshold
>> >>>> >     262144
>> >>>> > mds                         advanced  mds_cache_trim_threshold
>> >>>> >     131072
>> >>>> > mds                         advanced  mds_heartbeat_grace
>> >>>> >    120.000000
>> >>>> > mds                         advanced  mds_heartbeat_reset_grace
>> >>>> >    7400
>> >>>> > mds                         advanced  mds_tick_interval
>> >>>> >    3.000000
>> >>>> >
>> >>>> >
>> >>>> > [image: ariadne.ai Logo] Lars Köppel
>> >>>> > Developer
>> >>>> > Email: lars.koeppel@xxxxxxxxxx
>> >>>> > Phone: +49 6221 5993580 <+4962215993580>
>> >>>> > ariadne.ai (Germany) GmbH
>> >>>> > Häusserstraße 3, 69115 Heidelberg
>> >>>> > Amtsgericht Mannheim, HRB 744040
>> >>>> > Geschäftsführer: Dr. Fabian Svara
>> >>>> > https://ariadne.ai
>> >>>> >
>> >>>> >
>> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx>
wrote:
>> >>>> >
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> can you check if this thread [1] applies to your situation? You
>> don't
>> >>>> >> have multi-active MDS enabled, but maybe it's still some journal
>> >>>> >> trimming, or maybe misbehaving clients? In your first post there
>> were
>> >>>> >> health warnings regarding cache pressure and cache size. Are
those
>> >>>> >> resolved?
>> >>>> >>
>> >>>> >> [1]
>> >>>> >>
>> >>>> >>
>> >>>>
>>
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE
>> >>>> >>
>> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>> >>>> >>
>> >>>> >> > Hello everyone,
>> >>>> >> >
>> >>>> >> > short update to this problem.
>> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected
>> size
>> >>>> >> ~50%).
>> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They
>> jumped up
>> >>>> and
>> >>>> >> > down a lot but the higher one has now also reached 'nearfull'
>> >>>> status. How
>> >>>> >> > is this possible? What is going on?
>> >>>> >> >
>> >>>> >> > Does anyone have a solution how to fix this without zapping
the
>> OSD?
>> >>>> >> >
>> >>>> >> > Best regards,
>> >>>> >> > Lars
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > [image: ariadne.ai Logo] Lars Köppel
>> >>>> >> > Developer
>> >>>> >> > Email: lars.koeppel@xxxxxxxxxx
>> >>>> >> > Phone: +49 6221 5993580 <+4962215993580>
>> >>>> >> > ariadne.ai (Germany) GmbH
>> >>>> >> > Häusserstraße 3, 69115 Heidelberg
>> >>>> >> > Amtsgericht Mannheim, HRB 744040
>> >>>> >> > Geschäftsführer: Dr. Fabian Svara
>> >>>> >> > https://ariadne.ai
>> >>>> >> > _______________________________________________
>> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>> >>
>> >>>> >>
>> >>>> >> _______________________________________________
>> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>> >>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>>






_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux