Re: CephFS metadata pool size

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I am happy to help you with as much information as possible. I probably
just don't know where to look for it.
Below are the requested information. The cluster is rebuilding the
zapped OSD at the moment. This will probably take the next few days.


sudo ceph pg ls-by-pool metadata
PG     OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES     OMAP_BYTES*
 OMAP_KEYS*  LOG   LOG_DUPS  STATE
     SINCE  VERSION          REPORTED         UP         ACTING
SCRUB_STAMP                      DEEP_SCRUB_STAMP
LAST_SCRUB_DURATION  SCRUB_SCHEDULING
10.0   5217325   4994695          0        0   4194304   5880891340
9393865  1885      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'180849582  79875:391519635  [0,1,2]p0  [1,2]p1
 2024-06-11T09:08:09.829362+0000  2024-05-28T05:52:59.321589+0000
       627  periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000
10.1   5214785   5193424          0        0         0   5843682713
9410150  1912      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180914288  79875:342746928  [2,1,0]p2  [2,1]p2
 2024-06-01T15:56:28.927288+0000  2024-05-27T03:31:37.682966+0000
       966  queued for scrub
10.2   5218432   5187168          0        0         0   6402011266
9812513  1874      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180970531  79875:341340204  [0,1,2]p0  [1,2]p1
 2024-06-11T13:40:58.994256+0000  2024-06-11T13:40:58.994256+0000
      1942  periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000
10.3   5217413   5217413          0        0   8388788   5766005023
9271787  1923      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181012233  79875:388295881  [1,0,2]p1  [1,2]p1
 2024-06-12T00:35:56.965547+0000  2024-05-23T19:54:56.121729+0000
       492  periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000
10.4   5220069   5220069          0        0  12583466   6027548724
9537290  1959      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181576075  79875:405295868  [1,2,0]p1  [1,2]p1
 2024-06-11T17:47:22.923514+0000  2024-05-31T02:06:55.339574+0000
       581  periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000
10.5   5216162   5211999          0        0   4194304   5941347251
9542764  1930      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180455793  79875:338418517  [2,1,0]p2  [2,1]p2
 2024-06-11T22:50:16.170708+0000  2024-05-30T23:49:54.316379+0000
       528  periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000
10.6   5216100   4980459          0        0   4521984   6428088514
9850762  1911      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'184045876  79875:396809795  [0,2,1]p0  [1,2]p1
 2024-06-11T22:24:05.102716+0000  2024-06-11T22:24:05.102716+0000
      1082  periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000
10.7   5218232   5218232          0        0   4194304   6377065363
9849360  1919      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182672562  79875:342449062  [1,0,2]p1  [1,2]p1
 2024-06-11T06:22:15.689422+0000  2024-06-11T06:22:15.689422+0000
      8225  periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000
10.8   5219620   5182816          0        0         0   6167304290
9691796  1896      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179628377  79875:378022884  [2,1,0]p2  [2,1]p2
 2024-06-11T22:06:01.386763+0000  2024-06-11T22:06:01.386763+0000
      1286  periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000
10.9   5219448   5164591          0        0   8388698   5796048346
9338312  1868      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181739392  79875:387412389  [2,1,0]p2  [2,1]p2
 2024-06-12T05:21:00.586747+0000  2024-05-26T11:10:59.780673+0000
       539  periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000
10.a   5219861   5163635          0        0  12582912   5841839055
9387200  1916      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180205688  79875:379381294  [1,2,0]p1  [1,2]p1
 2024-06-11T12:35:05.571200+0000  2024-05-22T11:07:16.041773+0000
      1093  periodic deep scrub scheduled @ 2024-06-17T05:21:40.136463+0000
10.b   5217949   5217949          0        0  16777216   5935863260
9462127  1881      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181655745  79875:343806807  [0,1,2]p0  [1,2]p1
 2024-06-11T22:41:28.976920+0000  2024-05-26T08:43:29.217457+0000
       520  periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000
10.c   5221697   5217118          0        0   4194304   6015217841
9574445  1928      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180892826  79875:341490398  [2,1,0]p2  [2,1]p2
 2024-06-11T09:20:58.443473+0000  2024-05-30T00:13:50.306507+0000
       768  periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000
10.d   5217727   4908764          0        0         0   5825598519
9349877  1930      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'179050347  79875:387455993  [0,1,2]p0  [1,2]p1
 2024-06-02T07:27:27.873631+0000  2024-05-20T03:58:30.225170+0000
       952  queued for deep scrub
10.e   5215040   5210572          0        0   4194304   5790634469
9327651  1905      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180049044  79875:377196119  [2,1,0]p2  [2,1]p2
 2024-06-12T05:03:08.521423+0000  2024-05-22T13:47:46.753131+0000
       527  periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000
10.f   5217202   4885952          0        0   8388630   6005274626
9589178  1917      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'181953069  79875:396402453  [0,2,1]p0  [1,2]p1
 2024-06-11T12:20:49.661741+0000  2024-06-11T12:20:49.661741+0000
      3949  periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000
10.10  5214985   5157391          0        0   4194394   6275024632
9721505  1922      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181801708  79875:395055895  [2,0,1]p2  [2,1]p2
 2024-06-11T01:49:47.831191+0000  2024-06-11T01:49:47.831191+0000
      9579  periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000
10.11  5219236   4983134          0        0   4194304   5787580073
9316893  1901      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'181011816  79875:343488767  [1,0,2]p1  [1,2]p1
 2024-06-03T00:06:14.862209+0000  2024-05-27T00:05:56.468361+0000
       960  queued for scrub
10.12  5214103   5182770          0        0        90   5970750188
9505613  1891      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180210613  79875:341042616  [0,1,2]p0  [1,2]p1
 2024-06-12T06:39:48.252419+0000  2024-05-28T01:32:03.381577+0000
       527  periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000
10.13  5214580   5214580          0        0   8388642   5793432968
9335940  1957      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182394614  79875:393423214  [1,2,0]p1  [1,2]p1
 2024-06-10T17:52:04.690203+0000  2024-05-22T03:39:02.453336+0000
      1151  periodic deep scrub scheduled @ 2024-06-16T00:28:37.682765+0000
10.14  5218591   5218591          0        0   5053325   6046268958
9582481  1915      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'184576050  79875:425213800  [2,0,1]p2  [2,1]p2
 2024-06-03T05:36:07.656965+0000  2024-05-28T22:46:16.814318+0000
       985  queued for scrub
10.15  5215919   4907184          0        0  11939335   5752284246
9285870  1889      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'179713036  79875:337938973  [0,2,1]p0  [1,2]p1
 2024-06-11T14:06:50.138992+0000  2024-05-24T07:52:22.522455+0000
       786  periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000
10.16  5221091   5221091          0        0   8388608   5966308565
9487816  1916      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'183445984  79875:399413009  [0,2,1]p0  [1,2]p1
 2024-06-03T02:01:53.649417+0000  2024-06-03T02:01:53.649417+0000
      6935  queued for scrub
10.17  5218069   5194732          0        0   4194304   6231638001
9735915  1890      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182390128  79875:340719915  [2,0,1]p2  [2,1]p2
 2024-06-12T06:31:01.523916+0000  2024-06-12T06:31:01.523916+0000
      1272  periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000
10.18  5214253   5159897          0        0         0   6128506469
9673261  1886      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179754208  79875:389150273  [2,1,0]p2  [2,1]p2
 2024-06-12T02:53:31.027154+0000  2024-06-12T02:53:31.027154+0000
      1308  periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000
10.19  5220131   5183342          0        0         0   5869775360
9460583  1953      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181750923  79875:393162989  [2,0,1]p2  [2,1]p2
 2024-06-11T19:21:24.106504+0000  2024-05-22T15:42:01.721091+0000
       522  periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000
10.1a  5217968   5186700          0        0   4194304   6183652625
9733847  1896      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181056240  79875:394241128  [0,2,1]p0  [1,2]p1
 2024-06-02T15:37:49.730965+0000  2024-06-02T15:37:49.730965+0000
      7347  queued for scrub
10.1b  5221259   5161879          0        0         0   5989030303
9561035  1875      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181052305  79875:343144823  [0,1,2]p0  [1,2]p1
 2024-06-12T06:01:07.213894+0000  2024-06-02T00:23:51.063901+0000
       543  periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000
10.1c  5217034   5217034          0        0       554   5979861913
9504442  1875      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180913313  79875:341328275  [1,2,0]p1  [1,2]p1
 2024-06-01T20:02:47.025129+0000  2024-06-01T20:02:47.025129+0000
      6568  queued for scrub
10.1d  5219691   4997027          0        0  12582912   6081321658
9599164  1918      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'179638837  79875:391808225  [0,2,1]p0  [1,2]p1
 2024-06-11T20:27:02.506715+0000  2024-05-31T22:58:06.578167+0000
       517  periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000
10.1e  5213369   5208776          0        0   4194304   6295629499
9765317  1941      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181145210  79875:391326487  [2,1,0]p2  [2,1]p2
 2024-06-12T07:49:28.432084+0000  2024-06-12T07:49:28.432084+0000
      1990  periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000
10.1f  5218399   5000094          0        0   8388608   5821063844
9421222  1896      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'180960296  79875:400635456  [0,2,1]p0  [1,2]p1
 2024-06-11T13:53:40.278050+0000  2024-05-22T19:39:02.345663+0000
       761  periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000
10.20  5216437   5185379          0        0         0   6280994122
9719574  1945      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181610117  79875:391874431  [0,1,2]p0  [1,2]p1
 2024-06-11T12:58:57.754648+0000  2024-06-11T12:58:57.754648+0000
      2288  periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000
10.21  5217349   5217349          0        0   8388698   6348931429
9822197  1867      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181724357  79875:343883762  [0,1,2]p0  [1,2]p1
 2024-06-11T18:28:34.044546+0000  2024-06-11T18:28:34.044546+0000
      1817  periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000
10.22  5215544   4943475          0        0   4194304   5982815232
9482159  1899      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'180230331  79875:340839853  [2,0,1]p2  [2,1]p2
 2024-06-11T15:21:21.994674+0000  2024-05-30T08:01:03.301650+0000
       649  periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000
10.23  5217758   5217758          0        0       146   5804086623
9336298  1920      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180948432  79875:388158699  [1,2,0]p1  [1,2]p1
 2024-06-03T05:19:32.311846+0000  2024-05-27T08:13:15.250058+0000
       971  queued for scrub
10.24  5221505   5221505          0        0         0   6360995644
9816333  1925      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182198758  79875:406847098  [0,2,1]p0  [1,2]p1
 2024-06-12T04:00:44.315682+0000  2024-06-12T04:00:44.315682+0000
      2046  periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000
10.25  5214911   5214911          0        0  12583002   5726320657
9264352  1910      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180132024  79875:338167995  [1,2,0]p1  [1,2]p1
 2024-06-11T14:43:47.076906+0000  2024-05-23T12:16:34.271856+0000
       715  periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000
10.26  5217588   4973667          0        0         0   6040775867
9598406  1927      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'188144536  79875:400277134  [2,0,1]p2  [2,1]p2
 2024-06-02T06:55:09.016528+0000  2024-06-02T06:55:09.016528+0000
      7047  queued for scrub
10.27  5218490   5218490          0        0   8388630   6265942186
9818746  1941      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182400328  79875:342052414  [1,2,0]p1  [1,2]p1
 2024-06-12T01:01:06.674323+0000  2024-06-12T01:01:06.674323+0000
      1500  periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000
10.28  5221153   5221153          0        0         0   6049095371
9597182  1913      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179438777  79875:378265375  [1,2,0]p1  [1,2]p1
 2024-06-12T04:09:58.896192+0000  2024-06-01T04:32:46.011974+0000
       553  periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000
10.29  5214066   5214066          0        0   4194304   5767555623
9316899  1937      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181568997  79875:387683159  [0,2,1]p0  [1,2]p1
 2024-06-10T22:48:57.757493+0000  2024-05-21T07:14:53.348341+0000
       938  periodic deep scrub scheduled @ 2024-06-16T06:24:55.108695+0000
10.2a  5216314   4977811          0        0   4194304   6012567810
9563804  1951      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'181140734  79875:381017643  [2,1,0]p2  [2,1]p2
 2024-06-11T09:57:29.097708+0000  2024-05-30T03:47:22.798044+0000
      1169  periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000
10.2b  5219620   5219620          0        0   8388608   5829917912
9412426  1889      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180465782  79875:342263856  [1,0,2]p1  [1,2]p1
 2024-06-12T08:30:08.941782+0000  2024-05-24T22:00:29.535205+0000
       636  periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000
10.2c  5214087   5214087          0        0         0   5874182922
9375708  1871      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180616041  79875:341244736  [1,2,0]p1  [1,2]p1
 2024-06-10T23:10:08.184410+0000  2024-05-27T17:41:07.583255+0000
      1268  periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000
10.2d  5213968   4984822          0        0   4337824   5835754026
9417033  1947      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'178824607  79875:386427308  [2,0,1]p2  [2,1]p2
 2024-06-12T09:27:43.774331+0000  2024-05-27T05:33:45.293266+0000
       494  periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000
10.2e  5216263   4974456          0        0  12582912   6025714323
9553880  1940      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'180639631  79875:378284872  [1,0,2]p1  [1,2]p1
 2024-06-12T01:19:42.477063+0000  2024-05-31T08:42:41.293335+0000
       526  periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000
10.2f  5220057   5220057          0        0  13839831   5804027574
9370862  1882      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180961915  79875:394513378  [1,0,2]p1  [1,2]p1
 2024-06-11T19:58:05.145147+0000  2024-05-24T20:18:32.301831+0000
       535  periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000
10.30  5217090   4955469          0        0  12582934   6438521431
9924825  1869      3000  active+undersized+degraded+remapped+backfill_wait
    2h  79875'181488488  79875:404940886  [2,1,0]p2  [2,1]p2
 2024-06-12T07:16:18.095439+0000  2024-06-12T07:16:18.095439+0000
      2190  periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000
10.31  5218176   5218176          0        0   4194304   6215016361
9752698  1897      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181622879  79875:343554151  [1,2,0]p1  [1,2]p1
 2024-06-12T09:00:15.941065+0000  2024-06-12T09:00:15.941065+0000
      1806  periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000
10.32  5223171   4704732          0        0   8388608   6205669256
9800320  1920      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180911263  79875:342754128  [0,2,1]p0  [1,2]p1
 2024-06-11T19:49:09.010583+0000  2024-06-11T19:49:09.010583+0000
      1664  periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000
10.33  5215341   4525338          0        0   4194304   6066965588
9635403  1888      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181467497  79875:402958597  [1,2,0]p1  [1,2]p1
 2024-06-12T01:59:23.933458+0000  2024-05-31T17:01:38.620377+0000
       512  periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000
10.34  5217342   5217342          0        0   8388608   5999920961
9570527  1956      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182013407  79875:419219483  [1,0,2]p1  [1,2]p1
 2024-06-12T09:10:17.390128+0000  2024-06-01T15:40:23.241769+0000
       601  periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000
10.35  5222114   5222114          0        0   8388608   6021420075
9571300  1952      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180948342  79875:338598678  [1,2,0]p1  [1,2]p1
 2024-06-11T22:59:17.017317+0000  2024-05-30T15:59:15.553266+0000
       539  periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000
10.36  5223657   5223657          0        0   4331271   5844578100
9347405  1903      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'183363403  79875:408484683  [0,2,1]p0  [1,2]p1
 2024-06-11T18:37:46.653097+0000  2024-05-25T14:13:51.648712+0000
       553  periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000
10.37  5219248   5219248          0        0   4194304   6072826085
9579049  1936      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182091890  79875:341545087  [1,0,2]p1  [1,2]p1
 2024-06-11T14:19:40.744743+0000  2024-05-29T18:40:12.602405+0000
       771  periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000
10.38  5214022   5214022          0        0   4194858   5780805541
9299038  1951      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'178809936  79875:390113817  [1,0,2]p1  [1,2]p1
 2024-06-02T00:55:22.046433+0000  2024-05-25T16:49:10.279003+0000
       931  queued for scrub
10.39  5220692   4542094          0        0         0   6401799077
9864164  1872      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'182881827  79875:402437400  [2,0,1]p2  [2,1]p2
 2024-06-12T01:42:35.239155+0000  2024-06-12T01:42:35.239155+0000
      1370  periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000
10.3a  5215648   4552651          0        0         0   6194465524
9713558  1898      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181458295  79875:394916652  [2,1,0]p2  [2,1]p2
 2024-06-11T21:19:29.020889+0000  2024-06-11T21:19:29.020889+0000
      1227  periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000
10.3b  5218220   5218220          0        0   8388608   5951035110
9490698  1878      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180815984  79875:342438023  [2,1,0]p2  [2,1]p2
 2024-06-03T05:03:19.483049+0000  2024-05-28T18:50:55.369009+0000
      1004  queued for scrub
10.3c  5221615   5221615          0        0  12582912   6260894058
9727358  1938      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181561627  79875:342153262  [1,2,0]p1  [1,2]p1
 2024-06-11T19:12:06.206008+0000  2024-06-11T19:12:06.206008+0000
      1471  periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000
10.3d  5213771   5213771          0        0         0   6215975311
9699615  1946      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179354493  79875:399858588  [1,2,0]p1  [1,2]p1
 2024-06-11T04:05:10.877703+0000  2024-06-11T04:05:10.877703+0000
      8123  periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000
10.3e  5219446   4659577          0        0   4194304   5852588566
9370152  1933      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180035882  79875:391595513  [0,1,2]p0  [1,2]p1
 2024-06-11T20:35:43.624903+0000  2024-05-22T21:40:55.606650+0000
       520  periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000
10.3f  5219729   4889443          0        0  16777216   5925457842
9484458  1948      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'181330738  79875:409113809  [0,2,1]p0  [1,2]p1
 2024-06-12T05:12:01.852109+0000  2024-05-27T20:03:22.704122+0000
       533  periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000
10.40  5222241   4445164          0        0         0   6312545191
9773697  1873      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181341418  79875:393186178  [2,1,0]p2  [2,1]p2
 2024-06-10T20:00:37.561045+0000  2024-06-10T20:00:37.561045+0000
      7712  periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000
10.41  5215195   5183508          0        0   4194326   6000770355
9529537  1938      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181494334  79875:343712275  [0,2,1]p0  [1,2]p1
 2024-06-12T04:18:58.693597+0000  2024-06-01T11:05:59.370690+0000
       539  periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000
10.42  5219564   4681323          0        0  12582912   5822745109
9331625  1916      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'179811520  79875:340958258  [1,2,0]p1  [1,2]p1
 2024-06-02T19:58:06.439770+0000  2024-05-21T18:28:28.794738+0000
     11044  queued for deep scrub
10.43  5219412   4599458          0        0       594   5737440613
9239869  1950      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180634536  79875:388879991  [1,0,2]p1  [1,2]p1
 2024-06-01T18:13:17.068416+0000  2024-05-20T16:32:20.302230+0000
      1002  queued for deep scrub
10.44  5216008   5216008          0        0   4194304   6270195298
9786806  1944      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182088681  79875:406175480  [1,0,2]p1  [1,2]p1
 2024-06-12T00:19:01.240405+0000  2024-06-12T00:19:01.240405+0000
      1593  periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000
10.45  5216238   4694313          0        0         0   5714807184
9223666  1944      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'179709878  79875:338307727  [1,0,2]p1  [1,2]p1
 2024-06-02T00:39:50.657116+0000  2024-05-21T02:52:14.376959+0000
       958  queued for deep scrub
10.46  5217950   5217950          0        0   9729074   5945415128
9499020  1915      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'183726716  79875:395024265  [1,2,0]p1  [1,2]p1
 2024-06-11T14:56:28.875949+0000  2024-05-29T04:19:53.409918+0000
       724  periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000
10.47  5219125   5219125          0        0   4194304   5775128028
9299099  1864      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181328922  79875:340510799  [1,0,2]p1  [1,2]p1
 2024-06-11T17:58:16.753327+0000  2024-05-23T14:24:25.876796+0000
       652  periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000
10.48  5216471   5185269          0        0   4194326   6352053972
9843580  1929      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179547710  79875:378125137  [0,1,2]p0  [1,2]p1
 2024-06-11T23:22:17.435614+0000  2024-06-11T23:22:17.435614+0000
      1379  periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000
10.49  5217119   4580870          0        0   4194304   5971558944
9526047  1879      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'182489297  79875:389298429  [2,0,1]p2  [2,1]p2
 2024-06-11T08:57:42.791265+0000  2024-05-29T07:13:57.867295+0000
       959  periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000
10.4a  5217577   4909174          0        0   4194304   5960086599
9511657  1910      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'180982438  79875:381432369  [0,1,2]p0  [1,2]p1
 2024-06-12T02:31:42.576663+0000  2024-06-01T09:08:07.860775+0000
       480  periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000
10.4b  5220270   5220270          0        0    159808   6313986186
9763672  1934      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181279214  79875:342509944  [2,1,0]p2  [2,1]p2
 2024-06-11T11:03:38.997373+0000  2024-06-11T11:03:38.997373+0000
      3970  periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000
10.4c  5217864   5217864          0        0   4194304   6104537460
9679170  1948      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181081348  79875:341937760  [1,0,2]p1  [1,2]p1
 2024-06-12T06:09:49.667680+0000  2024-06-01T13:06:12.938513+0000
       520  periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000
10.4d  5217604   4905278          0        0   8388608   6250536027
9765571  1917      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'179562016  79875:387768172  [0,1,2]p0  [1,2]p1
 2024-06-11T13:38:28.479053+0000  2024-06-11T13:38:28.479053+0000
      2004  periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000
10.4e  5220202   5220202          0        0  12215022   5781813383
9291138  1940      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180360470  79875:377712114  [1,0,2]p1  [1,2]p1
 2024-06-01T22:06:12.347760+0000  2024-05-25T19:34:49.805104+0000
       982  queued for scrub
10.4f  5215243   4561466          0        0         0   6099447266
9605810  1930      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181794065  79875:396660094  [2,1,0]p2  [2,1]p2
 2024-06-02T13:35:21.451128+0000  2024-06-02T13:35:21.451128+0000
      7316  queued for scrub
10.50  5215219   5184003          0        0         0   6151789627
9674779  1871      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181695151  79875:405060298  [0,2,1]p0  [1,2]p1
 2024-06-11T16:42:00.221013+0000  2024-06-11T16:42:00.221013+0000
      2385  periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000
10.51  5218666   5218666          0        0   8388608   5774655330
9331373  1920      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180711664  79875:342978462  [0,1,2]p0  [1,2]p1
 2024-06-11T14:31:51.402844+0000  2024-05-24T11:26:15.539677+0000
       730  periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000
10.52  5218435   5187050          0        0   4194304   6375954134
9843553  1935      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180834874  79875:341502004  [0,1,2]p0  [1,2]p1
 2024-06-11T08:41:43.586833+0000  2024-06-11T08:41:43.586833+0000
      8368  periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000
10.53  5216822   5216822          0        0  10001995   5786896551
9368323  1928      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181337168  79875:402047940  [0,2,1]p0  [1,2]p1
 2024-06-12T04:54:21.147742+0000  2024-05-23T23:02:36.154347+0000
       557  periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000
10.54  5213949   4567626          0        0   8388608   6095478377
9664814  1944      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'182179986  79875:421118572  [2,1,0]p2  [2,1]p2
 2024-06-12T09:19:30.470319+0000  2024-06-01T17:56:32.089348+0000
       551  periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000
10.55  5217389   5217389          0        0   4194948   5905845521
9403344  1950      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'183481821  79875:341223864  [1,2,0]p1  [1,2]p1
 2024-06-11T15:31:05.524698+0000  2024-05-29T11:50:10.322128+0000
       584  periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000
10.56  5219142   5219142          0        0         0   6311549101
9773102  1897      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'184062267  79875:408792597  [1,2,0]p1  [1,2]p1
 2024-06-10T22:17:28.523159+0000  2024-06-10T22:17:28.523159+0000
      8211  periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000
10.57  5219035   4894148          0        0       180   5862209193
9396761  1895      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'181886155  79875:340148291  [0,1,2]p0  [1,2]p1
 2024-06-10T22:33:19.364284+0000  2024-05-28T12:05:26.762682+0000
       951  periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000
10.58  5218318   5185905          0        0   9878587   5911568798
9459707  1875      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179109213  79875:390200207  [2,0,1]p2  [2,1]p2
 2024-06-11T13:08:37.380203+0000  2024-05-30T10:19:35.091370+0000
       578  periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000
10.59  5218315   5185838          0        0   4194304   5727052522
9214424  1875      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181700141  79875:400297310  [2,0,1]p2  [2,1]p2
 2024-06-11T11:15:00.221765+0000  2024-05-21T11:28:39.056382+0000
       682  periodic deep scrub scheduled @ 2024-06-16T13:00:43.750185+0000
10.5a  5218301   5218301          0        0   4194304   5833097049
9365354  1942      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180699909  79875:393344045  [1,2,0]p1  [1,2]p1
 2024-06-12T01:50:51.954645+0000  2024-05-25T00:11:49.917556+0000
       497  periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000
10.5b  5215134   5215134          0        0   4194802   6070043467
9585134  1892      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181149500  79875:342539489  [2,0,1]p2  [2,1]p2
 2024-06-12T08:19:32.265112+0000  2024-06-02T11:33:23.094569+0000
       714  periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000
10.5c  5221503   5221503          0        0   4194304   5989312430
9524556  1939      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180748682  79875:340750417  [1,2,0]p1  [1,2]p1
 2024-06-11T09:37:59.483197+0000  2024-05-28T03:41:58.475567+0000
      1021  periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000
10.5d  5222202   5222202          0        0   8388608   6031557641
9520790  1928      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'179508607  79875:399919359  [1,0,2]p1  [1,2]p1
 2024-06-11T17:10:33.165278+0000  2024-05-30T05:53:10.772736+0000
       575  periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000
10.5e  5212770   4506643          0        0         0   5830165136
9388228  1938      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180002484  79875:391844220  [2,0,1]p2  [2,1]p2
 2024-06-11T22:32:49.026280+0000  2024-05-24T15:31:50.362016+0000
       523  periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000
10.5f  5221504   5221504          0        0   4194394   5807975316
9387383  1879      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181161493  79875:408530090  [0,2,1]p0  [1,2]p1
 2024-06-11T15:10:33.261745+0000  2024-05-25T06:37:01.629711+0000
       845  periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000
10.60  5217784   5186615          0        0   4194304   6208553838
9749044  1962      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181569072  79875:392211608  [0,1,2]p0  [1,2]p1
 2024-06-11T21:44:35.372861+0000  2024-06-11T21:44:35.372861+0000
      1505  periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000
10.61  5221292   4586636          0        0   8388608   5766320671
9280184  1910      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180945714  79875:344272511  [0,1,2]p0  [1,2]p1
 2024-06-11T16:51:25.519891+0000  2024-05-24T17:37:21.814566+0000
       566  periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000
10.62  5218498   4699002          0        0   4194466   6027128219
9555111  1884      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180387699  79875:341929273  [1,2,0]p1  [1,2]p1
 2024-06-12T03:10:08.547987+0000  2024-06-01T02:47:43.109035+0000
       509  periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000
10.63  5219101   5219101          0        0         0   6040382110
9500040  1885      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181623402  79875:389756489  [0,1,2]p0  [1,2]p1
 2024-06-02T23:50:14.947543+0000  2024-06-02T23:50:14.947543+0000
     13925  queued for scrub
10.64  5217025   5217025          0        0   2277792   6222337345
9701950  1866      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181853168  79875:406174284  [1,2,0]p1  [1,2]p1
 2024-06-02T04:57:41.872723+0000  2024-06-02T04:57:41.872723+0000
      6394  queued for scrub
10.65  5217908   4590550          0        0   4194326   6223151288
9709345  1916      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181196382  79875:340107600  [2,0,1]p2  [2,1]p2
 2024-06-11T15:53:28.725001+0000  2024-06-11T15:53:28.725001+0000
      1343  periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000
10.66  5218911   4893654          0        0   4194304   5815703055
9338987  1939      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'183062531  79875:395193073  [1,0,2]p1  [1,2]p1
 2024-06-12T01:10:56.919233+0000  2024-05-25T04:50:04.300293+0000
       590  periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000
10.67  5217539   4587208          0        0   4194304   6095526883
9640323  1955      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181875315  79875:342878178  [2,1,0]p2  [2,1]p2
 2024-06-12T07:58:41.522730+0000  2024-06-01T07:20:59.951966+0000
       553  periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000
10.68  5215852   4505902          0        0   8388608   6093858586
9596421  1926      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'179490643  79875:379632203  [0,1,2]p0  [1,2]p1
 2024-06-11T16:02:14.995195+0000  2024-05-30T22:02:42.335910+0000
       526  periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000
10.69  5216596   5216596          0        0   4194304   5833268044
9400414  1899      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181876551  79875:387059449  [1,2,0]p1  [1,2]p1
 2024-06-11T17:00:57.111851+0000  2024-05-23T07:05:05.989427+0000
       572  periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000
10.6a  5214106   4497370          0        0         0   5745504618
9296938  1869      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180109669  79875:381871149  [2,1,0]p2  [2,1]p2
 2024-06-12T00:27:45.155477+0000  2024-05-25T09:22:19.570011+0000
       494  periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000
10.6b  5219339   5219339          0        0         0   5801390847
9370084  1957      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180041524  79875:342270608  [0,1,2]p0  [1,2]p1
 2024-06-12T03:01:39.358464+0000  2024-05-23T01:04:22.522035+0000
       488  periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000
10.6c  5217917   5217917          0        0       554   6100979305
9612250  1955      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180968918  79875:341488318  [1,2,0]p1  [1,2]p1
 2024-06-11T18:47:35.969345+0000  2024-05-31T11:54:59.559316+0000
       588  periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000
10.6d  5221048   4550388          0        0   4194304   6024067862
9621947  1927      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'179373269  79875:388095690  [2,1,0]p2  [2,1]p2
 2024-06-12T03:26:37.931310+0000  2024-05-31T04:10:41.991457+0000
       502  periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000
10.6e  5215550   4888179          0        0   8388608   6110450001
9666892  1906      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'180885974  79875:380260263  [0,1,2]p0  [1,2]p1
 2024-06-11T20:18:23.657558+0000  2024-06-11T20:18:23.657558+0000
      1218  periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000
10.6f  5215850   4566820          0        0         0   6107146788
9634911  1923      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181752773  79875:396708021  [2,1,0]p2  [2,1]p2
 2024-06-10T17:32:51.735004+0000  2024-06-10T17:32:51.735004+0000
     10119  periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000
10.70  5218211   4672319          0        0   4194304   5793441438
9371563  1881      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180405012  79875:404578822  [1,2,0]p1  [1,2]p1
 2024-06-03T04:46:35.130293+0000  2024-05-22T07:50:20.898441+0000
       969  queued for deep scrub
10.71  5218299   4599312          0        0   4194304   5807950891
9349720  1906      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180690682  79875:344108361  [0,2,1]p0  [1,2]p1
 2024-06-12T03:18:16.645927+0000  2024-05-23T09:32:56.055703+0000
       487  periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000
10.72  5223470   4922347          0        0   8388608   6416468325
9922777  1958      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'180797342  79875:342356916  [0,1,2]p0  [1,2]p1
 2024-06-12T05:42:47.545762+0000  2024-06-12T05:42:47.545762+0000
      1307  periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000
10.73  5218301   4520651          0        0         0   6293832301
9738943  1953      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'182246986  79875:403792984  [2,0,1]p2  [2,1]p2
 2024-06-11T20:59:02.079157+0000  2024-06-11T20:59:02.079157+0000
      1399  periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000
10.74  5219061   4595112          0        0         0   5974775781
9469653  1943      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181947609  79875:420571221  [2,1,0]p2  [2,1]p2
 2024-06-03T02:17:19.687482+0000  2024-05-28T08:43:38.825954+0000
       927  queued for scrub
10.75  5220994   4596634          0        0  25165824   5822315216
9362548  1877      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'180376305  79875:338847231  [2,0,1]p2  [2,1]p2
 2024-06-03T05:52:28.743255+0000  2024-05-24T05:11:00.211501+0000
       981  queued for scrub
10.76  5222111   4892295          0        0         0   6067998823
9574254  1935      3000    active+undersized+degraded+remapped+backfilling
    2h  79875'183871180  79875:409218100  [1,0,2]p1  [1,2]p1
 2024-06-03T04:14:04.513004+0000  2024-06-03T04:14:04.513004+0000
      6987  queued for scrub
10.77  5219020   4574683          0        0         0   5889074277
9460824  1896      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181614825  79875:342558162  [2,0,1]p2  [2,1]p2
 2024-06-12T08:07:38.169523+0000  2024-05-27T15:15:35.587972+0000
       536  periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000
10.78  5220442   4693235          0        0   8388608   5879566566
9395501  1923      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'179031456  79875:391462415  [1,0,2]p1  [1,2]p1
 2024-06-02T09:32:42.834344+0000  2024-05-27T10:41:23.822124+0000
       994  queued for scrub
10.79  5219734   4702117          0        0   4194304   6129933545
9646669  1926      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'182574416  79875:402249812  [1,2,0]p1  [1,2]p1
 2024-06-12T05:52:02.959814+0000  2024-06-01T00:53:49.548055+0000
       555  periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000
10.7a  5215303   4526846          0        0         0   6316171107
9817029  1932      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181765231  79875:395877057  [2,0,1]p2  [2,1]p2
 2024-06-12T02:23:42.692096+0000  2024-06-12T02:23:42.692096+0000
      1457  periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000
10.7b  5217213   4606816          0        0   4194304   6226859283
9718621  1952      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'181413541  79875:343968658  [1,2,0]p1  [1,2]p1
 2024-06-11T23:52:28.414752+0000  2024-06-11T23:52:28.414752+0000
      1308  periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000
10.7c  5215075   5215075          0        0   4194304   6013877208
9602345  1867      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'180991860  79875:341775615  [0,2,1]p0  [1,2]p1
 2024-06-11T23:30:40.614412+0000  2024-05-31T14:10:30.542603+0000
       503  periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000
10.7d  5220700   4569720          0        0   5837609   6258736025
9764836  1897      3000    active+undersized+degraded+remapped+backfilling
    3h  79875'179741885  79875:401448508  [2,1,0]p2  [2,1]p2
 2024-06-12T04:45:05.030905+0000  2024-06-12T04:45:05.030905+0000
      1565  periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000
10.7e  5217113   5217113          0        0  16777216   6072081676
9598390  1902      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'181040361  79875:391032348  [1,2,0]p1  [1,2]p1
 2024-06-11T17:37:26.589968+0000  2024-06-11T17:37:26.589968+0000
      1613  periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000
10.7f  5218719   5218719          0        0   9909061   6130469539
9657824  1943      3000  active+undersized+degraded+remapped+backfill_wait
    3h  79875'182283514  79875:408550888  [1,2,0]p1  [1,2]p1
 2024-06-11T13:05:03.552526+0000  2024-06-11T13:05:03.552526+0000
      1796  periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000

* NOTE: Omap statistics are gathered during deep scrub and may be
inaccurate soon afterwards depending on utilization. See
http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for
further details.



sudo ceph osd df
ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP      META
  AVAIL    %USE   VAR   PGS  STATUS
 0    ssd   3.63869   1.00000  3.6 TiB   82 GiB   50 MiB    40 GiB   42 GiB
 3.6 TiB   2.21  0.03    0      up
 1    ssd   3.63869   1.00000  3.6 TiB  2.0 TiB   16 GiB  1021 GiB  1.0 TiB
 1.6 TiB  55.41  0.75  161      up
 2    ssd   3.63869   1.00000  3.6 TiB  3.0 TiB   16 GiB   1.1 TiB  1.9 TiB
 676 GiB  81.86  1.11  161      up


sudo ceph df detail
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    1.2 PiB  308 TiB  871 TiB   871 TiB      73.86
ssd     11 TiB  5.8 TiB  5.1 TiB   5.1 TiB      46.51
TOTAL  1.2 PiB  314 TiB  876 TiB   876 TiB      73.61

--- POOLS ---
POOL      ID   PGS   STORED   (DATA)   (OMAP)  OBJECTS     USED   (DATA)
(OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR
 UNDER COMPR
.mgr       1     1  827 MiB  827 MiB      0 B      208  1.6 GiB  1.6 GiB
   0 B   0.11    734 GiB            N/A       50 GiB    N/A         0 B
     0 B
metadata  10   128  1.0 TiB  647 MiB  1.0 TiB  667.90M  2.1 TiB  1.3 GiB
 2.1 TiB  59.45    719 GiB            N/A      3.5 TiB    N/A         0 B
       0 B
data      11  2048  288 TiB  288 TiB      0 B  566.97M  863 TiB  863 TiB
   0 B  81.10     67 TiB            N/A          N/A    N/A     3.4 TiB
 6.8 TiB
ssd-data  18    32   14 GiB   14 GiB      0 B  265.17k   29 GiB   29 GiB
   0 B   1.90    734 GiB            N/A          N/A    N/A         0 B
     0 B


sudo ceph balancer status
{
    "active": true,
    "last_optimize_duration": "0:00:00.000170",
    "last_optimize_started": "Wed Jun 12 13:14:49 2024",
    "mode": "upmap",
    "no_optimization_needed": true,
    "optimize_result": "Some objects (0.172816) are degraded; try again
later",
    "plans": []
}



[image: ariadne.ai Logo] Lars Köppel
Developer
Email: lars.koeppel@xxxxxxxxxx
Phone: +49 6221 5993580 <+4962215993580>
ariadne.ai (Germany) GmbH
Häusserstraße 3, 69115 Heidelberg
Amtsgericht Mannheim, HRB 744040
Geschäftsführer: Dr. Fabian Svara
https://ariadne.ai


On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri <anthony.datri@xxxxxxxxx>
wrote:

> If you have:
>
> * pg_num too low (defaults are too low)
> * pg_num not a power of 2
> * pg_num != number of OSDs in the pool
> * balancer not enabled
>
> any of those might result in imbalance.
>
> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote:
> >
> > I don't have any good explanation at this point. Can you share some more
> information like:
> >
> > ceph pg ls-by-pool <cephfs_metadata>
> > ceph osd df (for the relevant OSDs)
> > ceph df
> >
> > Thanks,
> > Eugen
> >
> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >
> >> Since my last update the size of the largest OSD increased by 0.4 TiB
> while
> >> the smallest one only increased by 0.1 TiB. How is this possible?
> >>
> >> Because the metadata pool reported to have only 900MB space left, I
> stopped
> >> the hot-standby MDS. This gave me 8GB back but these filled up in the
> last
> >> 2h.
> >> I think I have to zap the next OSD because the filesystem is getting
> read
> >> only...
> >>
> >> How is it possible that an OSD has over 1 TiB less data on it after a
> >> rebuild? And how is it possible to have so different sizes of OSDs?
> >>
> >>
> >> [image: ariadne.ai Logo] Lars Köppel
> >> Developer
> >> Email: lars.koeppel@xxxxxxxxxx
> >> Phone: +49 6221 5993580 <+4962215993580>
> >> ariadne.ai (Germany) GmbH
> >> Häusserstraße 3, 69115 Heidelberg
> >> Amtsgericht Mannheim, HRB 744040
> >> Geschäftsführer: Dr. Fabian Svara
> >> https://ariadne.ai
> >>
> >>
> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel <lars.koeppel@xxxxxxxxxx>
> wrote:
> >>
> >>> Only in warning mode. And there were no PG splits or merges in the
> last 2
> >>> month.
> >>>
> >>>
> >>> [image: ariadne.ai Logo] Lars Köppel
> >>> Developer
> >>> Email: lars.koeppel@xxxxxxxxxx
> >>> Phone: +49 6221 5993580 <+4962215993580>
> >>> ariadne.ai (Germany) GmbH
> >>> Häusserstraße 3, 69115 Heidelberg
> >>> Amtsgericht Mannheim, HRB 744040
> >>> Geschäftsführer: Dr. Fabian Svara
> >>> https://ariadne.ai
> >>>
> >>>
> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> wrote:
> >>>
> >>>> I don't think scrubs can cause this. Do you have autoscaler enabled?
> >>>>
> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >>>>
> >>>> > Hi,
> >>>> >
> >>>> > thank you for your response.
> >>>> >
> >>>> > I don't think this thread covers my problem, because the OSDs for
> the
> >>>> > metadata pool fill up at different rates. So I would think this is
> no
> >>>> > direct problem with the journal.
> >>>> > Because we had earlier problems with the journal I changed some
> >>>> > settings(see below). I already restarted all MDS multiple times but
> no
> >>>> > change here.
> >>>> >
> >>>> > The health warnings regarding cache pressure resolve normally after
> a
> >>>> > short period of time, when the heavy load on the client ends.
> Sometimes
> >>>> it
> >>>> > stays a bit longer because an rsync is running and copying data on
> the
> >>>> > cluster(rsync is not good at releasing the caps).
> >>>> >
> >>>> > Could it be a problem if scrubs run most of the time in the
> background?
> >>>> Can
> >>>> > this block any other tasks or generate new data itself?
> >>>> >
> >>>> > Best regards,
> >>>> > Lars
> >>>> >
> >>>> >
> >>>> > global                      basic     mds_cache_memory_limit
> >>>> >     17179869184
> >>>> > global                      advanced  mds_max_caps_per_client
> >>>> >    16384
> >>>> > global                      advanced
> >>>> mds_recall_global_max_decay_threshold
> >>>> >    262144
> >>>> > global                      advanced  mds_recall_max_decay_rate
> >>>> >    1.000000
> >>>> > global                      advanced  mds_recall_max_decay_threshold
> >>>> >     262144
> >>>> > mds                         advanced  mds_cache_trim_threshold
> >>>> >     131072
> >>>> > mds                         advanced  mds_heartbeat_grace
> >>>> >    120.000000
> >>>> > mds                         advanced  mds_heartbeat_reset_grace
> >>>> >    7400
> >>>> > mds                         advanced  mds_tick_interval
> >>>> >    3.000000
> >>>> >
> >>>> >
> >>>> > [image: ariadne.ai Logo] Lars Köppel
> >>>> > Developer
> >>>> > Email: lars.koeppel@xxxxxxxxxx
> >>>> > Phone: +49 6221 5993580 <+4962215993580>
> >>>> > ariadne.ai (Germany) GmbH
> >>>> > Häusserstraße 3, 69115 Heidelberg
> >>>> > Amtsgericht Mannheim, HRB 744040
> >>>> > Geschäftsführer: Dr. Fabian Svara
> >>>> > https://ariadne.ai
> >>>> >
> >>>> >
> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx> wrote:
> >>>> >
> >>>> >> Hi,
> >>>> >>
> >>>> >> can you check if this thread [1] applies to your situation? You
> don't
> >>>> >> have multi-active MDS enabled, but maybe it's still some journal
> >>>> >> trimming, or maybe misbehaving clients? In your first post there
> were
> >>>> >> health warnings regarding cache pressure and cache size. Are those
> >>>> >> resolved?
> >>>> >>
> >>>> >> [1]
> >>>> >>
> >>>> >>
> >>>>
> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE
> >>>> >>
> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> >>>> >>
> >>>> >> > Hello everyone,
> >>>> >> >
> >>>> >> > short update to this problem.
> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected
> size
> >>>> >> ~50%).
> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They
> jumped up
> >>>> and
> >>>> >> > down a lot but the higher one has now also reached 'nearfull'
> >>>> status. How
> >>>> >> > is this possible? What is going on?
> >>>> >> >
> >>>> >> > Does anyone have a solution how to fix this without zapping the
> OSD?
> >>>> >> >
> >>>> >> > Best regards,
> >>>> >> > Lars
> >>>> >> >
> >>>> >> >
> >>>> >> > [image: ariadne.ai Logo] Lars Köppel
> >>>> >> > Developer
> >>>> >> > Email: lars.koeppel@xxxxxxxxxx
> >>>> >> > Phone: +49 6221 5993580 <+4962215993580>
> >>>> >> > ariadne.ai (Germany) GmbH
> >>>> >> > Häusserstraße 3, 69115 Heidelberg
> >>>> >> > Amtsgericht Mannheim, HRB 744040
> >>>> >> > Geschäftsführer: Dr. Fabian Svara
> >>>> >> > https://ariadne.ai
> >>>> >> > _______________________________________________
> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> >>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>> >>
> >>>> >>
> >>>> >> _______________________________________________
> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>> >>
> >>>>
> >>>>
> >>>>
> >>>>
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux