I am happy to help you with as much information as possible. I probably
just don't know where to look for it.
Below are the requested information. The cluster is rebuilding the
zapped OSD at the moment. This will probably take the next few days.
sudo ceph pg ls-by-pool metadata
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES*
OMAP_KEYS* LOG LOG_DUPS STATE
SINCE VERSION REPORTED UP ACTING
SCRUB_STAMP DEEP_SCRUB_STAMP
LAST_SCRUB_DURATION SCRUB_SCHEDULING
10.0 5217325 4994695 0 0 4194304 5880891340
9393865 1885 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'180849582 79875:391519635 [0,1,2]p0 [1,2]p1
2024-06-11T09:08:09.829362+0000 2024-05-28T05:52:59.321589+0000
627 periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000
10.1 5214785 5193424 0 0 0 5843682713
9410150 1912 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180914288 79875:342746928 [2,1,0]p2 [2,1]p2
2024-06-01T15:56:28.927288+0000 2024-05-27T03:31:37.682966+0000
966 queued for scrub
10.2 5218432 5187168 0 0 0 6402011266
9812513 1874 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180970531 79875:341340204 [0,1,2]p0 [1,2]p1
2024-06-11T13:40:58.994256+0000 2024-06-11T13:40:58.994256+0000
1942 periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000
10.3 5217413 5217413 0 0 8388788 5766005023
9271787 1923 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181012233 79875:388295881 [1,0,2]p1 [1,2]p1
2024-06-12T00:35:56.965547+0000 2024-05-23T19:54:56.121729+0000
492 periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000
10.4 5220069 5220069 0 0 12583466 6027548724
9537290 1959 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181576075 79875:405295868 [1,2,0]p1 [1,2]p1
2024-06-11T17:47:22.923514+0000 2024-05-31T02:06:55.339574+0000
581 periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000
10.5 5216162 5211999 0 0 4194304 5941347251
9542764 1930 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180455793 79875:338418517 [2,1,0]p2 [2,1]p2
2024-06-11T22:50:16.170708+0000 2024-05-30T23:49:54.316379+0000
528 periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000
10.6 5216100 4980459 0 0 4521984 6428088514
9850762 1911 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'184045876 79875:396809795 [0,2,1]p0 [1,2]p1
2024-06-11T22:24:05.102716+0000 2024-06-11T22:24:05.102716+0000
1082 periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000
10.7 5218232 5218232 0 0 4194304 6377065363
9849360 1919 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182672562 79875:342449062 [1,0,2]p1 [1,2]p1
2024-06-11T06:22:15.689422+0000 2024-06-11T06:22:15.689422+0000
8225 periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000
10.8 5219620 5182816 0 0 0 6167304290
9691796 1896 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179628377 79875:378022884 [2,1,0]p2 [2,1]p2
2024-06-11T22:06:01.386763+0000 2024-06-11T22:06:01.386763+0000
1286 periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000
10.9 5219448 5164591 0 0 8388698 5796048346
9338312 1868 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181739392 79875:387412389 [2,1,0]p2 [2,1]p2
2024-06-12T05:21:00.586747+0000 2024-05-26T11:10:59.780673+0000
539 periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000
10.a 5219861 5163635 0 0 12582912 5841839055
9387200 1916 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180205688 79875:379381294 [1,2,0]p1 [1,2]p1
2024-06-11T12:35:05.571200+0000 2024-05-22T11:07:16.041773+0000
1093 periodic deep scrub scheduled @ 2024-06-17T05:21:40.136463+0000
10.b 5217949 5217949 0 0 16777216 5935863260
9462127 1881 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181655745 79875:343806807 [0,1,2]p0 [1,2]p1
2024-06-11T22:41:28.976920+0000 2024-05-26T08:43:29.217457+0000
520 periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000
10.c 5221697 5217118 0 0 4194304 6015217841
9574445 1928 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180892826 79875:341490398 [2,1,0]p2 [2,1]p2
2024-06-11T09:20:58.443473+0000 2024-05-30T00:13:50.306507+0000
768 periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000
10.d 5217727 4908764 0 0 0 5825598519
9349877 1930 3000 active+undersized+degraded+remapped+backfilling
2h 79875'179050347 79875:387455993 [0,1,2]p0 [1,2]p1
2024-06-02T07:27:27.873631+0000 2024-05-20T03:58:30.225170+0000
952 queued for deep scrub
10.e 5215040 5210572 0 0 4194304 5790634469
9327651 1905 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180049044 79875:377196119 [2,1,0]p2 [2,1]p2
2024-06-12T05:03:08.521423+0000 2024-05-22T13:47:46.753131+0000
527 periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000
10.f 5217202 4885952 0 0 8388630 6005274626
9589178 1917 3000 active+undersized+degraded+remapped+backfilling
2h 79875'181953069 79875:396402453 [0,2,1]p0 [1,2]p1
2024-06-11T12:20:49.661741+0000 2024-06-11T12:20:49.661741+0000
3949 periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000
10.10 5214985 5157391 0 0 4194394 6275024632
9721505 1922 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181801708 79875:395055895 [2,0,1]p2 [2,1]p2
2024-06-11T01:49:47.831191+0000 2024-06-11T01:49:47.831191+0000
9579 periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000
10.11 5219236 4983134 0 0 4194304 5787580073
9316893 1901 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'181011816 79875:343488767 [1,0,2]p1 [1,2]p1
2024-06-03T00:06:14.862209+0000 2024-05-27T00:05:56.468361+0000
960 queued for scrub
10.12 5214103 5182770 0 0 90 5970750188
9505613 1891 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180210613 79875:341042616 [0,1,2]p0 [1,2]p1
2024-06-12T06:39:48.252419+0000 2024-05-28T01:32:03.381577+0000
527 periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000
10.13 5214580 5214580 0 0 8388642 5793432968
9335940 1957 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182394614 79875:393423214 [1,2,0]p1 [1,2]p1
2024-06-10T17:52:04.690203+0000 2024-05-22T03:39:02.453336+0000
1151 periodic deep scrub scheduled @ 2024-06-16T00:28:37.682765+0000
10.14 5218591 5218591 0 0 5053325 6046268958
9582481 1915 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'184576050 79875:425213800 [2,0,1]p2 [2,1]p2
2024-06-03T05:36:07.656965+0000 2024-05-28T22:46:16.814318+0000
985 queued for scrub
10.15 5215919 4907184 0 0 11939335 5752284246
9285870 1889 3000 active+undersized+degraded+remapped+backfilling
2h 79875'179713036 79875:337938973 [0,2,1]p0 [1,2]p1
2024-06-11T14:06:50.138992+0000 2024-05-24T07:52:22.522455+0000
786 periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000
10.16 5221091 5221091 0 0 8388608 5966308565
9487816 1916 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'183445984 79875:399413009 [0,2,1]p0 [1,2]p1
2024-06-03T02:01:53.649417+0000 2024-06-03T02:01:53.649417+0000
6935 queued for scrub
10.17 5218069 5194732 0 0 4194304 6231638001
9735915 1890 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182390128 79875:340719915 [2,0,1]p2 [2,1]p2
2024-06-12T06:31:01.523916+0000 2024-06-12T06:31:01.523916+0000
1272 periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000
10.18 5214253 5159897 0 0 0 6128506469
9673261 1886 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179754208 79875:389150273 [2,1,0]p2 [2,1]p2
2024-06-12T02:53:31.027154+0000 2024-06-12T02:53:31.027154+0000
1308 periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000
10.19 5220131 5183342 0 0 0 5869775360
9460583 1953 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181750923 79875:393162989 [2,0,1]p2 [2,1]p2
2024-06-11T19:21:24.106504+0000 2024-05-22T15:42:01.721091+0000
522 periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000
10.1a 5217968 5186700 0 0 4194304 6183652625
9733847 1896 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181056240 79875:394241128 [0,2,1]p0 [1,2]p1
2024-06-02T15:37:49.730965+0000 2024-06-02T15:37:49.730965+0000
7347 queued for scrub
10.1b 5221259 5161879 0 0 0 5989030303
9561035 1875 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181052305 79875:343144823 [0,1,2]p0 [1,2]p1
2024-06-12T06:01:07.213894+0000 2024-06-02T00:23:51.063901+0000
543 periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000
10.1c 5217034 5217034 0 0 554 5979861913
9504442 1875 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180913313 79875:341328275 [1,2,0]p1 [1,2]p1
2024-06-01T20:02:47.025129+0000 2024-06-01T20:02:47.025129+0000
6568 queued for scrub
10.1d 5219691 4997027 0 0 12582912 6081321658
9599164 1918 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'179638837 79875:391808225 [0,2,1]p0 [1,2]p1
2024-06-11T20:27:02.506715+0000 2024-05-31T22:58:06.578167+0000
517 periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000
10.1e 5213369 5208776 0 0 4194304 6295629499
9765317 1941 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181145210 79875:391326487 [2,1,0]p2 [2,1]p2
2024-06-12T07:49:28.432084+0000 2024-06-12T07:49:28.432084+0000
1990 periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000
10.1f 5218399 5000094 0 0 8388608 5821063844
9421222 1896 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'180960296 79875:400635456 [0,2,1]p0 [1,2]p1
2024-06-11T13:53:40.278050+0000 2024-05-22T19:39:02.345663+0000
761 periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000
10.20 5216437 5185379 0 0 0 6280994122
9719574 1945 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181610117 79875:391874431 [0,1,2]p0 [1,2]p1
2024-06-11T12:58:57.754648+0000 2024-06-11T12:58:57.754648+0000
2288 periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000
10.21 5217349 5217349 0 0 8388698 6348931429
9822197 1867 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181724357 79875:343883762 [0,1,2]p0 [1,2]p1
2024-06-11T18:28:34.044546+0000 2024-06-11T18:28:34.044546+0000
1817 periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000
10.22 5215544 4943475 0 0 4194304 5982815232
9482159 1899 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'180230331 79875:340839853 [2,0,1]p2 [2,1]p2
2024-06-11T15:21:21.994674+0000 2024-05-30T08:01:03.301650+0000
649 periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000
10.23 5217758 5217758 0 0 146 5804086623
9336298 1920 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180948432 79875:388158699 [1,2,0]p1 [1,2]p1
2024-06-03T05:19:32.311846+0000 2024-05-27T08:13:15.250058+0000
971 queued for scrub
10.24 5221505 5221505 0 0 0 6360995644
9816333 1925 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182198758 79875:406847098 [0,2,1]p0 [1,2]p1
2024-06-12T04:00:44.315682+0000 2024-06-12T04:00:44.315682+0000
2046 periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000
10.25 5214911 5214911 0 0 12583002 5726320657
9264352 1910 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180132024 79875:338167995 [1,2,0]p1 [1,2]p1
2024-06-11T14:43:47.076906+0000 2024-05-23T12:16:34.271856+0000
715 periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000
10.26 5217588 4973667 0 0 0 6040775867
9598406 1927 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'188144536 79875:400277134 [2,0,1]p2 [2,1]p2
2024-06-02T06:55:09.016528+0000 2024-06-02T06:55:09.016528+0000
7047 queued for scrub
10.27 5218490 5218490 0 0 8388630 6265942186
9818746 1941 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182400328 79875:342052414 [1,2,0]p1 [1,2]p1
2024-06-12T01:01:06.674323+0000 2024-06-12T01:01:06.674323+0000
1500 periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000
10.28 5221153 5221153 0 0 0 6049095371
9597182 1913 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179438777 79875:378265375 [1,2,0]p1 [1,2]p1
2024-06-12T04:09:58.896192+0000 2024-06-01T04:32:46.011974+0000
553 periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000
10.29 5214066 5214066 0 0 4194304 5767555623
9316899 1937 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181568997 79875:387683159 [0,2,1]p0 [1,2]p1
2024-06-10T22:48:57.757493+0000 2024-05-21T07:14:53.348341+0000
938 periodic deep scrub scheduled @ 2024-06-16T06:24:55.108695+0000
10.2a 5216314 4977811 0 0 4194304 6012567810
9563804 1951 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'181140734 79875:381017643 [2,1,0]p2 [2,1]p2
2024-06-11T09:57:29.097708+0000 2024-05-30T03:47:22.798044+0000
1169 periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000
10.2b 5219620 5219620 0 0 8388608 5829917912
9412426 1889 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180465782 79875:342263856 [1,0,2]p1 [1,2]p1
2024-06-12T08:30:08.941782+0000 2024-05-24T22:00:29.535205+0000
636 periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000
10.2c 5214087 5214087 0 0 0 5874182922
9375708 1871 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180616041 79875:341244736 [1,2,0]p1 [1,2]p1
2024-06-10T23:10:08.184410+0000 2024-05-27T17:41:07.583255+0000
1268 periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000
10.2d 5213968 4984822 0 0 4337824 5835754026
9417033 1947 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'178824607 79875:386427308 [2,0,1]p2 [2,1]p2
2024-06-12T09:27:43.774331+0000 2024-05-27T05:33:45.293266+0000
494 periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000
10.2e 5216263 4974456 0 0 12582912 6025714323
9553880 1940 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'180639631 79875:378284872 [1,0,2]p1 [1,2]p1
2024-06-12T01:19:42.477063+0000 2024-05-31T08:42:41.293335+0000
526 periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000
10.2f 5220057 5220057 0 0 13839831 5804027574
9370862 1882 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180961915 79875:394513378 [1,0,2]p1 [1,2]p1
2024-06-11T19:58:05.145147+0000 2024-05-24T20:18:32.301831+0000
535 periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000
10.30 5217090 4955469 0 0 12582934 6438521431
9924825 1869 3000 active+undersized+degraded+remapped+backfill_wait
2h 79875'181488488 79875:404940886 [2,1,0]p2 [2,1]p2
2024-06-12T07:16:18.095439+0000 2024-06-12T07:16:18.095439+0000
2190 periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000
10.31 5218176 5218176 0 0 4194304 6215016361
9752698 1897 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181622879 79875:343554151 [1,2,0]p1 [1,2]p1
2024-06-12T09:00:15.941065+0000 2024-06-12T09:00:15.941065+0000
1806 periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000
10.32 5223171 4704732 0 0 8388608 6205669256
9800320 1920 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180911263 79875:342754128 [0,2,1]p0 [1,2]p1
2024-06-11T19:49:09.010583+0000 2024-06-11T19:49:09.010583+0000
1664 periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000
10.33 5215341 4525338 0 0 4194304 6066965588
9635403 1888 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181467497 79875:402958597 [1,2,0]p1 [1,2]p1
2024-06-12T01:59:23.933458+0000 2024-05-31T17:01:38.620377+0000
512 periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000
10.34 5217342 5217342 0 0 8388608 5999920961
9570527 1956 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182013407 79875:419219483 [1,0,2]p1 [1,2]p1
2024-06-12T09:10:17.390128+0000 2024-06-01T15:40:23.241769+0000
601 periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000
10.35 5222114 5222114 0 0 8388608 6021420075
9571300 1952 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180948342 79875:338598678 [1,2,0]p1 [1,2]p1
2024-06-11T22:59:17.017317+0000 2024-05-30T15:59:15.553266+0000
539 periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000
10.36 5223657 5223657 0 0 4331271 5844578100
9347405 1903 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'183363403 79875:408484683 [0,2,1]p0 [1,2]p1
2024-06-11T18:37:46.653097+0000 2024-05-25T14:13:51.648712+0000
553 periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000
10.37 5219248 5219248 0 0 4194304 6072826085
9579049 1936 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182091890 79875:341545087 [1,0,2]p1 [1,2]p1
2024-06-11T14:19:40.744743+0000 2024-05-29T18:40:12.602405+0000
771 periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000
10.38 5214022 5214022 0 0 4194858 5780805541
9299038 1951 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'178809936 79875:390113817 [1,0,2]p1 [1,2]p1
2024-06-02T00:55:22.046433+0000 2024-05-25T16:49:10.279003+0000
931 queued for scrub
10.39 5220692 4542094 0 0 0 6401799077
9864164 1872 3000 active+undersized+degraded+remapped+backfilling
3h 79875'182881827 79875:402437400 [2,0,1]p2 [2,1]p2
2024-06-12T01:42:35.239155+0000 2024-06-12T01:42:35.239155+0000
1370 periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000
10.3a 5215648 4552651 0 0 0 6194465524
9713558 1898 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181458295 79875:394916652 [2,1,0]p2 [2,1]p2
2024-06-11T21:19:29.020889+0000 2024-06-11T21:19:29.020889+0000
1227 periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000
10.3b 5218220 5218220 0 0 8388608 5951035110
9490698 1878 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180815984 79875:342438023 [2,1,0]p2 [2,1]p2
2024-06-03T05:03:19.483049+0000 2024-05-28T18:50:55.369009+0000
1004 queued for scrub
10.3c 5221615 5221615 0 0 12582912 6260894058
9727358 1938 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181561627 79875:342153262 [1,2,0]p1 [1,2]p1
2024-06-11T19:12:06.206008+0000 2024-06-11T19:12:06.206008+0000
1471 periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000
10.3d 5213771 5213771 0 0 0 6215975311
9699615 1946 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179354493 79875:399858588 [1,2,0]p1 [1,2]p1
2024-06-11T04:05:10.877703+0000 2024-06-11T04:05:10.877703+0000
8123 periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000
10.3e 5219446 4659577 0 0 4194304 5852588566
9370152 1933 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180035882 79875:391595513 [0,1,2]p0 [1,2]p1
2024-06-11T20:35:43.624903+0000 2024-05-22T21:40:55.606650+0000
520 periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000
10.3f 5219729 4889443 0 0 16777216 5925457842
9484458 1948 3000 active+undersized+degraded+remapped+backfilling
2h 79875'181330738 79875:409113809 [0,2,1]p0 [1,2]p1
2024-06-12T05:12:01.852109+0000 2024-05-27T20:03:22.704122+0000
533 periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000
10.40 5222241 4445164 0 0 0 6312545191
9773697 1873 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181341418 79875:393186178 [2,1,0]p2 [2,1]p2
2024-06-10T20:00:37.561045+0000 2024-06-10T20:00:37.561045+0000
7712 periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000
10.41 5215195 5183508 0 0 4194326 6000770355
9529537 1938 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181494334 79875:343712275 [0,2,1]p0 [1,2]p1
2024-06-12T04:18:58.693597+0000 2024-06-01T11:05:59.370690+0000
539 periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000
10.42 5219564 4681323 0 0 12582912 5822745109
9331625 1916 3000 active+undersized+degraded+remapped+backfilling
3h 79875'179811520 79875:340958258 [1,2,0]p1 [1,2]p1
2024-06-02T19:58:06.439770+0000 2024-05-21T18:28:28.794738+0000
11044 queued for deep scrub
10.43 5219412 4599458 0 0 594 5737440613
9239869 1950 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180634536 79875:388879991 [1,0,2]p1 [1,2]p1
2024-06-01T18:13:17.068416+0000 2024-05-20T16:32:20.302230+0000
1002 queued for deep scrub
10.44 5216008 5216008 0 0 4194304 6270195298
9786806 1944 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182088681 79875:406175480 [1,0,2]p1 [1,2]p1
2024-06-12T00:19:01.240405+0000 2024-06-12T00:19:01.240405+0000
1593 periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000
10.45 5216238 4694313 0 0 0 5714807184
9223666 1944 3000 active+undersized+degraded+remapped+backfilling
3h 79875'179709878 79875:338307727 [1,0,2]p1 [1,2]p1
2024-06-02T00:39:50.657116+0000 2024-05-21T02:52:14.376959+0000
958 queued for deep scrub
10.46 5217950 5217950 0 0 9729074 5945415128
9499020 1915 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'183726716 79875:395024265 [1,2,0]p1 [1,2]p1
2024-06-11T14:56:28.875949+0000 2024-05-29T04:19:53.409918+0000
724 periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000
10.47 5219125 5219125 0 0 4194304 5775128028
9299099 1864 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181328922 79875:340510799 [1,0,2]p1 [1,2]p1
2024-06-11T17:58:16.753327+0000 2024-05-23T14:24:25.876796+0000
652 periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000
10.48 5216471 5185269 0 0 4194326 6352053972
9843580 1929 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179547710 79875:378125137 [0,1,2]p0 [1,2]p1
2024-06-11T23:22:17.435614+0000 2024-06-11T23:22:17.435614+0000
1379 periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000
10.49 5217119 4580870 0 0 4194304 5971558944
9526047 1879 3000 active+undersized+degraded+remapped+backfilling
3h 79875'182489297 79875:389298429 [2,0,1]p2 [2,1]p2
2024-06-11T08:57:42.791265+0000 2024-05-29T07:13:57.867295+0000
959 periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000
10.4a 5217577 4909174 0 0 4194304 5960086599
9511657 1910 3000 active+undersized+degraded+remapped+backfilling
2h 79875'180982438 79875:381432369 [0,1,2]p0 [1,2]p1
2024-06-12T02:31:42.576663+0000 2024-06-01T09:08:07.860775+0000
480 periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000
10.4b 5220270 5220270 0 0 159808 6313986186
9763672 1934 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181279214 79875:342509944 [2,1,0]p2 [2,1]p2
2024-06-11T11:03:38.997373+0000 2024-06-11T11:03:38.997373+0000
3970 periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000
10.4c 5217864 5217864 0 0 4194304 6104537460
9679170 1948 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181081348 79875:341937760 [1,0,2]p1 [1,2]p1
2024-06-12T06:09:49.667680+0000 2024-06-01T13:06:12.938513+0000
520 periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000
10.4d 5217604 4905278 0 0 8388608 6250536027
9765571 1917 3000 active+undersized+degraded+remapped+backfilling
2h 79875'179562016 79875:387768172 [0,1,2]p0 [1,2]p1
2024-06-11T13:38:28.479053+0000 2024-06-11T13:38:28.479053+0000
2004 periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000
10.4e 5220202 5220202 0 0 12215022 5781813383
9291138 1940 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180360470 79875:377712114 [1,0,2]p1 [1,2]p1
2024-06-01T22:06:12.347760+0000 2024-05-25T19:34:49.805104+0000
982 queued for scrub
10.4f 5215243 4561466 0 0 0 6099447266
9605810 1930 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181794065 79875:396660094 [2,1,0]p2 [2,1]p2
2024-06-02T13:35:21.451128+0000 2024-06-02T13:35:21.451128+0000
7316 queued for scrub
10.50 5215219 5184003 0 0 0 6151789627
9674779 1871 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181695151 79875:405060298 [0,2,1]p0 [1,2]p1
2024-06-11T16:42:00.221013+0000 2024-06-11T16:42:00.221013+0000
2385 periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000
10.51 5218666 5218666 0 0 8388608 5774655330
9331373 1920 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180711664 79875:342978462 [0,1,2]p0 [1,2]p1
2024-06-11T14:31:51.402844+0000 2024-05-24T11:26:15.539677+0000
730 periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000
10.52 5218435 5187050 0 0 4194304 6375954134
9843553 1935 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180834874 79875:341502004 [0,1,2]p0 [1,2]p1
2024-06-11T08:41:43.586833+0000 2024-06-11T08:41:43.586833+0000
8368 periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000
10.53 5216822 5216822 0 0 10001995 5786896551
9368323 1928 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181337168 79875:402047940 [0,2,1]p0 [1,2]p1
2024-06-12T04:54:21.147742+0000 2024-05-23T23:02:36.154347+0000
557 periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000
10.54 5213949 4567626 0 0 8388608 6095478377
9664814 1944 3000 active+undersized+degraded+remapped+backfilling
3h 79875'182179986 79875:421118572 [2,1,0]p2 [2,1]p2
2024-06-12T09:19:30.470319+0000 2024-06-01T17:56:32.089348+0000
551 periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000
10.55 5217389 5217389 0 0 4194948 5905845521
9403344 1950 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'183481821 79875:341223864 [1,2,0]p1 [1,2]p1
2024-06-11T15:31:05.524698+0000 2024-05-29T11:50:10.322128+0000
584 periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000
10.56 5219142 5219142 0 0 0 6311549101
9773102 1897 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'184062267 79875:408792597 [1,2,0]p1 [1,2]p1
2024-06-10T22:17:28.523159+0000 2024-06-10T22:17:28.523159+0000
8211 periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000
10.57 5219035 4894148 0 0 180 5862209193
9396761 1895 3000 active+undersized+degraded+remapped+backfilling
2h 79875'181886155 79875:340148291 [0,1,2]p0 [1,2]p1
2024-06-10T22:33:19.364284+0000 2024-05-28T12:05:26.762682+0000
951 periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000
10.58 5218318 5185905 0 0 9878587 5911568798
9459707 1875 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179109213 79875:390200207 [2,0,1]p2 [2,1]p2
2024-06-11T13:08:37.380203+0000 2024-05-30T10:19:35.091370+0000
578 periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000
10.59 5218315 5185838 0 0 4194304 5727052522
9214424 1875 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181700141 79875:400297310 [2,0,1]p2 [2,1]p2
2024-06-11T11:15:00.221765+0000 2024-05-21T11:28:39.056382+0000
682 periodic deep scrub scheduled @ 2024-06-16T13:00:43.750185+0000
10.5a 5218301 5218301 0 0 4194304 5833097049
9365354 1942 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180699909 79875:393344045 [1,2,0]p1 [1,2]p1
2024-06-12T01:50:51.954645+0000 2024-05-25T00:11:49.917556+0000
497 periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000
10.5b 5215134 5215134 0 0 4194802 6070043467
9585134 1892 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181149500 79875:342539489 [2,0,1]p2 [2,1]p2
2024-06-12T08:19:32.265112+0000 2024-06-02T11:33:23.094569+0000
714 periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000
10.5c 5221503 5221503 0 0 4194304 5989312430
9524556 1939 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180748682 79875:340750417 [1,2,0]p1 [1,2]p1
2024-06-11T09:37:59.483197+0000 2024-05-28T03:41:58.475567+0000
1021 periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000
10.5d 5222202 5222202 0 0 8388608 6031557641
9520790 1928 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'179508607 79875:399919359 [1,0,2]p1 [1,2]p1
2024-06-11T17:10:33.165278+0000 2024-05-30T05:53:10.772736+0000
575 periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000
10.5e 5212770 4506643 0 0 0 5830165136
9388228 1938 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180002484 79875:391844220 [2,0,1]p2 [2,1]p2
2024-06-11T22:32:49.026280+0000 2024-05-24T15:31:50.362016+0000
523 periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000
10.5f 5221504 5221504 0 0 4194394 5807975316
9387383 1879 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181161493 79875:408530090 [0,2,1]p0 [1,2]p1
2024-06-11T15:10:33.261745+0000 2024-05-25T06:37:01.629711+0000
845 periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000
10.60 5217784 5186615 0 0 4194304 6208553838
9749044 1962 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181569072 79875:392211608 [0,1,2]p0 [1,2]p1
2024-06-11T21:44:35.372861+0000 2024-06-11T21:44:35.372861+0000
1505 periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000
10.61 5221292 4586636 0 0 8388608 5766320671
9280184 1910 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180945714 79875:344272511 [0,1,2]p0 [1,2]p1
2024-06-11T16:51:25.519891+0000 2024-05-24T17:37:21.814566+0000
566 periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000
10.62 5218498 4699002 0 0 4194466 6027128219
9555111 1884 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180387699 79875:341929273 [1,2,0]p1 [1,2]p1
2024-06-12T03:10:08.547987+0000 2024-06-01T02:47:43.109035+0000
509 periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000
10.63 5219101 5219101 0 0 0 6040382110
9500040 1885 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181623402 79875:389756489 [0,1,2]p0 [1,2]p1
2024-06-02T23:50:14.947543+0000 2024-06-02T23:50:14.947543+0000
13925 queued for scrub
10.64 5217025 5217025 0 0 2277792 6222337345
9701950 1866 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181853168 79875:406174284 [1,2,0]p1 [1,2]p1
2024-06-02T04:57:41.872723+0000 2024-06-02T04:57:41.872723+0000
6394 queued for scrub
10.65 5217908 4590550 0 0 4194326 6223151288
9709345 1916 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181196382 79875:340107600 [2,0,1]p2 [2,1]p2
2024-06-11T15:53:28.725001+0000 2024-06-11T15:53:28.725001+0000
1343 periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000
10.66 5218911 4893654 0 0 4194304 5815703055
9338987 1939 3000 active+undersized+degraded+remapped+backfilling
2h 79875'183062531 79875:395193073 [1,0,2]p1 [1,2]p1
2024-06-12T01:10:56.919233+0000 2024-05-25T04:50:04.300293+0000
590 periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000
10.67 5217539 4587208 0 0 4194304 6095526883
9640323 1955 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181875315 79875:342878178 [2,1,0]p2 [2,1]p2
2024-06-12T07:58:41.522730+0000 2024-06-01T07:20:59.951966+0000
553 periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000
10.68 5215852 4505902 0 0 8388608 6093858586
9596421 1926 3000 active+undersized+degraded+remapped+backfilling
3h 79875'179490643 79875:379632203 [0,1,2]p0 [1,2]p1
2024-06-11T16:02:14.995195+0000 2024-05-30T22:02:42.335910+0000
526 periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000
10.69 5216596 5216596 0 0 4194304 5833268044
9400414 1899 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181876551 79875:387059449 [1,2,0]p1 [1,2]p1
2024-06-11T17:00:57.111851+0000 2024-05-23T07:05:05.989427+0000
572 periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000
10.6a 5214106 4497370 0 0 0 5745504618
9296938 1869 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180109669 79875:381871149 [2,1,0]p2 [2,1]p2
2024-06-12T00:27:45.155477+0000 2024-05-25T09:22:19.570011+0000
494 periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000
10.6b 5219339 5219339 0 0 0 5801390847
9370084 1957 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180041524 79875:342270608 [0,1,2]p0 [1,2]p1
2024-06-12T03:01:39.358464+0000 2024-05-23T01:04:22.522035+0000
488 periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000
10.6c 5217917 5217917 0 0 554 6100979305
9612250 1955 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180968918 79875:341488318 [1,2,0]p1 [1,2]p1
2024-06-11T18:47:35.969345+0000 2024-05-31T11:54:59.559316+0000
588 periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000
10.6d 5221048 4550388 0 0 4194304 6024067862
9621947 1927 3000 active+undersized+degraded+remapped+backfilling
3h 79875'179373269 79875:388095690 [2,1,0]p2 [2,1]p2
2024-06-12T03:26:37.931310+0000 2024-05-31T04:10:41.991457+0000
502 periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000
10.6e 5215550 4888179 0 0 8388608 6110450001
9666892 1906 3000 active+undersized+degraded+remapped+backfilling
2h 79875'180885974 79875:380260263 [0,1,2]p0 [1,2]p1
2024-06-11T20:18:23.657558+0000 2024-06-11T20:18:23.657558+0000
1218 periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000
10.6f 5215850 4566820 0 0 0 6107146788
9634911 1923 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181752773 79875:396708021 [2,1,0]p2 [2,1]p2
2024-06-10T17:32:51.735004+0000 2024-06-10T17:32:51.735004+0000
10119 periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000
10.70 5218211 4672319 0 0 4194304 5793441438
9371563 1881 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180405012 79875:404578822 [1,2,0]p1 [1,2]p1
2024-06-03T04:46:35.130293+0000 2024-05-22T07:50:20.898441+0000
969 queued for deep scrub
10.71 5218299 4599312 0 0 4194304 5807950891
9349720 1906 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180690682 79875:344108361 [0,2,1]p0 [1,2]p1
2024-06-12T03:18:16.645927+0000 2024-05-23T09:32:56.055703+0000
487 periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000
10.72 5223470 4922347 0 0 8388608 6416468325
9922777 1958 3000 active+undersized+degraded+remapped+backfilling
2h 79875'180797342 79875:342356916 [0,1,2]p0 [1,2]p1
2024-06-12T05:42:47.545762+0000 2024-06-12T05:42:47.545762+0000
1307 periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000
10.73 5218301 4520651 0 0 0 6293832301
9738943 1953 3000 active+undersized+degraded+remapped+backfilling
3h 79875'182246986 79875:403792984 [2,0,1]p2 [2,1]p2
2024-06-11T20:59:02.079157+0000 2024-06-11T20:59:02.079157+0000
1399 periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000
10.74 5219061 4595112 0 0 0 5974775781
9469653 1943 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181947609 79875:420571221 [2,1,0]p2 [2,1]p2
2024-06-03T02:17:19.687482+0000 2024-05-28T08:43:38.825954+0000
927 queued for scrub
10.75 5220994 4596634 0 0 25165824 5822315216
9362548 1877 3000 active+undersized+degraded+remapped+backfilling
3h 79875'180376305 79875:338847231 [2,0,1]p2 [2,1]p2
2024-06-03T05:52:28.743255+0000 2024-05-24T05:11:00.211501+0000
981 queued for scrub
10.76 5222111 4892295 0 0 0 6067998823
9574254 1935 3000 active+undersized+degraded+remapped+backfilling
2h 79875'183871180 79875:409218100 [1,0,2]p1 [1,2]p1
2024-06-03T04:14:04.513004+0000 2024-06-03T04:14:04.513004+0000
6987 queued for scrub
10.77 5219020 4574683 0 0 0 5889074277
9460824 1896 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181614825 79875:342558162 [2,0,1]p2 [2,1]p2
2024-06-12T08:07:38.169523+0000 2024-05-27T15:15:35.587972+0000
536 periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000
10.78 5220442 4693235 0 0 8388608 5879566566
9395501 1923 3000 active+undersized+degraded+remapped+backfilling
3h 79875'179031456 79875:391462415 [1,0,2]p1 [1,2]p1
2024-06-02T09:32:42.834344+0000 2024-05-27T10:41:23.822124+0000
994 queued for scrub
10.79 5219734 4702117 0 0 4194304 6129933545
9646669 1926 3000 active+undersized+degraded+remapped+backfilling
3h 79875'182574416 79875:402249812 [1,2,0]p1 [1,2]p1
2024-06-12T05:52:02.959814+0000 2024-06-01T00:53:49.548055+0000
555 periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000
10.7a 5215303 4526846 0 0 0 6316171107
9817029 1932 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181765231 79875:395877057 [2,0,1]p2 [2,1]p2
2024-06-12T02:23:42.692096+0000 2024-06-12T02:23:42.692096+0000
1457 periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000
10.7b 5217213 4606816 0 0 4194304 6226859283
9718621 1952 3000 active+undersized+degraded+remapped+backfilling
3h 79875'181413541 79875:343968658 [1,2,0]p1 [1,2]p1
2024-06-11T23:52:28.414752+0000 2024-06-11T23:52:28.414752+0000
1308 periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000
10.7c 5215075 5215075 0 0 4194304 6013877208
9602345 1867 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'180991860 79875:341775615 [0,2,1]p0 [1,2]p1
2024-06-11T23:30:40.614412+0000 2024-05-31T14:10:30.542603+0000
503 periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000
10.7d 5220700 4569720 0 0 5837609 6258736025
9764836 1897 3000 active+undersized+degraded+remapped+backfilling
3h 79875'179741885 79875:401448508 [2,1,0]p2 [2,1]p2
2024-06-12T04:45:05.030905+0000 2024-06-12T04:45:05.030905+0000
1565 periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000
10.7e 5217113 5217113 0 0 16777216 6072081676
9598390 1902 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'181040361 79875:391032348 [1,2,0]p1 [1,2]p1
2024-06-11T17:37:26.589968+0000 2024-06-11T17:37:26.589968+0000
1613 periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000
10.7f 5218719 5218719 0 0 9909061 6130469539
9657824 1943 3000 active+undersized+degraded+remapped+backfill_wait
3h 79875'182283514 79875:408550888 [1,2,0]p1 [1,2]p1
2024-06-11T13:05:03.552526+0000 2024-06-11T13:05:03.552526+0000
1796 periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000
* NOTE: Omap statistics are gathered during deep scrub and may be
inaccurate soon afterwards depending on utilization. See
http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for
further details.
sudo ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS
0 ssd 3.63869 1.00000 3.6 TiB 82 GiB 50 MiB 40 GiB 42 GiB
3.6 TiB 2.21 0.03 0 up
1 ssd 3.63869 1.00000 3.6 TiB 2.0 TiB 16 GiB 1021 GiB 1.0 TiB
1.6 TiB 55.41 0.75 161 up
2 ssd 3.63869 1.00000 3.6 TiB 3.0 TiB 16 GiB 1.1 TiB 1.9 TiB
676 GiB 81.86 1.11 161 up
sudo ceph df detail
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.2 PiB 308 TiB 871 TiB 871 TiB 73.86
ssd 11 TiB 5.8 TiB 5.1 TiB 5.1 TiB 46.51
TOTAL 1.2 PiB 314 TiB 876 TiB 876 TiB 73.61
--- POOLS ---
POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA)
(OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR
UNDER COMPR
.mgr 1 1 827 MiB 827 MiB 0 B 208 1.6 GiB 1.6 GiB
0 B 0.11 734 GiB N/A 50 GiB N/A 0 B
0 B
metadata 10 128 1.0 TiB 647 MiB 1.0 TiB 667.90M 2.1 TiB 1.3 GiB
2.1 TiB 59.45 719 GiB N/A 3.5 TiB N/A 0 B
0 B
data 11 2048 288 TiB 288 TiB 0 B 566.97M 863 TiB 863 TiB
0 B 81.10 67 TiB N/A N/A N/A 3.4 TiB
6.8 TiB
ssd-data 18 32 14 GiB 14 GiB 0 B 265.17k 29 GiB 29 GiB
0 B 1.90 734 GiB N/A N/A N/A 0 B
0 B
sudo ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:00.000170",
"last_optimize_started": "Wed Jun 12 13:14:49 2024",
"mode": "upmap",
"no_optimization_needed": true,
"optimize_result": "Some objects (0.172816) are degraded; try again
later",
"plans": []
}
[image: ariadne.ai Logo] Lars Köppel
Developer
Email: lars.koeppel@xxxxxxxxxx
Phone: +49 6221 5993580 <+4962215993580>
ariadne.ai (Germany) GmbH
Häusserstraße 3, 69115 Heidelberg
Amtsgericht Mannheim, HRB 744040
Geschäftsführer: Dr. Fabian Svara
https://ariadne.ai
On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri <anthony.datri@xxxxxxxxx>
wrote:
If you have:
* pg_num too low (defaults are too low)
* pg_num not a power of 2
* pg_num != number of OSDs in the pool
* balancer not enabled
any of those might result in imbalance.
> On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote:
>
> I don't have any good explanation at this point. Can you share some more
information like:
>
> ceph pg ls-by-pool <cephfs_metadata>
> ceph osd df (for the relevant OSDs)
> ceph df
>
> Thanks,
> Eugen
>
> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>
>> Since my last update the size of the largest OSD increased by 0.4 TiB
while
>> the smallest one only increased by 0.1 TiB. How is this possible?
>>
>> Because the metadata pool reported to have only 900MB space left, I
stopped
>> the hot-standby MDS. This gave me 8GB back but these filled up in the
last
>> 2h.
>> I think I have to zap the next OSD because the filesystem is getting
read
>> only...
>>
>> How is it possible that an OSD has over 1 TiB less data on it after a
>> rebuild? And how is it possible to have so different sizes of OSDs?
>>
>>
>> [image: ariadne.ai Logo] Lars Köppel
>> Developer
>> Email: lars.koeppel@xxxxxxxxxx
>> Phone: +49 6221 5993580 <+4962215993580>
>> ariadne.ai (Germany) GmbH
>> Häusserstraße 3, 69115 Heidelberg
>> Amtsgericht Mannheim, HRB 744040
>> Geschäftsführer: Dr. Fabian Svara
>> https://ariadne.ai
>>
>>
>> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel <lars.koeppel@xxxxxxxxxx>
wrote:
>>
>>> Only in warning mode. And there were no PG splits or merges in the
last 2
>>> month.
>>>
>>>
>>> [image: ariadne.ai Logo] Lars Köppel
>>> Developer
>>> Email: lars.koeppel@xxxxxxxxxx
>>> Phone: +49 6221 5993580 <+4962215993580>
>>> ariadne.ai (Germany) GmbH
>>> Häusserstraße 3, 69115 Heidelberg
>>> Amtsgericht Mannheim, HRB 744040
>>> Geschäftsführer: Dr. Fabian Svara
>>> https://ariadne.ai
>>>
>>>
>>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> wrote:
>>>
>>>> I don't think scrubs can cause this. Do you have autoscaler enabled?
>>>>
>>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>>>>
>>>> > Hi,
>>>> >
>>>> > thank you for your response.
>>>> >
>>>> > I don't think this thread covers my problem, because the OSDs for
the
>>>> > metadata pool fill up at different rates. So I would think this is
no
>>>> > direct problem with the journal.
>>>> > Because we had earlier problems with the journal I changed some
>>>> > settings(see below). I already restarted all MDS multiple times but
no
>>>> > change here.
>>>> >
>>>> > The health warnings regarding cache pressure resolve normally after
a
>>>> > short period of time, when the heavy load on the client ends.
Sometimes
>>>> it
>>>> > stays a bit longer because an rsync is running and copying data on
the
>>>> > cluster(rsync is not good at releasing the caps).
>>>> >
>>>> > Could it be a problem if scrubs run most of the time in the
background?
>>>> Can
>>>> > this block any other tasks or generate new data itself?
>>>> >
>>>> > Best regards,
>>>> > Lars
>>>> >
>>>> >
>>>> > global basic mds_cache_memory_limit
>>>> > 17179869184
>>>> > global advanced mds_max_caps_per_client
>>>> > 16384
>>>> > global advanced
>>>> mds_recall_global_max_decay_threshold
>>>> > 262144
>>>> > global advanced mds_recall_max_decay_rate
>>>> > 1.000000
>>>> > global advanced mds_recall_max_decay_threshold
>>>> > 262144
>>>> > mds advanced mds_cache_trim_threshold
>>>> > 131072
>>>> > mds advanced mds_heartbeat_grace
>>>> > 120.000000
>>>> > mds advanced mds_heartbeat_reset_grace
>>>> > 7400
>>>> > mds advanced mds_tick_interval
>>>> > 3.000000
>>>> >
>>>> >
>>>> > [image: ariadne.ai Logo] Lars Köppel
>>>> > Developer
>>>> > Email: lars.koeppel@xxxxxxxxxx
>>>> > Phone: +49 6221 5993580 <+4962215993580>
>>>> > ariadne.ai (Germany) GmbH
>>>> > Häusserstraße 3, 69115 Heidelberg
>>>> > Amtsgericht Mannheim, HRB 744040
>>>> > Geschäftsführer: Dr. Fabian Svara
>>>> > https://ariadne.ai
>>>> >
>>>> >
>>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx> wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> can you check if this thread [1] applies to your situation? You
don't
>>>> >> have multi-active MDS enabled, but maybe it's still some journal
>>>> >> trimming, or maybe misbehaving clients? In your first post there
were
>>>> >> health warnings regarding cache pressure and cache size. Are those
>>>> >> resolved?
>>>> >>
>>>> >> [1]
>>>> >>
>>>> >>
>>>>
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE
>>>> >>
>>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>>>> >>
>>>> >> > Hello everyone,
>>>> >> >
>>>> >> > short update to this problem.
>>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected
size
>>>> >> ~50%).
>>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They
jumped up
>>>> and
>>>> >> > down a lot but the higher one has now also reached 'nearfull'
>>>> status. How
>>>> >> > is this possible? What is going on?
>>>> >> >
>>>> >> > Does anyone have a solution how to fix this without zapping the
OSD?
>>>> >> >
>>>> >> > Best regards,
>>>> >> > Lars
>>>> >> >
>>>> >> >
>>>> >> > [image: ariadne.ai Logo] Lars Köppel
>>>> >> > Developer
>>>> >> > Email: lars.koeppel@xxxxxxxxxx
>>>> >> > Phone: +49 6221 5993580 <+4962215993580>
>>>> >> > ariadne.ai (Germany) GmbH
>>>> >> > Häusserstraße 3, 69115 Heidelberg
>>>> >> > Amtsgericht Mannheim, HRB 744040
>>>> >> > Geschäftsführer: Dr. Fabian Svara
>>>> >> > https://ariadne.ai
>>>> >> > _______________________________________________
>>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
>>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
>>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>> >>
>>>>
>>>>
>>>>
>>>>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
