Which version did you upgrade from to 18.2.2?
I can’t pin it down to a specific issue, but somewhere in the back of
my mind is something related to a new omap format or something. But
I’m really not sure at all.
Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
> I am happy to help you with as much information as possible. I probably
> just don't know where to look for it.
> Below are the requested information. The cluster is rebuilding the
> zapped OSD at the moment. This will probably take the next few days.
>
>
> sudo ceph pg ls-by-pool metadata
> PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES*
> OMAP_KEYS* LOG LOG_DUPS STATE
> SINCE VERSION REPORTED UP ACTING
> SCRUB_STAMP DEEP_SCRUB_STAMP
> LAST_SCRUB_DURATION SCRUB_SCHEDULING
> 10.0 5217325 4994695 0 0 4194304 5880891340
> 9393865 1885 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'180849582 79875:391519635 [0,1,2]p0 [1,2]p1
> 2024-06-11T09:08:09.829362+0000 2024-05-28T05:52:59.321589+0000
> 627 periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000
> 10.1 5214785 5193424 0 0 0 5843682713
> 9410150 1912 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180914288 79875:342746928 [2,1,0]p2 [2,1]p2
> 2024-06-01T15:56:28.927288+0000 2024-05-27T03:31:37.682966+0000
> 966 queued for scrub
> 10.2 5218432 5187168 0 0 0 6402011266
> 9812513 1874 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180970531 79875:341340204 [0,1,2]p0 [1,2]p1
> 2024-06-11T13:40:58.994256+0000 2024-06-11T13:40:58.994256+0000
> 1942 periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000
> 10.3 5217413 5217413 0 0 8388788 5766005023
> 9271787 1923 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181012233 79875:388295881 [1,0,2]p1 [1,2]p1
> 2024-06-12T00:35:56.965547+0000 2024-05-23T19:54:56.121729+0000
> 492 periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000
> 10.4 5220069 5220069 0 0 12583466 6027548724
> 9537290 1959 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181576075 79875:405295868 [1,2,0]p1 [1,2]p1
> 2024-06-11T17:47:22.923514+0000 2024-05-31T02:06:55.339574+0000
> 581 periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000
> 10.5 5216162 5211999 0 0 4194304 5941347251
> 9542764 1930 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180455793 79875:338418517 [2,1,0]p2 [2,1]p2
> 2024-06-11T22:50:16.170708+0000 2024-05-30T23:49:54.316379+0000
> 528 periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000
> 10.6 5216100 4980459 0 0 4521984 6428088514
> 9850762 1911 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'184045876 79875:396809795 [0,2,1]p0 [1,2]p1
> 2024-06-11T22:24:05.102716+0000 2024-06-11T22:24:05.102716+0000
> 1082 periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000
> 10.7 5218232 5218232 0 0 4194304 6377065363
> 9849360 1919 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182672562 79875:342449062 [1,0,2]p1 [1,2]p1
> 2024-06-11T06:22:15.689422+0000 2024-06-11T06:22:15.689422+0000
> 8225 periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000
> 10.8 5219620 5182816 0 0 0 6167304290
> 9691796 1896 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179628377 79875:378022884 [2,1,0]p2 [2,1]p2
> 2024-06-11T22:06:01.386763+0000 2024-06-11T22:06:01.386763+0000
> 1286 periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000
> 10.9 5219448 5164591 0 0 8388698 5796048346
> 9338312 1868 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181739392 79875:387412389 [2,1,0]p2 [2,1]p2
> 2024-06-12T05:21:00.586747+0000 2024-05-26T11:10:59.780673+0000
> 539 periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000
> 10.a 5219861 5163635 0 0 12582912 5841839055
> 9387200 1916 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180205688 79875:379381294 [1,2,0]p1 [1,2]p1
> 2024-06-11T12:35:05.571200+0000 2024-05-22T11:07:16.041773+0000
> 1093 periodic deep scrub scheduled @
2024-06-17T05:21:40.136463+0000
> 10.b 5217949 5217949 0 0 16777216 5935863260
> 9462127 1881 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181655745 79875:343806807 [0,1,2]p0 [1,2]p1
> 2024-06-11T22:41:28.976920+0000 2024-05-26T08:43:29.217457+0000
> 520 periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000
> 10.c 5221697 5217118 0 0 4194304 6015217841
> 9574445 1928 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180892826 79875:341490398 [2,1,0]p2 [2,1]p2
> 2024-06-11T09:20:58.443473+0000 2024-05-30T00:13:50.306507+0000
> 768 periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000
> 10.d 5217727 4908764 0 0 0 5825598519
> 9349877 1930 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'179050347 79875:387455993 [0,1,2]p0 [1,2]p1
> 2024-06-02T07:27:27.873631+0000 2024-05-20T03:58:30.225170+0000
> 952 queued for deep scrub
> 10.e 5215040 5210572 0 0 4194304 5790634469
> 9327651 1905 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180049044 79875:377196119 [2,1,0]p2 [2,1]p2
> 2024-06-12T05:03:08.521423+0000 2024-05-22T13:47:46.753131+0000
> 527 periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000
> 10.f 5217202 4885952 0 0 8388630 6005274626
> 9589178 1917 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'181953069 79875:396402453 [0,2,1]p0 [1,2]p1
> 2024-06-11T12:20:49.661741+0000 2024-06-11T12:20:49.661741+0000
> 3949 periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000
> 10.10 5214985 5157391 0 0 4194394 6275024632
> 9721505 1922 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181801708 79875:395055895 [2,0,1]p2 [2,1]p2
> 2024-06-11T01:49:47.831191+0000 2024-06-11T01:49:47.831191+0000
> 9579 periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000
> 10.11 5219236 4983134 0 0 4194304 5787580073
> 9316893 1901 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'181011816 79875:343488767 [1,0,2]p1 [1,2]p1
> 2024-06-03T00:06:14.862209+0000 2024-05-27T00:05:56.468361+0000
> 960 queued for scrub
> 10.12 5214103 5182770 0 0 90 5970750188
> 9505613 1891 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180210613 79875:341042616 [0,1,2]p0 [1,2]p1
> 2024-06-12T06:39:48.252419+0000 2024-05-28T01:32:03.381577+0000
> 527 periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000
> 10.13 5214580 5214580 0 0 8388642 5793432968
> 9335940 1957 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182394614 79875:393423214 [1,2,0]p1 [1,2]p1
> 2024-06-10T17:52:04.690203+0000 2024-05-22T03:39:02.453336+0000
> 1151 periodic deep scrub scheduled @
2024-06-16T00:28:37.682765+0000
> 10.14 5218591 5218591 0 0 5053325 6046268958
> 9582481 1915 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'184576050 79875:425213800 [2,0,1]p2 [2,1]p2
> 2024-06-03T05:36:07.656965+0000 2024-05-28T22:46:16.814318+0000
> 985 queued for scrub
> 10.15 5215919 4907184 0 0 11939335 5752284246
> 9285870 1889 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'179713036 79875:337938973 [0,2,1]p0 [1,2]p1
> 2024-06-11T14:06:50.138992+0000 2024-05-24T07:52:22.522455+0000
> 786 periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000
> 10.16 5221091 5221091 0 0 8388608 5966308565
> 9487816 1916 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'183445984 79875:399413009 [0,2,1]p0 [1,2]p1
> 2024-06-03T02:01:53.649417+0000 2024-06-03T02:01:53.649417+0000
> 6935 queued for scrub
> 10.17 5218069 5194732 0 0 4194304 6231638001
> 9735915 1890 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182390128 79875:340719915 [2,0,1]p2 [2,1]p2
> 2024-06-12T06:31:01.523916+0000 2024-06-12T06:31:01.523916+0000
> 1272 periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000
> 10.18 5214253 5159897 0 0 0 6128506469
> 9673261 1886 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179754208 79875:389150273 [2,1,0]p2 [2,1]p2
> 2024-06-12T02:53:31.027154+0000 2024-06-12T02:53:31.027154+0000
> 1308 periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000
> 10.19 5220131 5183342 0 0 0 5869775360
> 9460583 1953 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181750923 79875:393162989 [2,0,1]p2 [2,1]p2
> 2024-06-11T19:21:24.106504+0000 2024-05-22T15:42:01.721091+0000
> 522 periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000
> 10.1a 5217968 5186700 0 0 4194304 6183652625
> 9733847 1896 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181056240 79875:394241128 [0,2,1]p0 [1,2]p1
> 2024-06-02T15:37:49.730965+0000 2024-06-02T15:37:49.730965+0000
> 7347 queued for scrub
> 10.1b 5221259 5161879 0 0 0 5989030303
> 9561035 1875 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181052305 79875:343144823 [0,1,2]p0 [1,2]p1
> 2024-06-12T06:01:07.213894+0000 2024-06-02T00:23:51.063901+0000
> 543 periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000
> 10.1c 5217034 5217034 0 0 554 5979861913
> 9504442 1875 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180913313 79875:341328275 [1,2,0]p1 [1,2]p1
> 2024-06-01T20:02:47.025129+0000 2024-06-01T20:02:47.025129+0000
> 6568 queued for scrub
> 10.1d 5219691 4997027 0 0 12582912 6081321658
> 9599164 1918 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'179638837 79875:391808225 [0,2,1]p0 [1,2]p1
> 2024-06-11T20:27:02.506715+0000 2024-05-31T22:58:06.578167+0000
> 517 periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000
> 10.1e 5213369 5208776 0 0 4194304 6295629499
> 9765317 1941 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181145210 79875:391326487 [2,1,0]p2 [2,1]p2
> 2024-06-12T07:49:28.432084+0000 2024-06-12T07:49:28.432084+0000
> 1990 periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000
> 10.1f 5218399 5000094 0 0 8388608 5821063844
> 9421222 1896 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'180960296 79875:400635456 [0,2,1]p0 [1,2]p1
> 2024-06-11T13:53:40.278050+0000 2024-05-22T19:39:02.345663+0000
> 761 periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000
> 10.20 5216437 5185379 0 0 0 6280994122
> 9719574 1945 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181610117 79875:391874431 [0,1,2]p0 [1,2]p1
> 2024-06-11T12:58:57.754648+0000 2024-06-11T12:58:57.754648+0000
> 2288 periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000
> 10.21 5217349 5217349 0 0 8388698 6348931429
> 9822197 1867 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181724357 79875:343883762 [0,1,2]p0 [1,2]p1
> 2024-06-11T18:28:34.044546+0000 2024-06-11T18:28:34.044546+0000
> 1817 periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000
> 10.22 5215544 4943475 0 0 4194304 5982815232
> 9482159 1899 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'180230331 79875:340839853 [2,0,1]p2 [2,1]p2
> 2024-06-11T15:21:21.994674+0000 2024-05-30T08:01:03.301650+0000
> 649 periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000
> 10.23 5217758 5217758 0 0 146 5804086623
> 9336298 1920 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180948432 79875:388158699 [1,2,0]p1 [1,2]p1
> 2024-06-03T05:19:32.311846+0000 2024-05-27T08:13:15.250058+0000
> 971 queued for scrub
> 10.24 5221505 5221505 0 0 0 6360995644
> 9816333 1925 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182198758 79875:406847098 [0,2,1]p0 [1,2]p1
> 2024-06-12T04:00:44.315682+0000 2024-06-12T04:00:44.315682+0000
> 2046 periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000
> 10.25 5214911 5214911 0 0 12583002 5726320657
> 9264352 1910 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180132024 79875:338167995 [1,2,0]p1 [1,2]p1
> 2024-06-11T14:43:47.076906+0000 2024-05-23T12:16:34.271856+0000
> 715 periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000
> 10.26 5217588 4973667 0 0 0 6040775867
> 9598406 1927 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'188144536 79875:400277134 [2,0,1]p2 [2,1]p2
> 2024-06-02T06:55:09.016528+0000 2024-06-02T06:55:09.016528+0000
> 7047 queued for scrub
> 10.27 5218490 5218490 0 0 8388630 6265942186
> 9818746 1941 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182400328 79875:342052414 [1,2,0]p1 [1,2]p1
> 2024-06-12T01:01:06.674323+0000 2024-06-12T01:01:06.674323+0000
> 1500 periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000
> 10.28 5221153 5221153 0 0 0 6049095371
> 9597182 1913 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179438777 79875:378265375 [1,2,0]p1 [1,2]p1
> 2024-06-12T04:09:58.896192+0000 2024-06-01T04:32:46.011974+0000
> 553 periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000
> 10.29 5214066 5214066 0 0 4194304 5767555623
> 9316899 1937 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181568997 79875:387683159 [0,2,1]p0 [1,2]p1
> 2024-06-10T22:48:57.757493+0000 2024-05-21T07:14:53.348341+0000
> 938 periodic deep scrub scheduled @
2024-06-16T06:24:55.108695+0000
> 10.2a 5216314 4977811 0 0 4194304 6012567810
> 9563804 1951 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'181140734 79875:381017643 [2,1,0]p2 [2,1]p2
> 2024-06-11T09:57:29.097708+0000 2024-05-30T03:47:22.798044+0000
> 1169 periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000
> 10.2b 5219620 5219620 0 0 8388608 5829917912
> 9412426 1889 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180465782 79875:342263856 [1,0,2]p1 [1,2]p1
> 2024-06-12T08:30:08.941782+0000 2024-05-24T22:00:29.535205+0000
> 636 periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000
> 10.2c 5214087 5214087 0 0 0 5874182922
> 9375708 1871 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180616041 79875:341244736 [1,2,0]p1 [1,2]p1
> 2024-06-10T23:10:08.184410+0000 2024-05-27T17:41:07.583255+0000
> 1268 periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000
> 10.2d 5213968 4984822 0 0 4337824 5835754026
> 9417033 1947 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'178824607 79875:386427308 [2,0,1]p2 [2,1]p2
> 2024-06-12T09:27:43.774331+0000 2024-05-27T05:33:45.293266+0000
> 494 periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000
> 10.2e 5216263 4974456 0 0 12582912 6025714323
> 9553880 1940 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'180639631 79875:378284872 [1,0,2]p1 [1,2]p1
> 2024-06-12T01:19:42.477063+0000 2024-05-31T08:42:41.293335+0000
> 526 periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000
> 10.2f 5220057 5220057 0 0 13839831 5804027574
> 9370862 1882 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180961915 79875:394513378 [1,0,2]p1 [1,2]p1
> 2024-06-11T19:58:05.145147+0000 2024-05-24T20:18:32.301831+0000
> 535 periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000
> 10.30 5217090 4955469 0 0 12582934 6438521431
> 9924825 1869 3000
active+undersized+degraded+remapped+backfill_wait
> 2h 79875'181488488 79875:404940886 [2,1,0]p2 [2,1]p2
> 2024-06-12T07:16:18.095439+0000 2024-06-12T07:16:18.095439+0000
> 2190 periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000
> 10.31 5218176 5218176 0 0 4194304 6215016361
> 9752698 1897 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181622879 79875:343554151 [1,2,0]p1 [1,2]p1
> 2024-06-12T09:00:15.941065+0000 2024-06-12T09:00:15.941065+0000
> 1806 periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000
> 10.32 5223171 4704732 0 0 8388608 6205669256
> 9800320 1920 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180911263 79875:342754128 [0,2,1]p0 [1,2]p1
> 2024-06-11T19:49:09.010583+0000 2024-06-11T19:49:09.010583+0000
> 1664 periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000
> 10.33 5215341 4525338 0 0 4194304 6066965588
> 9635403 1888 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181467497 79875:402958597 [1,2,0]p1 [1,2]p1
> 2024-06-12T01:59:23.933458+0000 2024-05-31T17:01:38.620377+0000
> 512 periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000
> 10.34 5217342 5217342 0 0 8388608 5999920961
> 9570527 1956 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182013407 79875:419219483 [1,0,2]p1 [1,2]p1
> 2024-06-12T09:10:17.390128+0000 2024-06-01T15:40:23.241769+0000
> 601 periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000
> 10.35 5222114 5222114 0 0 8388608 6021420075
> 9571300 1952 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180948342 79875:338598678 [1,2,0]p1 [1,2]p1
> 2024-06-11T22:59:17.017317+0000 2024-05-30T15:59:15.553266+0000
> 539 periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000
> 10.36 5223657 5223657 0 0 4331271 5844578100
> 9347405 1903 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'183363403 79875:408484683 [0,2,1]p0 [1,2]p1
> 2024-06-11T18:37:46.653097+0000 2024-05-25T14:13:51.648712+0000
> 553 periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000
> 10.37 5219248 5219248 0 0 4194304 6072826085
> 9579049 1936 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182091890 79875:341545087 [1,0,2]p1 [1,2]p1
> 2024-06-11T14:19:40.744743+0000 2024-05-29T18:40:12.602405+0000
> 771 periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000
> 10.38 5214022 5214022 0 0 4194858 5780805541
> 9299038 1951 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'178809936 79875:390113817 [1,0,2]p1 [1,2]p1
> 2024-06-02T00:55:22.046433+0000 2024-05-25T16:49:10.279003+0000
> 931 queued for scrub
> 10.39 5220692 4542094 0 0 0 6401799077
> 9864164 1872 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'182881827 79875:402437400 [2,0,1]p2 [2,1]p2
> 2024-06-12T01:42:35.239155+0000 2024-06-12T01:42:35.239155+0000
> 1370 periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000
> 10.3a 5215648 4552651 0 0 0 6194465524
> 9713558 1898 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181458295 79875:394916652 [2,1,0]p2 [2,1]p2
> 2024-06-11T21:19:29.020889+0000 2024-06-11T21:19:29.020889+0000
> 1227 periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000
> 10.3b 5218220 5218220 0 0 8388608 5951035110
> 9490698 1878 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180815984 79875:342438023 [2,1,0]p2 [2,1]p2
> 2024-06-03T05:03:19.483049+0000 2024-05-28T18:50:55.369009+0000
> 1004 queued for scrub
> 10.3c 5221615 5221615 0 0 12582912 6260894058
> 9727358 1938 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181561627 79875:342153262 [1,2,0]p1 [1,2]p1
> 2024-06-11T19:12:06.206008+0000 2024-06-11T19:12:06.206008+0000
> 1471 periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000
> 10.3d 5213771 5213771 0 0 0 6215975311
> 9699615 1946 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179354493 79875:399858588 [1,2,0]p1 [1,2]p1
> 2024-06-11T04:05:10.877703+0000 2024-06-11T04:05:10.877703+0000
> 8123 periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000
> 10.3e 5219446 4659577 0 0 4194304 5852588566
> 9370152 1933 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180035882 79875:391595513 [0,1,2]p0 [1,2]p1
> 2024-06-11T20:35:43.624903+0000 2024-05-22T21:40:55.606650+0000
> 520 periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000
> 10.3f 5219729 4889443 0 0 16777216 5925457842
> 9484458 1948 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'181330738 79875:409113809 [0,2,1]p0 [1,2]p1
> 2024-06-12T05:12:01.852109+0000 2024-05-27T20:03:22.704122+0000
> 533 periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000
> 10.40 5222241 4445164 0 0 0 6312545191
> 9773697 1873 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181341418 79875:393186178 [2,1,0]p2 [2,1]p2
> 2024-06-10T20:00:37.561045+0000 2024-06-10T20:00:37.561045+0000
> 7712 periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000
> 10.41 5215195 5183508 0 0 4194326 6000770355
> 9529537 1938 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181494334 79875:343712275 [0,2,1]p0 [1,2]p1
> 2024-06-12T04:18:58.693597+0000 2024-06-01T11:05:59.370690+0000
> 539 periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000
> 10.42 5219564 4681323 0 0 12582912 5822745109
> 9331625 1916 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'179811520 79875:340958258 [1,2,0]p1 [1,2]p1
> 2024-06-02T19:58:06.439770+0000 2024-05-21T18:28:28.794738+0000
> 11044 queued for deep scrub
> 10.43 5219412 4599458 0 0 594 5737440613
> 9239869 1950 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180634536 79875:388879991 [1,0,2]p1 [1,2]p1
> 2024-06-01T18:13:17.068416+0000 2024-05-20T16:32:20.302230+0000
> 1002 queued for deep scrub
> 10.44 5216008 5216008 0 0 4194304 6270195298
> 9786806 1944 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182088681 79875:406175480 [1,0,2]p1 [1,2]p1
> 2024-06-12T00:19:01.240405+0000 2024-06-12T00:19:01.240405+0000
> 1593 periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000
> 10.45 5216238 4694313 0 0 0 5714807184
> 9223666 1944 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'179709878 79875:338307727 [1,0,2]p1 [1,2]p1
> 2024-06-02T00:39:50.657116+0000 2024-05-21T02:52:14.376959+0000
> 958 queued for deep scrub
> 10.46 5217950 5217950 0 0 9729074 5945415128
> 9499020 1915 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'183726716 79875:395024265 [1,2,0]p1 [1,2]p1
> 2024-06-11T14:56:28.875949+0000 2024-05-29T04:19:53.409918+0000
> 724 periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000
> 10.47 5219125 5219125 0 0 4194304 5775128028
> 9299099 1864 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181328922 79875:340510799 [1,0,2]p1 [1,2]p1
> 2024-06-11T17:58:16.753327+0000 2024-05-23T14:24:25.876796+0000
> 652 periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000
> 10.48 5216471 5185269 0 0 4194326 6352053972
> 9843580 1929 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179547710 79875:378125137 [0,1,2]p0 [1,2]p1
> 2024-06-11T23:22:17.435614+0000 2024-06-11T23:22:17.435614+0000
> 1379 periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000
> 10.49 5217119 4580870 0 0 4194304 5971558944
> 9526047 1879 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'182489297 79875:389298429 [2,0,1]p2 [2,1]p2
> 2024-06-11T08:57:42.791265+0000 2024-05-29T07:13:57.867295+0000
> 959 periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000
> 10.4a 5217577 4909174 0 0 4194304 5960086599
> 9511657 1910 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'180982438 79875:381432369 [0,1,2]p0 [1,2]p1
> 2024-06-12T02:31:42.576663+0000 2024-06-01T09:08:07.860775+0000
> 480 periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000
> 10.4b 5220270 5220270 0 0 159808 6313986186
> 9763672 1934 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181279214 79875:342509944 [2,1,0]p2 [2,1]p2
> 2024-06-11T11:03:38.997373+0000 2024-06-11T11:03:38.997373+0000
> 3970 periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000
> 10.4c 5217864 5217864 0 0 4194304 6104537460
> 9679170 1948 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181081348 79875:341937760 [1,0,2]p1 [1,2]p1
> 2024-06-12T06:09:49.667680+0000 2024-06-01T13:06:12.938513+0000
> 520 periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000
> 10.4d 5217604 4905278 0 0 8388608 6250536027
> 9765571 1917 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'179562016 79875:387768172 [0,1,2]p0 [1,2]p1
> 2024-06-11T13:38:28.479053+0000 2024-06-11T13:38:28.479053+0000
> 2004 periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000
> 10.4e 5220202 5220202 0 0 12215022 5781813383
> 9291138 1940 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180360470 79875:377712114 [1,0,2]p1 [1,2]p1
> 2024-06-01T22:06:12.347760+0000 2024-05-25T19:34:49.805104+0000
> 982 queued for scrub
> 10.4f 5215243 4561466 0 0 0 6099447266
> 9605810 1930 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181794065 79875:396660094 [2,1,0]p2 [2,1]p2
> 2024-06-02T13:35:21.451128+0000 2024-06-02T13:35:21.451128+0000
> 7316 queued for scrub
> 10.50 5215219 5184003 0 0 0 6151789627
> 9674779 1871 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181695151 79875:405060298 [0,2,1]p0 [1,2]p1
> 2024-06-11T16:42:00.221013+0000 2024-06-11T16:42:00.221013+0000
> 2385 periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000
> 10.51 5218666 5218666 0 0 8388608 5774655330
> 9331373 1920 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180711664 79875:342978462 [0,1,2]p0 [1,2]p1
> 2024-06-11T14:31:51.402844+0000 2024-05-24T11:26:15.539677+0000
> 730 periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000
> 10.52 5218435 5187050 0 0 4194304 6375954134
> 9843553 1935 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180834874 79875:341502004 [0,1,2]p0 [1,2]p1
> 2024-06-11T08:41:43.586833+0000 2024-06-11T08:41:43.586833+0000
> 8368 periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000
> 10.53 5216822 5216822 0 0 10001995 5786896551
> 9368323 1928 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181337168 79875:402047940 [0,2,1]p0 [1,2]p1
> 2024-06-12T04:54:21.147742+0000 2024-05-23T23:02:36.154347+0000
> 557 periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000
> 10.54 5213949 4567626 0 0 8388608 6095478377
> 9664814 1944 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'182179986 79875:421118572 [2,1,0]p2 [2,1]p2
> 2024-06-12T09:19:30.470319+0000 2024-06-01T17:56:32.089348+0000
> 551 periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000
> 10.55 5217389 5217389 0 0 4194948 5905845521
> 9403344 1950 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'183481821 79875:341223864 [1,2,0]p1 [1,2]p1
> 2024-06-11T15:31:05.524698+0000 2024-05-29T11:50:10.322128+0000
> 584 periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000
> 10.56 5219142 5219142 0 0 0 6311549101
> 9773102 1897 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'184062267 79875:408792597 [1,2,0]p1 [1,2]p1
> 2024-06-10T22:17:28.523159+0000 2024-06-10T22:17:28.523159+0000
> 8211 periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000
> 10.57 5219035 4894148 0 0 180 5862209193
> 9396761 1895 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'181886155 79875:340148291 [0,1,2]p0 [1,2]p1
> 2024-06-10T22:33:19.364284+0000 2024-05-28T12:05:26.762682+0000
> 951 periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000
> 10.58 5218318 5185905 0 0 9878587 5911568798
> 9459707 1875 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179109213 79875:390200207 [2,0,1]p2 [2,1]p2
> 2024-06-11T13:08:37.380203+0000 2024-05-30T10:19:35.091370+0000
> 578 periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000
> 10.59 5218315 5185838 0 0 4194304 5727052522
> 9214424 1875 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181700141 79875:400297310 [2,0,1]p2 [2,1]p2
> 2024-06-11T11:15:00.221765+0000 2024-05-21T11:28:39.056382+0000
> 682 periodic deep scrub scheduled @
2024-06-16T13:00:43.750185+0000
> 10.5a 5218301 5218301 0 0 4194304 5833097049
> 9365354 1942 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180699909 79875:393344045 [1,2,0]p1 [1,2]p1
> 2024-06-12T01:50:51.954645+0000 2024-05-25T00:11:49.917556+0000
> 497 periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000
> 10.5b 5215134 5215134 0 0 4194802 6070043467
> 9585134 1892 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181149500 79875:342539489 [2,0,1]p2 [2,1]p2
> 2024-06-12T08:19:32.265112+0000 2024-06-02T11:33:23.094569+0000
> 714 periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000
> 10.5c 5221503 5221503 0 0 4194304 5989312430
> 9524556 1939 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180748682 79875:340750417 [1,2,0]p1 [1,2]p1
> 2024-06-11T09:37:59.483197+0000 2024-05-28T03:41:58.475567+0000
> 1021 periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000
> 10.5d 5222202 5222202 0 0 8388608 6031557641
> 9520790 1928 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'179508607 79875:399919359 [1,0,2]p1 [1,2]p1
> 2024-06-11T17:10:33.165278+0000 2024-05-30T05:53:10.772736+0000
> 575 periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000
> 10.5e 5212770 4506643 0 0 0 5830165136
> 9388228 1938 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180002484 79875:391844220 [2,0,1]p2 [2,1]p2
> 2024-06-11T22:32:49.026280+0000 2024-05-24T15:31:50.362016+0000
> 523 periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000
> 10.5f 5221504 5221504 0 0 4194394 5807975316
> 9387383 1879 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181161493 79875:408530090 [0,2,1]p0 [1,2]p1
> 2024-06-11T15:10:33.261745+0000 2024-05-25T06:37:01.629711+0000
> 845 periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000
> 10.60 5217784 5186615 0 0 4194304 6208553838
> 9749044 1962 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181569072 79875:392211608 [0,1,2]p0 [1,2]p1
> 2024-06-11T21:44:35.372861+0000 2024-06-11T21:44:35.372861+0000
> 1505 periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000
> 10.61 5221292 4586636 0 0 8388608 5766320671
> 9280184 1910 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180945714 79875:344272511 [0,1,2]p0 [1,2]p1
> 2024-06-11T16:51:25.519891+0000 2024-05-24T17:37:21.814566+0000
> 566 periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000
> 10.62 5218498 4699002 0 0 4194466 6027128219
> 9555111 1884 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180387699 79875:341929273 [1,2,0]p1 [1,2]p1
> 2024-06-12T03:10:08.547987+0000 2024-06-01T02:47:43.109035+0000
> 509 periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000
> 10.63 5219101 5219101 0 0 0 6040382110
> 9500040 1885 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181623402 79875:389756489 [0,1,2]p0 [1,2]p1
> 2024-06-02T23:50:14.947543+0000 2024-06-02T23:50:14.947543+0000
> 13925 queued for scrub
> 10.64 5217025 5217025 0 0 2277792 6222337345
> 9701950 1866 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181853168 79875:406174284 [1,2,0]p1 [1,2]p1
> 2024-06-02T04:57:41.872723+0000 2024-06-02T04:57:41.872723+0000
> 6394 queued for scrub
> 10.65 5217908 4590550 0 0 4194326 6223151288
> 9709345 1916 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181196382 79875:340107600 [2,0,1]p2 [2,1]p2
> 2024-06-11T15:53:28.725001+0000 2024-06-11T15:53:28.725001+0000
> 1343 periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000
> 10.66 5218911 4893654 0 0 4194304 5815703055
> 9338987 1939 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'183062531 79875:395193073 [1,0,2]p1 [1,2]p1
> 2024-06-12T01:10:56.919233+0000 2024-05-25T04:50:04.300293+0000
> 590 periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000
> 10.67 5217539 4587208 0 0 4194304 6095526883
> 9640323 1955 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181875315 79875:342878178 [2,1,0]p2 [2,1]p2
> 2024-06-12T07:58:41.522730+0000 2024-06-01T07:20:59.951966+0000
> 553 periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000
> 10.68 5215852 4505902 0 0 8388608 6093858586
> 9596421 1926 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'179490643 79875:379632203 [0,1,2]p0 [1,2]p1
> 2024-06-11T16:02:14.995195+0000 2024-05-30T22:02:42.335910+0000
> 526 periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000
> 10.69 5216596 5216596 0 0 4194304 5833268044
> 9400414 1899 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181876551 79875:387059449 [1,2,0]p1 [1,2]p1
> 2024-06-11T17:00:57.111851+0000 2024-05-23T07:05:05.989427+0000
> 572 periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000
> 10.6a 5214106 4497370 0 0 0 5745504618
> 9296938 1869 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180109669 79875:381871149 [2,1,0]p2 [2,1]p2
> 2024-06-12T00:27:45.155477+0000 2024-05-25T09:22:19.570011+0000
> 494 periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000
> 10.6b 5219339 5219339 0 0 0 5801390847
> 9370084 1957 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180041524 79875:342270608 [0,1,2]p0 [1,2]p1
> 2024-06-12T03:01:39.358464+0000 2024-05-23T01:04:22.522035+0000
> 488 periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000
> 10.6c 5217917 5217917 0 0 554 6100979305
> 9612250 1955 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180968918 79875:341488318 [1,2,0]p1 [1,2]p1
> 2024-06-11T18:47:35.969345+0000 2024-05-31T11:54:59.559316+0000
> 588 periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000
> 10.6d 5221048 4550388 0 0 4194304 6024067862
> 9621947 1927 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'179373269 79875:388095690 [2,1,0]p2 [2,1]p2
> 2024-06-12T03:26:37.931310+0000 2024-05-31T04:10:41.991457+0000
> 502 periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000
> 10.6e 5215550 4888179 0 0 8388608 6110450001
> 9666892 1906 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'180885974 79875:380260263 [0,1,2]p0 [1,2]p1
> 2024-06-11T20:18:23.657558+0000 2024-06-11T20:18:23.657558+0000
> 1218 periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000
> 10.6f 5215850 4566820 0 0 0 6107146788
> 9634911 1923 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181752773 79875:396708021 [2,1,0]p2 [2,1]p2
> 2024-06-10T17:32:51.735004+0000 2024-06-10T17:32:51.735004+0000
> 10119 periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000
> 10.70 5218211 4672319 0 0 4194304 5793441438
> 9371563 1881 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180405012 79875:404578822 [1,2,0]p1 [1,2]p1
> 2024-06-03T04:46:35.130293+0000 2024-05-22T07:50:20.898441+0000
> 969 queued for deep scrub
> 10.71 5218299 4599312 0 0 4194304 5807950891
> 9349720 1906 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180690682 79875:344108361 [0,2,1]p0 [1,2]p1
> 2024-06-12T03:18:16.645927+0000 2024-05-23T09:32:56.055703+0000
> 487 periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000
> 10.72 5223470 4922347 0 0 8388608 6416468325
> 9922777 1958 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'180797342 79875:342356916 [0,1,2]p0 [1,2]p1
> 2024-06-12T05:42:47.545762+0000 2024-06-12T05:42:47.545762+0000
> 1307 periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000
> 10.73 5218301 4520651 0 0 0 6293832301
> 9738943 1953 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'182246986 79875:403792984 [2,0,1]p2 [2,1]p2
> 2024-06-11T20:59:02.079157+0000 2024-06-11T20:59:02.079157+0000
> 1399 periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000
> 10.74 5219061 4595112 0 0 0 5974775781
> 9469653 1943 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181947609 79875:420571221 [2,1,0]p2 [2,1]p2
> 2024-06-03T02:17:19.687482+0000 2024-05-28T08:43:38.825954+0000
> 927 queued for scrub
> 10.75 5220994 4596634 0 0 25165824 5822315216
> 9362548 1877 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'180376305 79875:338847231 [2,0,1]p2 [2,1]p2
> 2024-06-03T05:52:28.743255+0000 2024-05-24T05:11:00.211501+0000
> 981 queued for scrub
> 10.76 5222111 4892295 0 0 0 6067998823
> 9574254 1935 3000
active+undersized+degraded+remapped+backfilling
> 2h 79875'183871180 79875:409218100 [1,0,2]p1 [1,2]p1
> 2024-06-03T04:14:04.513004+0000 2024-06-03T04:14:04.513004+0000
> 6987 queued for scrub
> 10.77 5219020 4574683 0 0 0 5889074277
> 9460824 1896 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181614825 79875:342558162 [2,0,1]p2 [2,1]p2
> 2024-06-12T08:07:38.169523+0000 2024-05-27T15:15:35.587972+0000
> 536 periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000
> 10.78 5220442 4693235 0 0 8388608 5879566566
> 9395501 1923 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'179031456 79875:391462415 [1,0,2]p1 [1,2]p1
> 2024-06-02T09:32:42.834344+0000 2024-05-27T10:41:23.822124+0000
> 994 queued for scrub
> 10.79 5219734 4702117 0 0 4194304 6129933545
> 9646669 1926 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'182574416 79875:402249812 [1,2,0]p1 [1,2]p1
> 2024-06-12T05:52:02.959814+0000 2024-06-01T00:53:49.548055+0000
> 555 periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000
> 10.7a 5215303 4526846 0 0 0 6316171107
> 9817029 1932 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181765231 79875:395877057 [2,0,1]p2 [2,1]p2
> 2024-06-12T02:23:42.692096+0000 2024-06-12T02:23:42.692096+0000
> 1457 periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000
> 10.7b 5217213 4606816 0 0 4194304 6226859283
> 9718621 1952 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'181413541 79875:343968658 [1,2,0]p1 [1,2]p1
> 2024-06-11T23:52:28.414752+0000 2024-06-11T23:52:28.414752+0000
> 1308 periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000
> 10.7c 5215075 5215075 0 0 4194304 6013877208
> 9602345 1867 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'180991860 79875:341775615 [0,2,1]p0 [1,2]p1
> 2024-06-11T23:30:40.614412+0000 2024-05-31T14:10:30.542603+0000
> 503 periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000
> 10.7d 5220700 4569720 0 0 5837609 6258736025
> 9764836 1897 3000
active+undersized+degraded+remapped+backfilling
> 3h 79875'179741885 79875:401448508 [2,1,0]p2 [2,1]p2
> 2024-06-12T04:45:05.030905+0000 2024-06-12T04:45:05.030905+0000
> 1565 periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000
> 10.7e 5217113 5217113 0 0 16777216 6072081676
> 9598390 1902 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'181040361 79875:391032348 [1,2,0]p1 [1,2]p1
> 2024-06-11T17:37:26.589968+0000 2024-06-11T17:37:26.589968+0000
> 1613 periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000
> 10.7f 5218719 5218719 0 0 9909061 6130469539
> 9657824 1943 3000
active+undersized+degraded+remapped+backfill_wait
> 3h 79875'182283514 79875:408550888 [1,2,0]p1 [1,2]p1
> 2024-06-11T13:05:03.552526+0000 2024-06-11T13:05:03.552526+0000
> 1796 periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000
>
> * NOTE: Omap statistics are gathered during deep scrub and may be
> inaccurate soon afterwards depending on utilization. See
> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for
> further details.
>
>
>
> sudo ceph osd df
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS
> 0 ssd 3.63869 1.00000 3.6 TiB 82 GiB 50 MiB 40 GiB 42
GiB
> 3.6 TiB 2.21 0.03 0 up
> 1 ssd 3.63869 1.00000 3.6 TiB 2.0 TiB 16 GiB 1021 GiB 1.0
TiB
> 1.6 TiB 55.41 0.75 161 up
> 2 ssd 3.63869 1.00000 3.6 TiB 3.0 TiB 16 GiB 1.1 TiB 1.9
TiB
> 676 GiB 81.86 1.11 161 up
>
>
> sudo ceph df detail
> --- RAW STORAGE ---
> CLASS SIZE AVAIL USED RAW USED %RAW USED
> hdd 1.2 PiB 308 TiB 871 TiB 871 TiB 73.86
> ssd 11 TiB 5.8 TiB 5.1 TiB 5.1 TiB 46.51
> TOTAL 1.2 PiB 314 TiB 876 TiB 876 TiB 73.61
>
> --- POOLS ---
> POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA)
> (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR
> UNDER COMPR
> .mgr 1 1 827 MiB 827 MiB 0 B 208 1.6 GiB 1.6 GiB
> 0 B 0.11 734 GiB N/A 50 GiB N/A 0 B
> 0 B
> metadata 10 128 1.0 TiB 647 MiB 1.0 TiB 667.90M 2.1 TiB 1.3 GiB
> 2.1 TiB 59.45 719 GiB N/A 3.5 TiB N/A 0 B
> 0 B
> data 11 2048 288 TiB 288 TiB 0 B 566.97M 863 TiB 863 TiB
> 0 B 81.10 67 TiB N/A N/A N/A 3.4 TiB
> 6.8 TiB
> ssd-data 18 32 14 GiB 14 GiB 0 B 265.17k 29 GiB 29 GiB
> 0 B 1.90 734 GiB N/A N/A N/A 0 B
> 0 B
>
>
> sudo ceph balancer status
> {
> "active": true,
> "last_optimize_duration": "0:00:00.000170",
> "last_optimize_started": "Wed Jun 12 13:14:49 2024",
> "mode": "upmap",
> "no_optimization_needed": true,
> "optimize_result": "Some objects (0.172816) are degraded; try again
> later",
> "plans": []
> }
>
>
>
> [image: ariadne.ai Logo] Lars Köppel
> Developer
> Email: lars.koeppel@xxxxxxxxxx
> Phone: +49 6221 5993580 <+4962215993580>
> ariadne.ai (Germany) GmbH
> Häusserstraße 3, 69115 Heidelberg
> Amtsgericht Mannheim, HRB 744040
> Geschäftsführer: Dr. Fabian Svara
> https://ariadne.ai
>
>
> On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri <anthony.datri@xxxxxxxxx>
> wrote:
>
>> If you have:
>>
>> * pg_num too low (defaults are too low)
>> * pg_num not a power of 2
>> * pg_num != number of OSDs in the pool
>> * balancer not enabled
>>
>> any of those might result in imbalance.
>>
>> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote:
>> >
>> > I don't have any good explanation at this point. Can you share some
more
>> information like:
>> >
>> > ceph pg ls-by-pool <cephfs_metadata>
>> > ceph osd df (for the relevant OSDs)
>> > ceph df
>> >
>> > Thanks,
>> > Eugen
>> >
>> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>> >
>> >> Since my last update the size of the largest OSD increased by 0.4 TiB
>> while
>> >> the smallest one only increased by 0.1 TiB. How is this possible?
>> >>
>> >> Because the metadata pool reported to have only 900MB space left, I
>> stopped
>> >> the hot-standby MDS. This gave me 8GB back but these filled up in the
>> last
>> >> 2h.
>> >> I think I have to zap the next OSD because the filesystem is getting
>> read
>> >> only...
>> >>
>> >> How is it possible that an OSD has over 1 TiB less data on it after a
>> >> rebuild? And how is it possible to have so different sizes of OSDs?
>> >>
>> >>
>> >> [image: ariadne.ai Logo] Lars Köppel
>> >> Developer
>> >> Email: lars.koeppel@xxxxxxxxxx
>> >> Phone: +49 6221 5993580 <+4962215993580>
>> >> ariadne.ai (Germany) GmbH
>> >> Häusserstraße 3, 69115 Heidelberg
>> >> Amtsgericht Mannheim, HRB 744040
>> >> Geschäftsführer: Dr. Fabian Svara
>> >> https://ariadne.ai
>> >>
>> >>
>> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel <lars.koeppel@xxxxxxxxxx
>
>> wrote:
>> >>
>> >>> Only in warning mode. And there were no PG splits or merges in the
>> last 2
>> >>> month.
>> >>>
>> >>>
>> >>> [image: ariadne.ai Logo] Lars Köppel
>> >>> Developer
>> >>> Email: lars.koeppel@xxxxxxxxxx
>> >>> Phone: +49 6221 5993580 <+4962215993580>
>> >>> ariadne.ai (Germany) GmbH
>> >>> Häusserstraße 3, 69115 Heidelberg
>> >>> Amtsgericht Mannheim, HRB 744040
>> >>> Geschäftsführer: Dr. Fabian Svara
>> >>> https://ariadne.ai
>> >>>
>> >>>
>> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> wrote:
>> >>>
>> >>>> I don't think scrubs can cause this. Do you have autoscaler
enabled?
>> >>>>
>> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>> >>>>
>> >>>> > Hi,
>> >>>> >
>> >>>> > thank you for your response.
>> >>>> >
>> >>>> > I don't think this thread covers my problem, because the OSDs for
>> the
>> >>>> > metadata pool fill up at different rates. So I would think this
is
>> no
>> >>>> > direct problem with the journal.
>> >>>> > Because we had earlier problems with the journal I changed some
>> >>>> > settings(see below). I already restarted all MDS multiple times
but
>> no
>> >>>> > change here.
>> >>>> >
>> >>>> > The health warnings regarding cache pressure resolve normally
after
>> a
>> >>>> > short period of time, when the heavy load on the client ends.
>> Sometimes
>> >>>> it
>> >>>> > stays a bit longer because an rsync is running and copying data
on
>> the
>> >>>> > cluster(rsync is not good at releasing the caps).
>> >>>> >
>> >>>> > Could it be a problem if scrubs run most of the time in the
>> background?
>> >>>> Can
>> >>>> > this block any other tasks or generate new data itself?
>> >>>> >
>> >>>> > Best regards,
>> >>>> > Lars
>> >>>> >
>> >>>> >
>> >>>> > global basic mds_cache_memory_limit
>> >>>> > 17179869184
>> >>>> > global advanced mds_max_caps_per_client
>> >>>> > 16384
>> >>>> > global advanced
>> >>>> mds_recall_global_max_decay_threshold
>> >>>> > 262144
>> >>>> > global advanced mds_recall_max_decay_rate
>> >>>> > 1.000000
>> >>>> > global advanced
mds_recall_max_decay_threshold
>> >>>> > 262144
>> >>>> > mds advanced mds_cache_trim_threshold
>> >>>> > 131072
>> >>>> > mds advanced mds_heartbeat_grace
>> >>>> > 120.000000
>> >>>> > mds advanced mds_heartbeat_reset_grace
>> >>>> > 7400
>> >>>> > mds advanced mds_tick_interval
>> >>>> > 3.000000
>> >>>> >
>> >>>> >
>> >>>> > [image: ariadne.ai Logo] Lars Köppel
>> >>>> > Developer
>> >>>> > Email: lars.koeppel@xxxxxxxxxx
>> >>>> > Phone: +49 6221 5993580 <+4962215993580>
>> >>>> > ariadne.ai (Germany) GmbH
>> >>>> > Häusserstraße 3, 69115 Heidelberg
>> >>>> > Amtsgericht Mannheim, HRB 744040
>> >>>> > Geschäftsführer: Dr. Fabian Svara
>> >>>> > https://ariadne.ai
>> >>>> >
>> >>>> >
>> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx>
wrote:
>> >>>> >
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> can you check if this thread [1] applies to your situation? You
>> don't
>> >>>> >> have multi-active MDS enabled, but maybe it's still some journal
>> >>>> >> trimming, or maybe misbehaving clients? In your first post there
>> were
>> >>>> >> health warnings regarding cache pressure and cache size. Are
those
>> >>>> >> resolved?
>> >>>> >>
>> >>>> >> [1]
>> >>>> >>
>> >>>> >>
>> >>>>
>>
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE
>> >>>> >>
>> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>:
>> >>>> >>
>> >>>> >> > Hello everyone,
>> >>>> >> >
>> >>>> >> > short update to this problem.
>> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected
>> size
>> >>>> >> ~50%).
>> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They
>> jumped up
>> >>>> and
>> >>>> >> > down a lot but the higher one has now also reached 'nearfull'
>> >>>> status. How
>> >>>> >> > is this possible? What is going on?
>> >>>> >> >
>> >>>> >> > Does anyone have a solution how to fix this without zapping
the
>> OSD?
>> >>>> >> >
>> >>>> >> > Best regards,
>> >>>> >> > Lars
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > [image: ariadne.ai Logo] Lars Köppel
>> >>>> >> > Developer
>> >>>> >> > Email: lars.koeppel@xxxxxxxxxx
>> >>>> >> > Phone: +49 6221 5993580 <+4962215993580>
>> >>>> >> > ariadne.ai (Germany) GmbH
>> >>>> >> > Häusserstraße 3, 69115 Heidelberg
>> >>>> >> > Amtsgericht Mannheim, HRB 744040
>> >>>> >> > Geschäftsführer: Dr. Fabian Svara
>> >>>> >> > https://ariadne.ai
>> >>>> >> > _______________________________________________
>> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>> >>
>> >>>> >>
>> >>>> >> _______________________________________________
>> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
>> >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>> >>>> >>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>>
