I updated from 17.2.6 to 17.2.7 and a few hours later to 18.2.2. Would it be an option to go back to 17.2.6? [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@xxxxxxxxxx Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Wed, Jun 12, 2024 at 5:30 PM Eugen Block <eblock@xxxxxx> wrote: > Which version did you upgrade from to 18.2.2? > I can’t pin it down to a specific issue, but somewhere in the back of > my mind is something related to a new omap format or something. But > I’m really not sure at all. > > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > > > I am happy to help you with as much information as possible. I probably > > just don't know where to look for it. > > Below are the requested information. The cluster is rebuilding the > > zapped OSD at the moment. This will probably take the next few days. > > > > > > sudo ceph pg ls-by-pool metadata > > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* > > OMAP_KEYS* LOG LOG_DUPS STATE > > SINCE VERSION REPORTED UP ACTING > > SCRUB_STAMP DEEP_SCRUB_STAMP > > LAST_SCRUB_DURATION SCRUB_SCHEDULING > > 10.0 5217325 4994695 0 0 4194304 5880891340 > > 9393865 1885 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180849582 79875:391519635 [0,1,2]p0 [1,2]p1 > > 2024-06-11T09:08:09.829362+0000 2024-05-28T05:52:59.321589+0000 > > 627 periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000 > > 10.1 5214785 5193424 0 0 0 5843682713 > > 9410150 1912 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180914288 79875:342746928 [2,1,0]p2 [2,1]p2 > > 2024-06-01T15:56:28.927288+0000 2024-05-27T03:31:37.682966+0000 > > 966 queued for scrub > > 10.2 5218432 5187168 0 0 0 6402011266 > > 9812513 1874 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180970531 79875:341340204 [0,1,2]p0 [1,2]p1 > > 2024-06-11T13:40:58.994256+0000 2024-06-11T13:40:58.994256+0000 > > 1942 periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000 > > 10.3 5217413 5217413 0 0 8388788 5766005023 > > 9271787 1923 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181012233 79875:388295881 [1,0,2]p1 [1,2]p1 > > 2024-06-12T00:35:56.965547+0000 2024-05-23T19:54:56.121729+0000 > > 492 periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000 > > 10.4 5220069 5220069 0 0 12583466 6027548724 > > 9537290 1959 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181576075 79875:405295868 [1,2,0]p1 [1,2]p1 > > 2024-06-11T17:47:22.923514+0000 2024-05-31T02:06:55.339574+0000 > > 581 periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000 > > 10.5 5216162 5211999 0 0 4194304 5941347251 > > 9542764 1930 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180455793 79875:338418517 [2,1,0]p2 [2,1]p2 > > 2024-06-11T22:50:16.170708+0000 2024-05-30T23:49:54.316379+0000 > > 528 periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000 > > 10.6 5216100 4980459 0 0 4521984 6428088514 > > 9850762 1911 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'184045876 79875:396809795 [0,2,1]p0 [1,2]p1 > > 2024-06-11T22:24:05.102716+0000 2024-06-11T22:24:05.102716+0000 > > 1082 periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000 > > 10.7 5218232 5218232 0 0 4194304 6377065363 > > 9849360 1919 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182672562 79875:342449062 [1,0,2]p1 [1,2]p1 > > 2024-06-11T06:22:15.689422+0000 2024-06-11T06:22:15.689422+0000 > > 8225 periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000 > > 10.8 5219620 5182816 0 0 0 6167304290 > > 9691796 1896 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179628377 79875:378022884 [2,1,0]p2 [2,1]p2 > > 2024-06-11T22:06:01.386763+0000 2024-06-11T22:06:01.386763+0000 > > 1286 periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000 > > 10.9 5219448 5164591 0 0 8388698 5796048346 > > 9338312 1868 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181739392 79875:387412389 [2,1,0]p2 [2,1]p2 > > 2024-06-12T05:21:00.586747+0000 2024-05-26T11:10:59.780673+0000 > > 539 periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000 > > 10.a 5219861 5163635 0 0 12582912 5841839055 > > 9387200 1916 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180205688 79875:379381294 [1,2,0]p1 [1,2]p1 > > 2024-06-11T12:35:05.571200+0000 2024-05-22T11:07:16.041773+0000 > > 1093 periodic deep scrub scheduled @ > 2024-06-17T05:21:40.136463+0000 > > 10.b 5217949 5217949 0 0 16777216 5935863260 > > 9462127 1881 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181655745 79875:343806807 [0,1,2]p0 [1,2]p1 > > 2024-06-11T22:41:28.976920+0000 2024-05-26T08:43:29.217457+0000 > > 520 periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000 > > 10.c 5221697 5217118 0 0 4194304 6015217841 > > 9574445 1928 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180892826 79875:341490398 [2,1,0]p2 [2,1]p2 > > 2024-06-11T09:20:58.443473+0000 2024-05-30T00:13:50.306507+0000 > > 768 periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000 > > 10.d 5217727 4908764 0 0 0 5825598519 > > 9349877 1930 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'179050347 79875:387455993 [0,1,2]p0 [1,2]p1 > > 2024-06-02T07:27:27.873631+0000 2024-05-20T03:58:30.225170+0000 > > 952 queued for deep scrub > > 10.e 5215040 5210572 0 0 4194304 5790634469 > > 9327651 1905 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180049044 79875:377196119 [2,1,0]p2 [2,1]p2 > > 2024-06-12T05:03:08.521423+0000 2024-05-22T13:47:46.753131+0000 > > 527 periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000 > > 10.f 5217202 4885952 0 0 8388630 6005274626 > > 9589178 1917 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'181953069 79875:396402453 [0,2,1]p0 [1,2]p1 > > 2024-06-11T12:20:49.661741+0000 2024-06-11T12:20:49.661741+0000 > > 3949 periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000 > > 10.10 5214985 5157391 0 0 4194394 6275024632 > > 9721505 1922 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181801708 79875:395055895 [2,0,1]p2 [2,1]p2 > > 2024-06-11T01:49:47.831191+0000 2024-06-11T01:49:47.831191+0000 > > 9579 periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000 > > 10.11 5219236 4983134 0 0 4194304 5787580073 > > 9316893 1901 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'181011816 79875:343488767 [1,0,2]p1 [1,2]p1 > > 2024-06-03T00:06:14.862209+0000 2024-05-27T00:05:56.468361+0000 > > 960 queued for scrub > > 10.12 5214103 5182770 0 0 90 5970750188 > > 9505613 1891 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180210613 79875:341042616 [0,1,2]p0 [1,2]p1 > > 2024-06-12T06:39:48.252419+0000 2024-05-28T01:32:03.381577+0000 > > 527 periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000 > > 10.13 5214580 5214580 0 0 8388642 5793432968 > > 9335940 1957 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182394614 79875:393423214 [1,2,0]p1 [1,2]p1 > > 2024-06-10T17:52:04.690203+0000 2024-05-22T03:39:02.453336+0000 > > 1151 periodic deep scrub scheduled @ > 2024-06-16T00:28:37.682765+0000 > > 10.14 5218591 5218591 0 0 5053325 6046268958 > > 9582481 1915 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'184576050 79875:425213800 [2,0,1]p2 [2,1]p2 > > 2024-06-03T05:36:07.656965+0000 2024-05-28T22:46:16.814318+0000 > > 985 queued for scrub > > 10.15 5215919 4907184 0 0 11939335 5752284246 > > 9285870 1889 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'179713036 79875:337938973 [0,2,1]p0 [1,2]p1 > > 2024-06-11T14:06:50.138992+0000 2024-05-24T07:52:22.522455+0000 > > 786 periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000 > > 10.16 5221091 5221091 0 0 8388608 5966308565 > > 9487816 1916 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183445984 79875:399413009 [0,2,1]p0 [1,2]p1 > > 2024-06-03T02:01:53.649417+0000 2024-06-03T02:01:53.649417+0000 > > 6935 queued for scrub > > 10.17 5218069 5194732 0 0 4194304 6231638001 > > 9735915 1890 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182390128 79875:340719915 [2,0,1]p2 [2,1]p2 > > 2024-06-12T06:31:01.523916+0000 2024-06-12T06:31:01.523916+0000 > > 1272 periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000 > > 10.18 5214253 5159897 0 0 0 6128506469 > > 9673261 1886 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179754208 79875:389150273 [2,1,0]p2 [2,1]p2 > > 2024-06-12T02:53:31.027154+0000 2024-06-12T02:53:31.027154+0000 > > 1308 periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000 > > 10.19 5220131 5183342 0 0 0 5869775360 > > 9460583 1953 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181750923 79875:393162989 [2,0,1]p2 [2,1]p2 > > 2024-06-11T19:21:24.106504+0000 2024-05-22T15:42:01.721091+0000 > > 522 periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000 > > 10.1a 5217968 5186700 0 0 4194304 6183652625 > > 9733847 1896 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181056240 79875:394241128 [0,2,1]p0 [1,2]p1 > > 2024-06-02T15:37:49.730965+0000 2024-06-02T15:37:49.730965+0000 > > 7347 queued for scrub > > 10.1b 5221259 5161879 0 0 0 5989030303 > > 9561035 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181052305 79875:343144823 [0,1,2]p0 [1,2]p1 > > 2024-06-12T06:01:07.213894+0000 2024-06-02T00:23:51.063901+0000 > > 543 periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000 > > 10.1c 5217034 5217034 0 0 554 5979861913 > > 9504442 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180913313 79875:341328275 [1,2,0]p1 [1,2]p1 > > 2024-06-01T20:02:47.025129+0000 2024-06-01T20:02:47.025129+0000 > > 6568 queued for scrub > > 10.1d 5219691 4997027 0 0 12582912 6081321658 > > 9599164 1918 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'179638837 79875:391808225 [0,2,1]p0 [1,2]p1 > > 2024-06-11T20:27:02.506715+0000 2024-05-31T22:58:06.578167+0000 > > 517 periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000 > > 10.1e 5213369 5208776 0 0 4194304 6295629499 > > 9765317 1941 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181145210 79875:391326487 [2,1,0]p2 [2,1]p2 > > 2024-06-12T07:49:28.432084+0000 2024-06-12T07:49:28.432084+0000 > > 1990 periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000 > > 10.1f 5218399 5000094 0 0 8388608 5821063844 > > 9421222 1896 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180960296 79875:400635456 [0,2,1]p0 [1,2]p1 > > 2024-06-11T13:53:40.278050+0000 2024-05-22T19:39:02.345663+0000 > > 761 periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000 > > 10.20 5216437 5185379 0 0 0 6280994122 > > 9719574 1945 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181610117 79875:391874431 [0,1,2]p0 [1,2]p1 > > 2024-06-11T12:58:57.754648+0000 2024-06-11T12:58:57.754648+0000 > > 2288 periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000 > > 10.21 5217349 5217349 0 0 8388698 6348931429 > > 9822197 1867 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181724357 79875:343883762 [0,1,2]p0 [1,2]p1 > > 2024-06-11T18:28:34.044546+0000 2024-06-11T18:28:34.044546+0000 > > 1817 periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000 > > 10.22 5215544 4943475 0 0 4194304 5982815232 > > 9482159 1899 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180230331 79875:340839853 [2,0,1]p2 [2,1]p2 > > 2024-06-11T15:21:21.994674+0000 2024-05-30T08:01:03.301650+0000 > > 649 periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000 > > 10.23 5217758 5217758 0 0 146 5804086623 > > 9336298 1920 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180948432 79875:388158699 [1,2,0]p1 [1,2]p1 > > 2024-06-03T05:19:32.311846+0000 2024-05-27T08:13:15.250058+0000 > > 971 queued for scrub > > 10.24 5221505 5221505 0 0 0 6360995644 > > 9816333 1925 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182198758 79875:406847098 [0,2,1]p0 [1,2]p1 > > 2024-06-12T04:00:44.315682+0000 2024-06-12T04:00:44.315682+0000 > > 2046 periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000 > > 10.25 5214911 5214911 0 0 12583002 5726320657 > > 9264352 1910 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180132024 79875:338167995 [1,2,0]p1 [1,2]p1 > > 2024-06-11T14:43:47.076906+0000 2024-05-23T12:16:34.271856+0000 > > 715 periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000 > > 10.26 5217588 4973667 0 0 0 6040775867 > > 9598406 1927 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'188144536 79875:400277134 [2,0,1]p2 [2,1]p2 > > 2024-06-02T06:55:09.016528+0000 2024-06-02T06:55:09.016528+0000 > > 7047 queued for scrub > > 10.27 5218490 5218490 0 0 8388630 6265942186 > > 9818746 1941 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182400328 79875:342052414 [1,2,0]p1 [1,2]p1 > > 2024-06-12T01:01:06.674323+0000 2024-06-12T01:01:06.674323+0000 > > 1500 periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000 > > 10.28 5221153 5221153 0 0 0 6049095371 > > 9597182 1913 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179438777 79875:378265375 [1,2,0]p1 [1,2]p1 > > 2024-06-12T04:09:58.896192+0000 2024-06-01T04:32:46.011974+0000 > > 553 periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000 > > 10.29 5214066 5214066 0 0 4194304 5767555623 > > 9316899 1937 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181568997 79875:387683159 [0,2,1]p0 [1,2]p1 > > 2024-06-10T22:48:57.757493+0000 2024-05-21T07:14:53.348341+0000 > > 938 periodic deep scrub scheduled @ > 2024-06-16T06:24:55.108695+0000 > > 10.2a 5216314 4977811 0 0 4194304 6012567810 > > 9563804 1951 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'181140734 79875:381017643 [2,1,0]p2 [2,1]p2 > > 2024-06-11T09:57:29.097708+0000 2024-05-30T03:47:22.798044+0000 > > 1169 periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000 > > 10.2b 5219620 5219620 0 0 8388608 5829917912 > > 9412426 1889 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180465782 79875:342263856 [1,0,2]p1 [1,2]p1 > > 2024-06-12T08:30:08.941782+0000 2024-05-24T22:00:29.535205+0000 > > 636 periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000 > > 10.2c 5214087 5214087 0 0 0 5874182922 > > 9375708 1871 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180616041 79875:341244736 [1,2,0]p1 [1,2]p1 > > 2024-06-10T23:10:08.184410+0000 2024-05-27T17:41:07.583255+0000 > > 1268 periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000 > > 10.2d 5213968 4984822 0 0 4337824 5835754026 > > 9417033 1947 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'178824607 79875:386427308 [2,0,1]p2 [2,1]p2 > > 2024-06-12T09:27:43.774331+0000 2024-05-27T05:33:45.293266+0000 > > 494 periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000 > > 10.2e 5216263 4974456 0 0 12582912 6025714323 > > 9553880 1940 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'180639631 79875:378284872 [1,0,2]p1 [1,2]p1 > > 2024-06-12T01:19:42.477063+0000 2024-05-31T08:42:41.293335+0000 > > 526 periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000 > > 10.2f 5220057 5220057 0 0 13839831 5804027574 > > 9370862 1882 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180961915 79875:394513378 [1,0,2]p1 [1,2]p1 > > 2024-06-11T19:58:05.145147+0000 2024-05-24T20:18:32.301831+0000 > > 535 periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000 > > 10.30 5217090 4955469 0 0 12582934 6438521431 > > 9924825 1869 3000 > active+undersized+degraded+remapped+backfill_wait > > 2h 79875'181488488 79875:404940886 [2,1,0]p2 [2,1]p2 > > 2024-06-12T07:16:18.095439+0000 2024-06-12T07:16:18.095439+0000 > > 2190 periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000 > > 10.31 5218176 5218176 0 0 4194304 6215016361 > > 9752698 1897 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181622879 79875:343554151 [1,2,0]p1 [1,2]p1 > > 2024-06-12T09:00:15.941065+0000 2024-06-12T09:00:15.941065+0000 > > 1806 periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000 > > 10.32 5223171 4704732 0 0 8388608 6205669256 > > 9800320 1920 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180911263 79875:342754128 [0,2,1]p0 [1,2]p1 > > 2024-06-11T19:49:09.010583+0000 2024-06-11T19:49:09.010583+0000 > > 1664 periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000 > > 10.33 5215341 4525338 0 0 4194304 6066965588 > > 9635403 1888 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181467497 79875:402958597 [1,2,0]p1 [1,2]p1 > > 2024-06-12T01:59:23.933458+0000 2024-05-31T17:01:38.620377+0000 > > 512 periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000 > > 10.34 5217342 5217342 0 0 8388608 5999920961 > > 9570527 1956 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182013407 79875:419219483 [1,0,2]p1 [1,2]p1 > > 2024-06-12T09:10:17.390128+0000 2024-06-01T15:40:23.241769+0000 > > 601 periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000 > > 10.35 5222114 5222114 0 0 8388608 6021420075 > > 9571300 1952 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180948342 79875:338598678 [1,2,0]p1 [1,2]p1 > > 2024-06-11T22:59:17.017317+0000 2024-05-30T15:59:15.553266+0000 > > 539 periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000 > > 10.36 5223657 5223657 0 0 4331271 5844578100 > > 9347405 1903 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183363403 79875:408484683 [0,2,1]p0 [1,2]p1 > > 2024-06-11T18:37:46.653097+0000 2024-05-25T14:13:51.648712+0000 > > 553 periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000 > > 10.37 5219248 5219248 0 0 4194304 6072826085 > > 9579049 1936 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182091890 79875:341545087 [1,0,2]p1 [1,2]p1 > > 2024-06-11T14:19:40.744743+0000 2024-05-29T18:40:12.602405+0000 > > 771 periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000 > > 10.38 5214022 5214022 0 0 4194858 5780805541 > > 9299038 1951 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'178809936 79875:390113817 [1,0,2]p1 [1,2]p1 > > 2024-06-02T00:55:22.046433+0000 2024-05-25T16:49:10.279003+0000 > > 931 queued for scrub > > 10.39 5220692 4542094 0 0 0 6401799077 > > 9864164 1872 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182881827 79875:402437400 [2,0,1]p2 [2,1]p2 > > 2024-06-12T01:42:35.239155+0000 2024-06-12T01:42:35.239155+0000 > > 1370 periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000 > > 10.3a 5215648 4552651 0 0 0 6194465524 > > 9713558 1898 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181458295 79875:394916652 [2,1,0]p2 [2,1]p2 > > 2024-06-11T21:19:29.020889+0000 2024-06-11T21:19:29.020889+0000 > > 1227 periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000 > > 10.3b 5218220 5218220 0 0 8388608 5951035110 > > 9490698 1878 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180815984 79875:342438023 [2,1,0]p2 [2,1]p2 > > 2024-06-03T05:03:19.483049+0000 2024-05-28T18:50:55.369009+0000 > > 1004 queued for scrub > > 10.3c 5221615 5221615 0 0 12582912 6260894058 > > 9727358 1938 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181561627 79875:342153262 [1,2,0]p1 [1,2]p1 > > 2024-06-11T19:12:06.206008+0000 2024-06-11T19:12:06.206008+0000 > > 1471 periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000 > > 10.3d 5213771 5213771 0 0 0 6215975311 > > 9699615 1946 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179354493 79875:399858588 [1,2,0]p1 [1,2]p1 > > 2024-06-11T04:05:10.877703+0000 2024-06-11T04:05:10.877703+0000 > > 8123 periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000 > > 10.3e 5219446 4659577 0 0 4194304 5852588566 > > 9370152 1933 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180035882 79875:391595513 [0,1,2]p0 [1,2]p1 > > 2024-06-11T20:35:43.624903+0000 2024-05-22T21:40:55.606650+0000 > > 520 periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000 > > 10.3f 5219729 4889443 0 0 16777216 5925457842 > > 9484458 1948 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'181330738 79875:409113809 [0,2,1]p0 [1,2]p1 > > 2024-06-12T05:12:01.852109+0000 2024-05-27T20:03:22.704122+0000 > > 533 periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000 > > 10.40 5222241 4445164 0 0 0 6312545191 > > 9773697 1873 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181341418 79875:393186178 [2,1,0]p2 [2,1]p2 > > 2024-06-10T20:00:37.561045+0000 2024-06-10T20:00:37.561045+0000 > > 7712 periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000 > > 10.41 5215195 5183508 0 0 4194326 6000770355 > > 9529537 1938 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181494334 79875:343712275 [0,2,1]p0 [1,2]p1 > > 2024-06-12T04:18:58.693597+0000 2024-06-01T11:05:59.370690+0000 > > 539 periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000 > > 10.42 5219564 4681323 0 0 12582912 5822745109 > > 9331625 1916 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179811520 79875:340958258 [1,2,0]p1 [1,2]p1 > > 2024-06-02T19:58:06.439770+0000 2024-05-21T18:28:28.794738+0000 > > 11044 queued for deep scrub > > 10.43 5219412 4599458 0 0 594 5737440613 > > 9239869 1950 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180634536 79875:388879991 [1,0,2]p1 [1,2]p1 > > 2024-06-01T18:13:17.068416+0000 2024-05-20T16:32:20.302230+0000 > > 1002 queued for deep scrub > > 10.44 5216008 5216008 0 0 4194304 6270195298 > > 9786806 1944 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182088681 79875:406175480 [1,0,2]p1 [1,2]p1 > > 2024-06-12T00:19:01.240405+0000 2024-06-12T00:19:01.240405+0000 > > 1593 periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000 > > 10.45 5216238 4694313 0 0 0 5714807184 > > 9223666 1944 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179709878 79875:338307727 [1,0,2]p1 [1,2]p1 > > 2024-06-02T00:39:50.657116+0000 2024-05-21T02:52:14.376959+0000 > > 958 queued for deep scrub > > 10.46 5217950 5217950 0 0 9729074 5945415128 > > 9499020 1915 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183726716 79875:395024265 [1,2,0]p1 [1,2]p1 > > 2024-06-11T14:56:28.875949+0000 2024-05-29T04:19:53.409918+0000 > > 724 periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000 > > 10.47 5219125 5219125 0 0 4194304 5775128028 > > 9299099 1864 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181328922 79875:340510799 [1,0,2]p1 [1,2]p1 > > 2024-06-11T17:58:16.753327+0000 2024-05-23T14:24:25.876796+0000 > > 652 periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000 > > 10.48 5216471 5185269 0 0 4194326 6352053972 > > 9843580 1929 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179547710 79875:378125137 [0,1,2]p0 [1,2]p1 > > 2024-06-11T23:22:17.435614+0000 2024-06-11T23:22:17.435614+0000 > > 1379 periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000 > > 10.49 5217119 4580870 0 0 4194304 5971558944 > > 9526047 1879 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182489297 79875:389298429 [2,0,1]p2 [2,1]p2 > > 2024-06-11T08:57:42.791265+0000 2024-05-29T07:13:57.867295+0000 > > 959 periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000 > > 10.4a 5217577 4909174 0 0 4194304 5960086599 > > 9511657 1910 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'180982438 79875:381432369 [0,1,2]p0 [1,2]p1 > > 2024-06-12T02:31:42.576663+0000 2024-06-01T09:08:07.860775+0000 > > 480 periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000 > > 10.4b 5220270 5220270 0 0 159808 6313986186 > > 9763672 1934 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181279214 79875:342509944 [2,1,0]p2 [2,1]p2 > > 2024-06-11T11:03:38.997373+0000 2024-06-11T11:03:38.997373+0000 > > 3970 periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000 > > 10.4c 5217864 5217864 0 0 4194304 6104537460 > > 9679170 1948 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181081348 79875:341937760 [1,0,2]p1 [1,2]p1 > > 2024-06-12T06:09:49.667680+0000 2024-06-01T13:06:12.938513+0000 > > 520 periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000 > > 10.4d 5217604 4905278 0 0 8388608 6250536027 > > 9765571 1917 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'179562016 79875:387768172 [0,1,2]p0 [1,2]p1 > > 2024-06-11T13:38:28.479053+0000 2024-06-11T13:38:28.479053+0000 > > 2004 periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000 > > 10.4e 5220202 5220202 0 0 12215022 5781813383 > > 9291138 1940 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180360470 79875:377712114 [1,0,2]p1 [1,2]p1 > > 2024-06-01T22:06:12.347760+0000 2024-05-25T19:34:49.805104+0000 > > 982 queued for scrub > > 10.4f 5215243 4561466 0 0 0 6099447266 > > 9605810 1930 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181794065 79875:396660094 [2,1,0]p2 [2,1]p2 > > 2024-06-02T13:35:21.451128+0000 2024-06-02T13:35:21.451128+0000 > > 7316 queued for scrub > > 10.50 5215219 5184003 0 0 0 6151789627 > > 9674779 1871 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181695151 79875:405060298 [0,2,1]p0 [1,2]p1 > > 2024-06-11T16:42:00.221013+0000 2024-06-11T16:42:00.221013+0000 > > 2385 periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000 > > 10.51 5218666 5218666 0 0 8388608 5774655330 > > 9331373 1920 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180711664 79875:342978462 [0,1,2]p0 [1,2]p1 > > 2024-06-11T14:31:51.402844+0000 2024-05-24T11:26:15.539677+0000 > > 730 periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000 > > 10.52 5218435 5187050 0 0 4194304 6375954134 > > 9843553 1935 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180834874 79875:341502004 [0,1,2]p0 [1,2]p1 > > 2024-06-11T08:41:43.586833+0000 2024-06-11T08:41:43.586833+0000 > > 8368 periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000 > > 10.53 5216822 5216822 0 0 10001995 5786896551 > > 9368323 1928 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181337168 79875:402047940 [0,2,1]p0 [1,2]p1 > > 2024-06-12T04:54:21.147742+0000 2024-05-23T23:02:36.154347+0000 > > 557 periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000 > > 10.54 5213949 4567626 0 0 8388608 6095478377 > > 9664814 1944 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182179986 79875:421118572 [2,1,0]p2 [2,1]p2 > > 2024-06-12T09:19:30.470319+0000 2024-06-01T17:56:32.089348+0000 > > 551 periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000 > > 10.55 5217389 5217389 0 0 4194948 5905845521 > > 9403344 1950 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'183481821 79875:341223864 [1,2,0]p1 [1,2]p1 > > 2024-06-11T15:31:05.524698+0000 2024-05-29T11:50:10.322128+0000 > > 584 periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000 > > 10.56 5219142 5219142 0 0 0 6311549101 > > 9773102 1897 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'184062267 79875:408792597 [1,2,0]p1 [1,2]p1 > > 2024-06-10T22:17:28.523159+0000 2024-06-10T22:17:28.523159+0000 > > 8211 periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000 > > 10.57 5219035 4894148 0 0 180 5862209193 > > 9396761 1895 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'181886155 79875:340148291 [0,1,2]p0 [1,2]p1 > > 2024-06-10T22:33:19.364284+0000 2024-05-28T12:05:26.762682+0000 > > 951 periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000 > > 10.58 5218318 5185905 0 0 9878587 5911568798 > > 9459707 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179109213 79875:390200207 [2,0,1]p2 [2,1]p2 > > 2024-06-11T13:08:37.380203+0000 2024-05-30T10:19:35.091370+0000 > > 578 periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000 > > 10.59 5218315 5185838 0 0 4194304 5727052522 > > 9214424 1875 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181700141 79875:400297310 [2,0,1]p2 [2,1]p2 > > 2024-06-11T11:15:00.221765+0000 2024-05-21T11:28:39.056382+0000 > > 682 periodic deep scrub scheduled @ > 2024-06-16T13:00:43.750185+0000 > > 10.5a 5218301 5218301 0 0 4194304 5833097049 > > 9365354 1942 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180699909 79875:393344045 [1,2,0]p1 [1,2]p1 > > 2024-06-12T01:50:51.954645+0000 2024-05-25T00:11:49.917556+0000 > > 497 periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000 > > 10.5b 5215134 5215134 0 0 4194802 6070043467 > > 9585134 1892 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181149500 79875:342539489 [2,0,1]p2 [2,1]p2 > > 2024-06-12T08:19:32.265112+0000 2024-06-02T11:33:23.094569+0000 > > 714 periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000 > > 10.5c 5221503 5221503 0 0 4194304 5989312430 > > 9524556 1939 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180748682 79875:340750417 [1,2,0]p1 [1,2]p1 > > 2024-06-11T09:37:59.483197+0000 2024-05-28T03:41:58.475567+0000 > > 1021 periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000 > > 10.5d 5222202 5222202 0 0 8388608 6031557641 > > 9520790 1928 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'179508607 79875:399919359 [1,0,2]p1 [1,2]p1 > > 2024-06-11T17:10:33.165278+0000 2024-05-30T05:53:10.772736+0000 > > 575 periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000 > > 10.5e 5212770 4506643 0 0 0 5830165136 > > 9388228 1938 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180002484 79875:391844220 [2,0,1]p2 [2,1]p2 > > 2024-06-11T22:32:49.026280+0000 2024-05-24T15:31:50.362016+0000 > > 523 periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000 > > 10.5f 5221504 5221504 0 0 4194394 5807975316 > > 9387383 1879 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181161493 79875:408530090 [0,2,1]p0 [1,2]p1 > > 2024-06-11T15:10:33.261745+0000 2024-05-25T06:37:01.629711+0000 > > 845 periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000 > > 10.60 5217784 5186615 0 0 4194304 6208553838 > > 9749044 1962 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181569072 79875:392211608 [0,1,2]p0 [1,2]p1 > > 2024-06-11T21:44:35.372861+0000 2024-06-11T21:44:35.372861+0000 > > 1505 periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000 > > 10.61 5221292 4586636 0 0 8388608 5766320671 > > 9280184 1910 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180945714 79875:344272511 [0,1,2]p0 [1,2]p1 > > 2024-06-11T16:51:25.519891+0000 2024-05-24T17:37:21.814566+0000 > > 566 periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000 > > 10.62 5218498 4699002 0 0 4194466 6027128219 > > 9555111 1884 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180387699 79875:341929273 [1,2,0]p1 [1,2]p1 > > 2024-06-12T03:10:08.547987+0000 2024-06-01T02:47:43.109035+0000 > > 509 periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000 > > 10.63 5219101 5219101 0 0 0 6040382110 > > 9500040 1885 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181623402 79875:389756489 [0,1,2]p0 [1,2]p1 > > 2024-06-02T23:50:14.947543+0000 2024-06-02T23:50:14.947543+0000 > > 13925 queued for scrub > > 10.64 5217025 5217025 0 0 2277792 6222337345 > > 9701950 1866 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181853168 79875:406174284 [1,2,0]p1 [1,2]p1 > > 2024-06-02T04:57:41.872723+0000 2024-06-02T04:57:41.872723+0000 > > 6394 queued for scrub > > 10.65 5217908 4590550 0 0 4194326 6223151288 > > 9709345 1916 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181196382 79875:340107600 [2,0,1]p2 [2,1]p2 > > 2024-06-11T15:53:28.725001+0000 2024-06-11T15:53:28.725001+0000 > > 1343 periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000 > > 10.66 5218911 4893654 0 0 4194304 5815703055 > > 9338987 1939 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'183062531 79875:395193073 [1,0,2]p1 [1,2]p1 > > 2024-06-12T01:10:56.919233+0000 2024-05-25T04:50:04.300293+0000 > > 590 periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000 > > 10.67 5217539 4587208 0 0 4194304 6095526883 > > 9640323 1955 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181875315 79875:342878178 [2,1,0]p2 [2,1]p2 > > 2024-06-12T07:58:41.522730+0000 2024-06-01T07:20:59.951966+0000 > > 553 periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000 > > 10.68 5215852 4505902 0 0 8388608 6093858586 > > 9596421 1926 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179490643 79875:379632203 [0,1,2]p0 [1,2]p1 > > 2024-06-11T16:02:14.995195+0000 2024-05-30T22:02:42.335910+0000 > > 526 periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000 > > 10.69 5216596 5216596 0 0 4194304 5833268044 > > 9400414 1899 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181876551 79875:387059449 [1,2,0]p1 [1,2]p1 > > 2024-06-11T17:00:57.111851+0000 2024-05-23T07:05:05.989427+0000 > > 572 periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000 > > 10.6a 5214106 4497370 0 0 0 5745504618 > > 9296938 1869 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180109669 79875:381871149 [2,1,0]p2 [2,1]p2 > > 2024-06-12T00:27:45.155477+0000 2024-05-25T09:22:19.570011+0000 > > 494 periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000 > > 10.6b 5219339 5219339 0 0 0 5801390847 > > 9370084 1957 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180041524 79875:342270608 [0,1,2]p0 [1,2]p1 > > 2024-06-12T03:01:39.358464+0000 2024-05-23T01:04:22.522035+0000 > > 488 periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000 > > 10.6c 5217917 5217917 0 0 554 6100979305 > > 9612250 1955 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180968918 79875:341488318 [1,2,0]p1 [1,2]p1 > > 2024-06-11T18:47:35.969345+0000 2024-05-31T11:54:59.559316+0000 > > 588 periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000 > > 10.6d 5221048 4550388 0 0 4194304 6024067862 > > 9621947 1927 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179373269 79875:388095690 [2,1,0]p2 [2,1]p2 > > 2024-06-12T03:26:37.931310+0000 2024-05-31T04:10:41.991457+0000 > > 502 periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000 > > 10.6e 5215550 4888179 0 0 8388608 6110450001 > > 9666892 1906 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'180885974 79875:380260263 [0,1,2]p0 [1,2]p1 > > 2024-06-11T20:18:23.657558+0000 2024-06-11T20:18:23.657558+0000 > > 1218 periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000 > > 10.6f 5215850 4566820 0 0 0 6107146788 > > 9634911 1923 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181752773 79875:396708021 [2,1,0]p2 [2,1]p2 > > 2024-06-10T17:32:51.735004+0000 2024-06-10T17:32:51.735004+0000 > > 10119 periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000 > > 10.70 5218211 4672319 0 0 4194304 5793441438 > > 9371563 1881 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180405012 79875:404578822 [1,2,0]p1 [1,2]p1 > > 2024-06-03T04:46:35.130293+0000 2024-05-22T07:50:20.898441+0000 > > 969 queued for deep scrub > > 10.71 5218299 4599312 0 0 4194304 5807950891 > > 9349720 1906 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180690682 79875:344108361 [0,2,1]p0 [1,2]p1 > > 2024-06-12T03:18:16.645927+0000 2024-05-23T09:32:56.055703+0000 > > 487 periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000 > > 10.72 5223470 4922347 0 0 8388608 6416468325 > > 9922777 1958 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'180797342 79875:342356916 [0,1,2]p0 [1,2]p1 > > 2024-06-12T05:42:47.545762+0000 2024-06-12T05:42:47.545762+0000 > > 1307 periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000 > > 10.73 5218301 4520651 0 0 0 6293832301 > > 9738943 1953 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182246986 79875:403792984 [2,0,1]p2 [2,1]p2 > > 2024-06-11T20:59:02.079157+0000 2024-06-11T20:59:02.079157+0000 > > 1399 periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000 > > 10.74 5219061 4595112 0 0 0 5974775781 > > 9469653 1943 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181947609 79875:420571221 [2,1,0]p2 [2,1]p2 > > 2024-06-03T02:17:19.687482+0000 2024-05-28T08:43:38.825954+0000 > > 927 queued for scrub > > 10.75 5220994 4596634 0 0 25165824 5822315216 > > 9362548 1877 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'180376305 79875:338847231 [2,0,1]p2 [2,1]p2 > > 2024-06-03T05:52:28.743255+0000 2024-05-24T05:11:00.211501+0000 > > 981 queued for scrub > > 10.76 5222111 4892295 0 0 0 6067998823 > > 9574254 1935 3000 > active+undersized+degraded+remapped+backfilling > > 2h 79875'183871180 79875:409218100 [1,0,2]p1 [1,2]p1 > > 2024-06-03T04:14:04.513004+0000 2024-06-03T04:14:04.513004+0000 > > 6987 queued for scrub > > 10.77 5219020 4574683 0 0 0 5889074277 > > 9460824 1896 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181614825 79875:342558162 [2,0,1]p2 [2,1]p2 > > 2024-06-12T08:07:38.169523+0000 2024-05-27T15:15:35.587972+0000 > > 536 periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000 > > 10.78 5220442 4693235 0 0 8388608 5879566566 > > 9395501 1923 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179031456 79875:391462415 [1,0,2]p1 [1,2]p1 > > 2024-06-02T09:32:42.834344+0000 2024-05-27T10:41:23.822124+0000 > > 994 queued for scrub > > 10.79 5219734 4702117 0 0 4194304 6129933545 > > 9646669 1926 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'182574416 79875:402249812 [1,2,0]p1 [1,2]p1 > > 2024-06-12T05:52:02.959814+0000 2024-06-01T00:53:49.548055+0000 > > 555 periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000 > > 10.7a 5215303 4526846 0 0 0 6316171107 > > 9817029 1932 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181765231 79875:395877057 [2,0,1]p2 [2,1]p2 > > 2024-06-12T02:23:42.692096+0000 2024-06-12T02:23:42.692096+0000 > > 1457 periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000 > > 10.7b 5217213 4606816 0 0 4194304 6226859283 > > 9718621 1952 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'181413541 79875:343968658 [1,2,0]p1 [1,2]p1 > > 2024-06-11T23:52:28.414752+0000 2024-06-11T23:52:28.414752+0000 > > 1308 periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000 > > 10.7c 5215075 5215075 0 0 4194304 6013877208 > > 9602345 1867 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'180991860 79875:341775615 [0,2,1]p0 [1,2]p1 > > 2024-06-11T23:30:40.614412+0000 2024-05-31T14:10:30.542603+0000 > > 503 periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000 > > 10.7d 5220700 4569720 0 0 5837609 6258736025 > > 9764836 1897 3000 > active+undersized+degraded+remapped+backfilling > > 3h 79875'179741885 79875:401448508 [2,1,0]p2 [2,1]p2 > > 2024-06-12T04:45:05.030905+0000 2024-06-12T04:45:05.030905+0000 > > 1565 periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000 > > 10.7e 5217113 5217113 0 0 16777216 6072081676 > > 9598390 1902 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'181040361 79875:391032348 [1,2,0]p1 [1,2]p1 > > 2024-06-11T17:37:26.589968+0000 2024-06-11T17:37:26.589968+0000 > > 1613 periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000 > > 10.7f 5218719 5218719 0 0 9909061 6130469539 > > 9657824 1943 3000 > active+undersized+degraded+remapped+backfill_wait > > 3h 79875'182283514 79875:408550888 [1,2,0]p1 [1,2]p1 > > 2024-06-11T13:05:03.552526+0000 2024-06-11T13:05:03.552526+0000 > > 1796 periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000 > > > > * NOTE: Omap statistics are gathered during deep scrub and may be > > inaccurate soon afterwards depending on utilization. See > > http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for > > further details. > > > > > > > > sudo ceph osd df > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > > AVAIL %USE VAR PGS STATUS > > 0 ssd 3.63869 1.00000 3.6 TiB 82 GiB 50 MiB 40 GiB 42 > GiB > > 3.6 TiB 2.21 0.03 0 up > > 1 ssd 3.63869 1.00000 3.6 TiB 2.0 TiB 16 GiB 1021 GiB 1.0 > TiB > > 1.6 TiB 55.41 0.75 161 up > > 2 ssd 3.63869 1.00000 3.6 TiB 3.0 TiB 16 GiB 1.1 TiB 1.9 > TiB > > 676 GiB 81.86 1.11 161 up > > > > > > sudo ceph df detail > > --- RAW STORAGE --- > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 1.2 PiB 308 TiB 871 TiB 871 TiB 73.86 > > ssd 11 TiB 5.8 TiB 5.1 TiB 5.1 TiB 46.51 > > TOTAL 1.2 PiB 314 TiB 876 TiB 876 TiB 73.61 > > > > --- POOLS --- > > POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) > > (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR > > UNDER COMPR > > .mgr 1 1 827 MiB 827 MiB 0 B 208 1.6 GiB 1.6 GiB > > 0 B 0.11 734 GiB N/A 50 GiB N/A 0 B > > 0 B > > metadata 10 128 1.0 TiB 647 MiB 1.0 TiB 667.90M 2.1 TiB 1.3 GiB > > 2.1 TiB 59.45 719 GiB N/A 3.5 TiB N/A 0 B > > 0 B > > data 11 2048 288 TiB 288 TiB 0 B 566.97M 863 TiB 863 TiB > > 0 B 81.10 67 TiB N/A N/A N/A 3.4 TiB > > 6.8 TiB > > ssd-data 18 32 14 GiB 14 GiB 0 B 265.17k 29 GiB 29 GiB > > 0 B 1.90 734 GiB N/A N/A N/A 0 B > > 0 B > > > > > > sudo ceph balancer status > > { > > "active": true, > > "last_optimize_duration": "0:00:00.000170", > > "last_optimize_started": "Wed Jun 12 13:14:49 2024", > > "mode": "upmap", > > "no_optimization_needed": true, > > "optimize_result": "Some objects (0.172816) are degraded; try again > > later", > > "plans": [] > > } > > > > > > > > [image: ariadne.ai Logo] Lars Köppel > > Developer > > Email: lars.koeppel@xxxxxxxxxx > > Phone: +49 6221 5993580 <+4962215993580> > > ariadne.ai (Germany) GmbH > > Häusserstraße 3, 69115 Heidelberg > > Amtsgericht Mannheim, HRB 744040 > > Geschäftsführer: Dr. Fabian Svara > > https://ariadne.ai > > > > > > On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri <anthony.datri@xxxxxxxxx> > > wrote: > > > >> If you have: > >> > >> * pg_num too low (defaults are too low) > >> * pg_num not a power of 2 > >> * pg_num != number of OSDs in the pool > >> * balancer not enabled > >> > >> any of those might result in imbalance. > >> > >> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote: > >> > > >> > I don't have any good explanation at this point. Can you share some > more > >> information like: > >> > > >> > ceph pg ls-by-pool <cephfs_metadata> > >> > ceph osd df (for the relevant OSDs) > >> > ceph df > >> > > >> > Thanks, > >> > Eugen > >> > > >> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> > > >> >> Since my last update the size of the largest OSD increased by 0.4 TiB > >> while > >> >> the smallest one only increased by 0.1 TiB. How is this possible? > >> >> > >> >> Because the metadata pool reported to have only 900MB space left, I > >> stopped > >> >> the hot-standby MDS. This gave me 8GB back but these filled up in the > >> last > >> >> 2h. > >> >> I think I have to zap the next OSD because the filesystem is getting > >> read > >> >> only... > >> >> > >> >> How is it possible that an OSD has over 1 TiB less data on it after a > >> >> rebuild? And how is it possible to have so different sizes of OSDs? > >> >> > >> >> > >> >> [image: ariadne.ai Logo] Lars Köppel > >> >> Developer > >> >> Email: lars.koeppel@xxxxxxxxxx > >> >> Phone: +49 6221 5993580 <+4962215993580> > >> >> ariadne.ai (Germany) GmbH > >> >> Häusserstraße 3, 69115 Heidelberg > >> >> Amtsgericht Mannheim, HRB 744040 > >> >> Geschäftsführer: Dr. Fabian Svara > >> >> https://ariadne.ai > >> >> > >> >> > >> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel <lars.koeppel@xxxxxxxxxx > > > >> wrote: > >> >> > >> >>> Only in warning mode. And there were no PG splits or merges in the > >> last 2 > >> >>> month. > >> >>> > >> >>> > >> >>> [image: ariadne.ai Logo] Lars Köppel > >> >>> Developer > >> >>> Email: lars.koeppel@xxxxxxxxxx > >> >>> Phone: +49 6221 5993580 <+4962215993580> > >> >>> ariadne.ai (Germany) GmbH > >> >>> Häusserstraße 3, 69115 Heidelberg > >> >>> Amtsgericht Mannheim, HRB 744040 > >> >>> Geschäftsführer: Dr. Fabian Svara > >> >>> https://ariadne.ai > >> >>> > >> >>> > >> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> wrote: > >> >>> > >> >>>> I don't think scrubs can cause this. Do you have autoscaler > enabled? > >> >>>> > >> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> >>>> > >> >>>> > Hi, > >> >>>> > > >> >>>> > thank you for your response. > >> >>>> > > >> >>>> > I don't think this thread covers my problem, because the OSDs for > >> the > >> >>>> > metadata pool fill up at different rates. So I would think this > is > >> no > >> >>>> > direct problem with the journal. > >> >>>> > Because we had earlier problems with the journal I changed some > >> >>>> > settings(see below). I already restarted all MDS multiple times > but > >> no > >> >>>> > change here. > >> >>>> > > >> >>>> > The health warnings regarding cache pressure resolve normally > after > >> a > >> >>>> > short period of time, when the heavy load on the client ends. > >> Sometimes > >> >>>> it > >> >>>> > stays a bit longer because an rsync is running and copying data > on > >> the > >> >>>> > cluster(rsync is not good at releasing the caps). > >> >>>> > > >> >>>> > Could it be a problem if scrubs run most of the time in the > >> background? > >> >>>> Can > >> >>>> > this block any other tasks or generate new data itself? > >> >>>> > > >> >>>> > Best regards, > >> >>>> > Lars > >> >>>> > > >> >>>> > > >> >>>> > global basic mds_cache_memory_limit > >> >>>> > 17179869184 > >> >>>> > global advanced mds_max_caps_per_client > >> >>>> > 16384 > >> >>>> > global advanced > >> >>>> mds_recall_global_max_decay_threshold > >> >>>> > 262144 > >> >>>> > global advanced mds_recall_max_decay_rate > >> >>>> > 1.000000 > >> >>>> > global advanced > mds_recall_max_decay_threshold > >> >>>> > 262144 > >> >>>> > mds advanced mds_cache_trim_threshold > >> >>>> > 131072 > >> >>>> > mds advanced mds_heartbeat_grace > >> >>>> > 120.000000 > >> >>>> > mds advanced mds_heartbeat_reset_grace > >> >>>> > 7400 > >> >>>> > mds advanced mds_tick_interval > >> >>>> > 3.000000 > >> >>>> > > >> >>>> > > >> >>>> > [image: ariadne.ai Logo] Lars Köppel > >> >>>> > Developer > >> >>>> > Email: lars.koeppel@xxxxxxxxxx > >> >>>> > Phone: +49 6221 5993580 <+4962215993580> > >> >>>> > ariadne.ai (Germany) GmbH > >> >>>> > Häusserstraße 3, 69115 Heidelberg > >> >>>> > Amtsgericht Mannheim, HRB 744040 > >> >>>> > Geschäftsführer: Dr. Fabian Svara > >> >>>> > https://ariadne.ai > >> >>>> > > >> >>>> > > >> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx> > wrote: > >> >>>> > > >> >>>> >> Hi, > >> >>>> >> > >> >>>> >> can you check if this thread [1] applies to your situation? You > >> don't > >> >>>> >> have multi-active MDS enabled, but maybe it's still some journal > >> >>>> >> trimming, or maybe misbehaving clients? In your first post there > >> were > >> >>>> >> health warnings regarding cache pressure and cache size. Are > those > >> >>>> >> resolved? > >> >>>> >> > >> >>>> >> [1] > >> >>>> >> > >> >>>> >> > >> >>>> > >> > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE > >> >>>> >> > >> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> >>>> >> > >> >>>> >> > Hello everyone, > >> >>>> >> > > >> >>>> >> > short update to this problem. > >> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the expected > >> size > >> >>>> >> ~50%). > >> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They > >> jumped up > >> >>>> and > >> >>>> >> > down a lot but the higher one has now also reached 'nearfull' > >> >>>> status. How > >> >>>> >> > is this possible? What is going on? > >> >>>> >> > > >> >>>> >> > Does anyone have a solution how to fix this without zapping > the > >> OSD? > >> >>>> >> > > >> >>>> >> > Best regards, > >> >>>> >> > Lars > >> >>>> >> > > >> >>>> >> > > >> >>>> >> > [image: ariadne.ai Logo] Lars Köppel > >> >>>> >> > Developer > >> >>>> >> > Email: lars.koeppel@xxxxxxxxxx > >> >>>> >> > Phone: +49 6221 5993580 <+4962215993580> > >> >>>> >> > ariadne.ai (Germany) GmbH > >> >>>> >> > Häusserstraße 3, 69115 Heidelberg > >> >>>> >> > Amtsgericht Mannheim, HRB 744040 > >> >>>> >> > Geschäftsführer: Dr. Fabian Svara > >> >>>> >> > https://ariadne.ai > >> >>>> >> > _______________________________________________ > >> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> >>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >>>> >> > >> >>>> >> > >> >>>> >> _______________________________________________ > >> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx > >> >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >>>> >> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> > > >> > > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > >> > > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |