Hello everyone, short update. We added 3 SSD OSDs to the cluster to have some more space and more IO. After all objects were in their new place, it took ~1 week that the OSDs reduced their size. The size of the metadata pool did not change in this time but the OSDs removed multiple TB of occupied space. Now everything looks fine. Let's hope this will stay like this. Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@xxxxxxxxxx Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Wed, Jul 3, 2024 at 2:06 PM Lars Köppel <lars.koeppel@xxxxxxxxxx> wrote: > Hello everyone, > > we reduced the load slightly for the last 2 weeks and the fast growth > stopped. > But after a short burst of heavy load 1 OSD had filled up to the max > again. So I had to zapp it. At the moment it is rebuilding the drive. > > After ejecting all clients and reducing the load, the growth is minimal. > We ordered some parts to add some more SSDs to this pool. I will update > when they have arrived and the balancing is done. > > Another thing I have observed is that the limits for the scrubs are not > respected. > We only have 75 OSDs and set osd_max_scrubs to 1 but there are often more > than 100 scrubs ongoing. Has there been anything changed since 17.2.6? > > Best regards, > Lars > > [image: ariadne.ai Logo] Lars Köppel > Developer > Email: lars.koeppel@xxxxxxxxxx > Phone: +49 6221 5993580 <+4962215993580> > ariadne.ai (Germany) GmbH > Häusserstraße 3, 69115 Heidelberg > Amtsgericht Mannheim, HRB 744040 > Geschäftsführer: Dr. Fabian Svara > https://ariadne.ai > > > On Thu, Jun 13, 2024 at 3:24 PM Lars Köppel <lars.koeppel@xxxxxxxxxx> > wrote: > >> ok. Thank you for your help. >> We will try this and report back in a few days. >> >> >> [image: ariadne.ai Logo] Lars Köppel >> Developer >> Email: lars.koeppel@xxxxxxxxxx >> Phone: +49 6221 5993580 <+4962215993580> >> ariadne.ai (Germany) GmbH >> Häusserstraße 3, 69115 Heidelberg >> Amtsgericht Mannheim, HRB 744040 >> Geschäftsführer: Dr. Fabian Svara >> https://ariadne.ai >> >> >> On Thu, Jun 13, 2024 at 3:01 PM Eugen Block <eblock@xxxxxx> wrote: >> >>> I'm quite sure that this could result in the impact you're seeing. To >>> confirm that suspicion you could stop deleting and wait a couple of >>> days to see if the usage stabilizes. And if it does, maybe delete less >>> files at once or so to see how far you can tweak it. That would be my >>> approach. >>> >>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: >>> >>> > We have been using snapshots for a long time. >>> > The only change in usage is that we are currently deleting many small >>> files >>> > from the system. Because this is slow (~150 requests/s) this is >>> running for >>> > the last few weeks. Could such a load result in a problem with the MDS? >>> > >>> > I have to ask for permission to order more drives. This could take some >>> > time. >>> > >>> > [image: ariadne.ai Logo] Lars Köppel >>> > Developer >>> > Email: lars.koeppel@xxxxxxxxxx >>> > Phone: +49 6221 5993580 <+4962215993580> >>> > ariadne.ai (Germany) GmbH >>> > Häusserstraße 3, 69115 Heidelberg >>> > Amtsgericht Mannheim, HRB 744040 >>> > Geschäftsführer: Dr. Fabian Svara >>> > https://ariadne.ai >>> > >>> > >>> > On Thu, Jun 13, 2024 at 12:55 PM Eugen Block <eblock@xxxxxx> wrote: >>> > >>> >> Downgrading isn't supported, I don't think that would be a good idea. >>> >> I also don't see anything obvious standing out in the pg output. Any >>> >> chance you can add more OSDs to the metadata pool to see if it stops >>> >> at some point? Did the cluster usage change in any way? For example >>> >> cephfs snapshots which haven't been used before or something like >>> that? >>> >> >>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: >>> >> >>> >> > I updated from 17.2.6 to 17.2.7 and a few hours later to 18.2.2. >>> >> > Would it be an option to go back to 17.2.6? >>> >> > >>> >> > >>> >> > [image: ariadne.ai Logo] Lars Köppel >>> >> > Developer >>> >> > Email: lars.koeppel@xxxxxxxxxx >>> >> > Phone: +49 6221 5993580 <+4962215993580> >>> >> > ariadne.ai (Germany) GmbH >>> >> > Häusserstraße 3, 69115 Heidelberg >>> >> > Amtsgericht Mannheim, HRB 744040 >>> >> > Geschäftsführer: Dr. Fabian Svara >>> >> > https://ariadne.ai >>> >> > >>> >> > >>> >> > On Wed, Jun 12, 2024 at 5:30 PM Eugen Block <eblock@xxxxxx> wrote: >>> >> > >>> >> >> Which version did you upgrade from to 18.2.2? >>> >> >> I can’t pin it down to a specific issue, but somewhere in the back >>> of >>> >> >> my mind is something related to a new omap format or something. But >>> >> >> I’m really not sure at all. >>> >> >> >>> >> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: >>> >> >> >>> >> >> > I am happy to help you with as much information as possible. I >>> >> probably >>> >> >> > just don't know where to look for it. >>> >> >> > Below are the requested information. The cluster is rebuilding >>> the >>> >> >> > zapped OSD at the moment. This will probably take the next few >>> days. >>> >> >> > >>> >> >> > >>> >> >> > sudo ceph pg ls-by-pool metadata >>> >> >> > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES >>> OMAP_BYTES* >>> >> >> > OMAP_KEYS* LOG LOG_DUPS STATE >>> >> >> > SINCE VERSION REPORTED UP ACTING >>> >> >> > SCRUB_STAMP DEEP_SCRUB_STAMP >>> >> >> > LAST_SCRUB_DURATION SCRUB_SCHEDULING >>> >> >> > 10.0 5217325 4994695 0 0 4194304 >>> 5880891340 >>> >> >> > 9393865 1885 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'180849582 79875:391519635 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T09:08:09.829362+0000 2024-05-28T05:52:59.321589+0000 >>> >> >> > 627 periodic scrub scheduled @ >>> 2024-06-17T08:21:31.808348+0000 >>> >> >> > 10.1 5214785 5193424 0 0 0 >>> 5843682713 >>> >> >> > 9410150 1912 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180914288 79875:342746928 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-01T15:56:28.927288+0000 2024-05-27T03:31:37.682966+0000 >>> >> >> > 966 queued for scrub >>> >> >> > 10.2 5218432 5187168 0 0 0 >>> 6402011266 >>> >> >> > 9812513 1874 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180970531 79875:341340204 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T13:40:58.994256+0000 2024-06-11T13:40:58.994256+0000 >>> >> >> > 1942 periodic scrub scheduled @ >>> 2024-06-17T06:07:15.329675+0000 >>> >> >> > 10.3 5217413 5217413 0 0 8388788 >>> 5766005023 >>> >> >> > 9271787 1923 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181012233 79875:388295881 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T00:35:56.965547+0000 2024-05-23T19:54:56.121729+0000 >>> >> >> > 492 periodic scrub scheduled @ >>> 2024-06-18T06:39:31.103864+0000 >>> >> >> > 10.4 5220069 5220069 0 0 12583466 >>> 6027548724 >>> >> >> > 9537290 1959 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181576075 79875:405295868 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T17:47:22.923514+0000 2024-05-31T02:06:55.339574+0000 >>> >> >> > 581 periodic scrub scheduled @ >>> 2024-06-17T00:59:37.214420+0000 >>> >> >> > 10.5 5216162 5211999 0 0 4194304 >>> 5941347251 >>> >> >> > 9542764 1930 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180455793 79875:338418517 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-11T22:50:16.170708+0000 2024-05-30T23:49:54.316379+0000 >>> >> >> > 528 periodic scrub scheduled @ >>> 2024-06-17T04:39:25.905185+0000 >>> >> >> > 10.6 5216100 4980459 0 0 4521984 >>> 6428088514 >>> >> >> > 9850762 1911 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'184045876 79875:396809795 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T22:24:05.102716+0000 2024-06-11T22:24:05.102716+0000 >>> >> >> > 1082 periodic scrub scheduled @ >>> 2024-06-17T07:58:44.289885+0000 >>> >> >> > 10.7 5218232 5218232 0 0 4194304 >>> 6377065363 >>> >> >> > 9849360 1919 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182672562 79875:342449062 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-11T06:22:15.689422+0000 2024-06-11T06:22:15.689422+0000 >>> >> >> > 8225 periodic scrub scheduled @ >>> 2024-06-17T13:05:59.225052+0000 >>> >> >> > 10.8 5219620 5182816 0 0 0 >>> 6167304290 >>> >> >> > 9691796 1896 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179628377 79875:378022884 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-11T22:06:01.386763+0000 2024-06-11T22:06:01.386763+0000 >>> >> >> > 1286 periodic scrub scheduled @ >>> 2024-06-17T07:54:54.133093+0000 >>> >> >> > 10.9 5219448 5164591 0 0 8388698 >>> 5796048346 >>> >> >> > 9338312 1868 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181739392 79875:387412389 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T05:21:00.586747+0000 2024-05-26T11:10:59.780673+0000 >>> >> >> > 539 periodic scrub scheduled @ >>> 2024-06-18T15:32:59.155092+0000 >>> >> >> > 10.a 5219861 5163635 0 0 12582912 >>> 5841839055 >>> >> >> > 9387200 1916 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180205688 79875:379381294 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T12:35:05.571200+0000 2024-05-22T11:07:16.041773+0000 >>> >> >> > 1093 periodic deep scrub scheduled @ >>> >> >> 2024-06-17T05:21:40.136463+0000 >>> >> >> > 10.b 5217949 5217949 0 0 16777216 >>> 5935863260 >>> >> >> > 9462127 1881 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181655745 79875:343806807 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T22:41:28.976920+0000 2024-05-26T08:43:29.217457+0000 >>> >> >> > 520 periodic scrub scheduled @ >>> 2024-06-17T17:44:32.764093+0000 >>> >> >> > 10.c 5221697 5217118 0 0 4194304 >>> 6015217841 >>> >> >> > 9574445 1928 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180892826 79875:341490398 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-11T09:20:58.443473+0000 2024-05-30T00:13:50.306507+0000 >>> >> >> > 768 periodic scrub scheduled @ >>> 2024-06-16T19:41:21.977436+0000 >>> >> >> > 10.d 5217727 4908764 0 0 0 >>> 5825598519 >>> >> >> > 9349877 1930 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'179050347 79875:387455993 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-02T07:27:27.873631+0000 2024-05-20T03:58:30.225170+0000 >>> >> >> > 952 queued for deep scrub >>> >> >> > 10.e 5215040 5210572 0 0 4194304 >>> 5790634469 >>> >> >> > 9327651 1905 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180049044 79875:377196119 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T05:03:08.521423+0000 2024-05-22T13:47:46.753131+0000 >>> >> >> > 527 periodic scrub scheduled @ >>> 2024-06-17T17:30:36.079456+0000 >>> >> >> > 10.f 5217202 4885952 0 0 8388630 >>> 6005274626 >>> >> >> > 9589178 1917 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'181953069 79875:396402453 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T12:20:49.661741+0000 2024-06-11T12:20:49.661741+0000 >>> >> >> > 3949 periodic scrub scheduled @ >>> 2024-06-17T18:37:10.974112+0000 >>> >> >> > 10.10 5214985 5157391 0 0 4194394 >>> 6275024632 >>> >> >> > 9721505 1922 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181801708 79875:395055895 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T01:49:47.831191+0000 2024-06-11T01:49:47.831191+0000 >>> >> >> > 9579 periodic scrub scheduled @ >>> 2024-06-16T11:40:28.305905+0000 >>> >> >> > 10.11 5219236 4983134 0 0 4194304 >>> 5787580073 >>> >> >> > 9316893 1901 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'181011816 79875:343488767 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-03T00:06:14.862209+0000 2024-05-27T00:05:56.468361+0000 >>> >> >> > 960 queued for scrub >>> >> >> > 10.12 5214103 5182770 0 0 90 >>> 5970750188 >>> >> >> > 9505613 1891 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180210613 79875:341042616 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-12T06:39:48.252419+0000 2024-05-28T01:32:03.381577+0000 >>> >> >> > 527 periodic scrub scheduled @ >>> 2024-06-18T11:23:33.994438+0000 >>> >> >> > 10.13 5214580 5214580 0 0 8388642 >>> 5793432968 >>> >> >> > 9335940 1957 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182394614 79875:393423214 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-10T17:52:04.690203+0000 2024-05-22T03:39:02.453336+0000 >>> >> >> > 1151 periodic deep scrub scheduled @ >>> >> >> 2024-06-16T00:28:37.682765+0000 >>> >> >> > 10.14 5218591 5218591 0 0 5053325 >>> 6046268958 >>> >> >> > 9582481 1915 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'184576050 79875:425213800 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-03T05:36:07.656965+0000 2024-05-28T22:46:16.814318+0000 >>> >> >> > 985 queued for scrub >>> >> >> > 10.15 5215919 4907184 0 0 11939335 >>> 5752284246 >>> >> >> > 9285870 1889 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'179713036 79875:337938973 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T14:06:50.138992+0000 2024-05-24T07:52:22.522455+0000 >>> >> >> > 786 periodic scrub scheduled @ >>> 2024-06-17T13:00:48.869790+0000 >>> >> >> > 10.16 5221091 5221091 0 0 8388608 >>> 5966308565 >>> >> >> > 9487816 1916 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'183445984 79875:399413009 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-03T02:01:53.649417+0000 2024-06-03T02:01:53.649417+0000 >>> >> >> > 6935 queued for scrub >>> >> >> > 10.17 5218069 5194732 0 0 4194304 >>> 6231638001 >>> >> >> > 9735915 1890 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182390128 79875:340719915 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-12T06:31:01.523916+0000 2024-06-12T06:31:01.523916+0000 >>> >> >> > 1272 periodic scrub scheduled @ >>> 2024-06-17T16:33:45.723673+0000 >>> >> >> > 10.18 5214253 5159897 0 0 0 >>> 6128506469 >>> >> >> > 9673261 1886 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179754208 79875:389150273 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T02:53:31.027154+0000 2024-06-12T02:53:31.027154+0000 >>> >> >> > 1308 periodic scrub scheduled @ >>> 2024-06-17T22:19:41.359070+0000 >>> >> >> > 10.19 5220131 5183342 0 0 0 >>> 5869775360 >>> >> >> > 9460583 1953 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181750923 79875:393162989 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T19:21:24.106504+0000 2024-05-22T15:42:01.721091+0000 >>> >> >> > 522 periodic scrub scheduled @ >>> 2024-06-17T20:09:55.173174+0000 >>> >> >> > 10.1a 5217968 5186700 0 0 4194304 >>> 6183652625 >>> >> >> > 9733847 1896 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181056240 79875:394241128 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-02T15:37:49.730965+0000 2024-06-02T15:37:49.730965+0000 >>> >> >> > 7347 queued for scrub >>> >> >> > 10.1b 5221259 5161879 0 0 0 >>> 5989030303 >>> >> >> > 9561035 1875 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181052305 79875:343144823 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-12T06:01:07.213894+0000 2024-06-02T00:23:51.063901+0000 >>> >> >> > 543 periodic scrub scheduled @ >>> 2024-06-17T15:51:21.700275+0000 >>> >> >> > 10.1c 5217034 5217034 0 0 554 >>> 5979861913 >>> >> >> > 9504442 1875 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180913313 79875:341328275 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-01T20:02:47.025129+0000 2024-06-01T20:02:47.025129+0000 >>> >> >> > 6568 queued for scrub >>> >> >> > 10.1d 5219691 4997027 0 0 12582912 >>> 6081321658 >>> >> >> > 9599164 1918 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'179638837 79875:391808225 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T20:27:02.506715+0000 2024-05-31T22:58:06.578167+0000 >>> >> >> > 517 periodic scrub scheduled @ >>> 2024-06-16T21:05:59.442200+0000 >>> >> >> > 10.1e 5213369 5208776 0 0 4194304 >>> 6295629499 >>> >> >> > 9765317 1941 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181145210 79875:391326487 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T07:49:28.432084+0000 2024-06-12T07:49:28.432084+0000 >>> >> >> > 1990 periodic scrub scheduled @ >>> 2024-06-18T01:04:07.677468+0000 >>> >> >> > 10.1f 5218399 5000094 0 0 8388608 >>> 5821063844 >>> >> >> > 9421222 1896 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'180960296 79875:400635456 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T13:53:40.278050+0000 2024-05-22T19:39:02.345663+0000 >>> >> >> > 761 periodic scrub scheduled @ >>> 2024-06-16T22:24:39.094932+0000 >>> >> >> > 10.20 5216437 5185379 0 0 0 >>> 6280994122 >>> >> >> > 9719574 1945 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181610117 79875:391874431 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T12:58:57.754648+0000 2024-06-11T12:58:57.754648+0000 >>> >> >> > 2288 periodic scrub scheduled @ >>> 2024-06-17T23:33:35.937406+0000 >>> >> >> > 10.21 5217349 5217349 0 0 8388698 >>> 6348931429 >>> >> >> > 9822197 1867 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181724357 79875:343883762 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T18:28:34.044546+0000 2024-06-11T18:28:34.044546+0000 >>> >> >> > 1817 periodic scrub scheduled @ >>> 2024-06-17T12:18:47.312213+0000 >>> >> >> > 10.22 5215544 4943475 0 0 4194304 >>> 5982815232 >>> >> >> > 9482159 1899 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'180230331 79875:340839853 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T15:21:21.994674+0000 2024-05-30T08:01:03.301650+0000 >>> >> >> > 649 periodic scrub scheduled @ >>> 2024-06-17T01:33:20.228420+0000 >>> >> >> > 10.23 5217758 5217758 0 0 146 >>> 5804086623 >>> >> >> > 9336298 1920 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180948432 79875:388158699 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-03T05:19:32.311846+0000 2024-05-27T08:13:15.250058+0000 >>> >> >> > 971 queued for scrub >>> >> >> > 10.24 5221505 5221505 0 0 0 >>> 6360995644 >>> >> >> > 9816333 1925 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182198758 79875:406847098 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-12T04:00:44.315682+0000 2024-06-12T04:00:44.315682+0000 >>> >> >> > 2046 periodic scrub scheduled @ >>> 2024-06-18T01:00:22.708069+0000 >>> >> >> > 10.25 5214911 5214911 0 0 12583002 >>> 5726320657 >>> >> >> > 9264352 1910 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180132024 79875:338167995 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T14:43:47.076906+0000 2024-05-23T12:16:34.271856+0000 >>> >> >> > 715 periodic scrub scheduled @ >>> 2024-06-17T14:42:04.951046+0000 >>> >> >> > 10.26 5217588 4973667 0 0 0 >>> 6040775867 >>> >> >> > 9598406 1927 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'188144536 79875:400277134 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-02T06:55:09.016528+0000 2024-06-02T06:55:09.016528+0000 >>> >> >> > 7047 queued for scrub >>> >> >> > 10.27 5218490 5218490 0 0 8388630 >>> 6265942186 >>> >> >> > 9818746 1941 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182400328 79875:342052414 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T01:01:06.674323+0000 2024-06-12T01:01:06.674323+0000 >>> >> >> > 1500 periodic scrub scheduled @ >>> 2024-06-18T09:04:18.289464+0000 >>> >> >> > 10.28 5221153 5221153 0 0 0 >>> 6049095371 >>> >> >> > 9597182 1913 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179438777 79875:378265375 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T04:09:58.896192+0000 2024-06-01T04:32:46.011974+0000 >>> >> >> > 553 periodic scrub scheduled @ >>> 2024-06-18T00:53:21.846375+0000 >>> >> >> > 10.29 5214066 5214066 0 0 4194304 >>> 5767555623 >>> >> >> > 9316899 1937 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181568997 79875:387683159 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-10T22:48:57.757493+0000 2024-05-21T07:14:53.348341+0000 >>> >> >> > 938 periodic deep scrub scheduled @ >>> >> >> 2024-06-16T06:24:55.108695+0000 >>> >> >> > 10.2a 5216314 4977811 0 0 4194304 >>> 6012567810 >>> >> >> > 9563804 1951 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'181140734 79875:381017643 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-11T09:57:29.097708+0000 2024-05-30T03:47:22.798044+0000 >>> >> >> > 1169 periodic scrub scheduled @ >>> 2024-06-16T10:08:48.053531+0000 >>> >> >> > 10.2b 5219620 5219620 0 0 8388608 >>> 5829917912 >>> >> >> > 9412426 1889 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180465782 79875:342263856 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T08:30:08.941782+0000 2024-05-24T22:00:29.535205+0000 >>> >> >> > 636 periodic scrub scheduled @ >>> 2024-06-17T09:00:09.155675+0000 >>> >> >> > 10.2c 5214087 5214087 0 0 0 >>> 5874182922 >>> >> >> > 9375708 1871 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180616041 79875:341244736 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-10T23:10:08.184410+0000 2024-05-27T17:41:07.583255+0000 >>> >> >> > 1268 periodic scrub scheduled @ >>> 2024-06-16T00:28:52.710385+0000 >>> >> >> > 10.2d 5213968 4984822 0 0 4337824 >>> 5835754026 >>> >> >> > 9417033 1947 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'178824607 79875:386427308 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-12T09:27:43.774331+0000 2024-05-27T05:33:45.293266+0000 >>> >> >> > 494 periodic scrub scheduled @ >>> 2024-06-18T10:19:18.219366+0000 >>> >> >> > 10.2e 5216263 4974456 0 0 12582912 >>> 6025714323 >>> >> >> > 9553880 1940 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'180639631 79875:378284872 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T01:19:42.477063+0000 2024-05-31T08:42:41.293335+0000 >>> >> >> > 526 periodic scrub scheduled @ >>> 2024-06-17T09:53:24.981953+0000 >>> >> >> > 10.2f 5220057 5220057 0 0 13839831 >>> 5804027574 >>> >> >> > 9370862 1882 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180961915 79875:394513378 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-11T19:58:05.145147+0000 2024-05-24T20:18:32.301831+0000 >>> >> >> > 535 periodic scrub scheduled @ >>> 2024-06-18T03:16:19.773648+0000 >>> >> >> > 10.30 5217090 4955469 0 0 12582934 >>> 6438521431 >>> >> >> > 9924825 1869 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 2h 79875'181488488 79875:404940886 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T07:16:18.095439+0000 2024-06-12T07:16:18.095439+0000 >>> >> >> > 2190 periodic scrub scheduled @ >>> 2024-06-18T10:42:55.574570+0000 >>> >> >> > 10.31 5218176 5218176 0 0 4194304 >>> 6215016361 >>> >> >> > 9752698 1897 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181622879 79875:343554151 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T09:00:15.941065+0000 2024-06-12T09:00:15.941065+0000 >>> >> >> > 1806 periodic scrub scheduled @ >>> 2024-06-18T02:46:00.756777+0000 >>> >> >> > 10.32 5223171 4704732 0 0 8388608 >>> 6205669256 >>> >> >> > 9800320 1920 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180911263 79875:342754128 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T19:49:09.010583+0000 2024-06-11T19:49:09.010583+0000 >>> >> >> > 1664 periodic scrub scheduled @ >>> 2024-06-17T07:07:21.062789+0000 >>> >> >> > 10.33 5215341 4525338 0 0 4194304 >>> 6066965588 >>> >> >> > 9635403 1888 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181467497 79875:402958597 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T01:59:23.933458+0000 2024-05-31T17:01:38.620377+0000 >>> >> >> > 512 periodic scrub scheduled @ >>> 2024-06-17T22:07:45.026120+0000 >>> >> >> > 10.34 5217342 5217342 0 0 8388608 >>> 5999920961 >>> >> >> > 9570527 1956 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182013407 79875:419219483 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T09:10:17.390128+0000 2024-06-01T15:40:23.241769+0000 >>> >> >> > 601 periodic scrub scheduled @ >>> 2024-06-18T09:56:03.061859+0000 >>> >> >> > 10.35 5222114 5222114 0 0 8388608 >>> 6021420075 >>> >> >> > 9571300 1952 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180948342 79875:338598678 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T22:59:17.017317+0000 2024-05-30T15:59:15.553266+0000 >>> >> >> > 539 periodic scrub scheduled @ >>> 2024-06-17T02:41:31.718321+0000 >>> >> >> > 10.36 5223657 5223657 0 0 4331271 >>> 5844578100 >>> >> >> > 9347405 1903 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'183363403 79875:408484683 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T18:37:46.653097+0000 2024-05-25T14:13:51.648712+0000 >>> >> >> > 553 periodic scrub scheduled @ >>> 2024-06-17T12:58:00.134657+0000 >>> >> >> > 10.37 5219248 5219248 0 0 4194304 >>> 6072826085 >>> >> >> > 9579049 1936 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182091890 79875:341545087 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-11T14:19:40.744743+0000 2024-05-29T18:40:12.602405+0000 >>> >> >> > 771 periodic scrub scheduled @ >>> 2024-06-16T18:24:26.104867+0000 >>> >> >> > 10.38 5214022 5214022 0 0 4194858 >>> 5780805541 >>> >> >> > 9299038 1951 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'178809936 79875:390113817 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-02T00:55:22.046433+0000 2024-05-25T16:49:10.279003+0000 >>> >> >> > 931 queued for scrub >>> >> >> > 10.39 5220692 4542094 0 0 0 >>> 6401799077 >>> >> >> > 9864164 1872 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'182881827 79875:402437400 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-12T01:42:35.239155+0000 2024-06-12T01:42:35.239155+0000 >>> >> >> > 1370 periodic scrub scheduled @ >>> 2024-06-17T10:51:19.580502+0000 >>> >> >> > 10.3a 5215648 4552651 0 0 0 >>> 6194465524 >>> >> >> > 9713558 1898 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181458295 79875:394916652 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-11T21:19:29.020889+0000 2024-06-11T21:19:29.020889+0000 >>> >> >> > 1227 periodic scrub scheduled @ >>> 2024-06-17T07:19:46.503819+0000 >>> >> >> > 10.3b 5218220 5218220 0 0 8388608 >>> 5951035110 >>> >> >> > 9490698 1878 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180815984 79875:342438023 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-03T05:03:19.483049+0000 2024-05-28T18:50:55.369009+0000 >>> >> >> > 1004 queued for scrub >>> >> >> > 10.3c 5221615 5221615 0 0 12582912 >>> 6260894058 >>> >> >> > 9727358 1938 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181561627 79875:342153262 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T19:12:06.206008+0000 2024-06-11T19:12:06.206008+0000 >>> >> >> > 1471 periodic scrub scheduled @ >>> 2024-06-17T02:09:19.786893+0000 >>> >> >> > 10.3d 5213771 5213771 0 0 0 >>> 6215975311 >>> >> >> > 9699615 1946 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179354493 79875:399858588 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T04:05:10.877703+0000 2024-06-11T04:05:10.877703+0000 >>> >> >> > 8123 periodic scrub scheduled @ >>> 2024-06-17T08:03:22.770035+0000 >>> >> >> > 10.3e 5219446 4659577 0 0 4194304 >>> 5852588566 >>> >> >> > 9370152 1933 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180035882 79875:391595513 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T20:35:43.624903+0000 2024-05-22T21:40:55.606650+0000 >>> >> >> > 520 periodic scrub scheduled @ >>> 2024-06-17T00:54:06.347642+0000 >>> >> >> > 10.3f 5219729 4889443 0 0 16777216 >>> 5925457842 >>> >> >> > 9484458 1948 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'181330738 79875:409113809 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-12T05:12:01.852109+0000 2024-05-27T20:03:22.704122+0000 >>> >> >> > 533 periodic scrub scheduled @ >>> 2024-06-17T13:07:19.370647+0000 >>> >> >> > 10.40 5222241 4445164 0 0 0 >>> 6312545191 >>> >> >> > 9773697 1873 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181341418 79875:393186178 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-10T20:00:37.561045+0000 2024-06-10T20:00:37.561045+0000 >>> >> >> > 7712 periodic scrub scheduled @ >>> 2024-06-16T05:59:45.317316+0000 >>> >> >> > 10.41 5215195 5183508 0 0 4194326 >>> 6000770355 >>> >> >> > 9529537 1938 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181494334 79875:343712275 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-12T04:18:58.693597+0000 2024-06-01T11:05:59.370690+0000 >>> >> >> > 539 periodic scrub scheduled @ >>> 2024-06-17T15:34:34.997281+0000 >>> >> >> > 10.42 5219564 4681323 0 0 12582912 >>> 5822745109 >>> >> >> > 9331625 1916 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'179811520 79875:340958258 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-02T19:58:06.439770+0000 2024-05-21T18:28:28.794738+0000 >>> >> >> > 11044 queued for deep scrub >>> >> >> > 10.43 5219412 4599458 0 0 594 >>> 5737440613 >>> >> >> > 9239869 1950 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180634536 79875:388879991 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-01T18:13:17.068416+0000 2024-05-20T16:32:20.302230+0000 >>> >> >> > 1002 queued for deep scrub >>> >> >> > 10.44 5216008 5216008 0 0 4194304 >>> 6270195298 >>> >> >> > 9786806 1944 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182088681 79875:406175480 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T00:19:01.240405+0000 2024-06-12T00:19:01.240405+0000 >>> >> >> > 1593 periodic scrub scheduled @ >>> 2024-06-17T12:19:53.888923+0000 >>> >> >> > 10.45 5216238 4694313 0 0 0 >>> 5714807184 >>> >> >> > 9223666 1944 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'179709878 79875:338307727 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-02T00:39:50.657116+0000 2024-05-21T02:52:14.376959+0000 >>> >> >> > 958 queued for deep scrub >>> >> >> > 10.46 5217950 5217950 0 0 9729074 >>> 5945415128 >>> >> >> > 9499020 1915 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'183726716 79875:395024265 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T14:56:28.875949+0000 2024-05-29T04:19:53.409918+0000 >>> >> >> > 724 periodic scrub scheduled @ >>> 2024-06-16T17:42:09.428674+0000 >>> >> >> > 10.47 5219125 5219125 0 0 4194304 >>> 5775128028 >>> >> >> > 9299099 1864 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181328922 79875:340510799 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-11T17:58:16.753327+0000 2024-05-23T14:24:25.876796+0000 >>> >> >> > 652 periodic scrub scheduled @ >>> 2024-06-17T20:23:31.713805+0000 >>> >> >> > 10.48 5216471 5185269 0 0 4194326 >>> 6352053972 >>> >> >> > 9843580 1929 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179547710 79875:378125137 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T23:22:17.435614+0000 2024-06-11T23:22:17.435614+0000 >>> >> >> > 1379 periodic scrub scheduled @ >>> 2024-06-17T19:10:56.456654+0000 >>> >> >> > 10.49 5217119 4580870 0 0 4194304 >>> 5971558944 >>> >> >> > 9526047 1879 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'182489297 79875:389298429 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T08:57:42.791265+0000 2024-05-29T07:13:57.867295+0000 >>> >> >> > 959 periodic scrub scheduled @ >>> 2024-06-17T06:44:34.852894+0000 >>> >> >> > 10.4a 5217577 4909174 0 0 4194304 >>> 5960086599 >>> >> >> > 9511657 1910 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'180982438 79875:381432369 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-12T02:31:42.576663+0000 2024-06-01T09:08:07.860775+0000 >>> >> >> > 480 periodic scrub scheduled @ >>> 2024-06-18T01:57:37.976185+0000 >>> >> >> > 10.4b 5220270 5220270 0 0 159808 >>> 6313986186 >>> >> >> > 9763672 1934 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181279214 79875:342509944 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-11T11:03:38.997373+0000 2024-06-11T11:03:38.997373+0000 >>> >> >> > 3970 periodic scrub scheduled @ >>> 2024-06-17T17:04:16.317699+0000 >>> >> >> > 10.4c 5217864 5217864 0 0 4194304 >>> 6104537460 >>> >> >> > 9679170 1948 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181081348 79875:341937760 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T06:09:49.667680+0000 2024-06-01T13:06:12.938513+0000 >>> >> >> > 520 periodic scrub scheduled @ >>> 2024-06-18T11:39:03.277402+0000 >>> >> >> > 10.4d 5217604 4905278 0 0 8388608 >>> 6250536027 >>> >> >> > 9765571 1917 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'179562016 79875:387768172 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T13:38:28.479053+0000 2024-06-11T13:38:28.479053+0000 >>> >> >> > 2004 periodic scrub scheduled @ >>> 2024-06-17T15:12:47.615309+0000 >>> >> >> > 10.4e 5220202 5220202 0 0 12215022 >>> 5781813383 >>> >> >> > 9291138 1940 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180360470 79875:377712114 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-01T22:06:12.347760+0000 2024-05-25T19:34:49.805104+0000 >>> >> >> > 982 queued for scrub >>> >> >> > 10.4f 5215243 4561466 0 0 0 >>> 6099447266 >>> >> >> > 9605810 1930 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181794065 79875:396660094 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-02T13:35:21.451128+0000 2024-06-02T13:35:21.451128+0000 >>> >> >> > 7316 queued for scrub >>> >> >> > 10.50 5215219 5184003 0 0 0 >>> 6151789627 >>> >> >> > 9674779 1871 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181695151 79875:405060298 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T16:42:00.221013+0000 2024-06-11T16:42:00.221013+0000 >>> >> >> > 2385 periodic scrub scheduled @ >>> 2024-06-17T10:54:13.580253+0000 >>> >> >> > 10.51 5218666 5218666 0 0 8388608 >>> 5774655330 >>> >> >> > 9331373 1920 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180711664 79875:342978462 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T14:31:51.402844+0000 2024-05-24T11:26:15.539677+0000 >>> >> >> > 730 periodic scrub scheduled @ >>> 2024-06-17T20:48:06.862863+0000 >>> >> >> > 10.52 5218435 5187050 0 0 4194304 >>> 6375954134 >>> >> >> > 9843553 1935 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180834874 79875:341502004 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T08:41:43.586833+0000 2024-06-11T08:41:43.586833+0000 >>> >> >> > 8368 periodic scrub scheduled @ >>> 2024-06-16T08:59:08.248156+0000 >>> >> >> > 10.53 5216822 5216822 0 0 10001995 >>> 5786896551 >>> >> >> > 9368323 1928 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181337168 79875:402047940 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-12T04:54:21.147742+0000 2024-05-23T23:02:36.154347+0000 >>> >> >> > 557 periodic scrub scheduled @ >>> 2024-06-17T11:39:03.144049+0000 >>> >> >> > 10.54 5213949 4567626 0 0 8388608 >>> 6095478377 >>> >> >> > 9664814 1944 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'182179986 79875:421118572 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T09:19:30.470319+0000 2024-06-01T17:56:32.089348+0000 >>> >> >> > 551 periodic scrub scheduled @ >>> 2024-06-18T09:51:50.591138+0000 >>> >> >> > 10.55 5217389 5217389 0 0 4194948 >>> 5905845521 >>> >> >> > 9403344 1950 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'183481821 79875:341223864 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T15:31:05.524698+0000 2024-05-29T11:50:10.322128+0000 >>> >> >> > 584 periodic scrub scheduled @ >>> 2024-06-17T04:28:55.809357+0000 >>> >> >> > 10.56 5219142 5219142 0 0 0 >>> 6311549101 >>> >> >> > 9773102 1897 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'184062267 79875:408792597 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-10T22:17:28.523159+0000 2024-06-10T22:17:28.523159+0000 >>> >> >> > 8211 periodic scrub scheduled @ >>> 2024-06-17T03:16:39.913869+0000 >>> >> >> > 10.57 5219035 4894148 0 0 180 >>> 5862209193 >>> >> >> > 9396761 1895 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'181886155 79875:340148291 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-10T22:33:19.364284+0000 2024-05-28T12:05:26.762682+0000 >>> >> >> > 951 periodic scrub scheduled @ >>> 2024-06-16T00:55:55.641233+0000 >>> >> >> > 10.58 5218318 5185905 0 0 9878587 >>> 5911568798 >>> >> >> > 9459707 1875 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179109213 79875:390200207 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T13:08:37.380203+0000 2024-05-30T10:19:35.091370+0000 >>> >> >> > 578 periodic scrub scheduled @ >>> 2024-06-17T15:53:56.734391+0000 >>> >> >> > 10.59 5218315 5185838 0 0 4194304 >>> 5727052522 >>> >> >> > 9214424 1875 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181700141 79875:400297310 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T11:15:00.221765+0000 2024-05-21T11:28:39.056382+0000 >>> >> >> > 682 periodic deep scrub scheduled @ >>> >> >> 2024-06-16T13:00:43.750185+0000 >>> >> >> > 10.5a 5218301 5218301 0 0 4194304 >>> 5833097049 >>> >> >> > 9365354 1942 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180699909 79875:393344045 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T01:50:51.954645+0000 2024-05-25T00:11:49.917556+0000 >>> >> >> > 497 periodic scrub scheduled @ >>> 2024-06-17T12:05:07.871109+0000 >>> >> >> > 10.5b 5215134 5215134 0 0 4194802 >>> 6070043467 >>> >> >> > 9585134 1892 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181149500 79875:342539489 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-12T08:19:32.265112+0000 2024-06-02T11:33:23.094569+0000 >>> >> >> > 714 periodic scrub scheduled @ >>> 2024-06-17T23:10:14.905316+0000 >>> >> >> > 10.5c 5221503 5221503 0 0 4194304 >>> 5989312430 >>> >> >> > 9524556 1939 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180748682 79875:340750417 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T09:37:59.483197+0000 2024-05-28T03:41:58.475567+0000 >>> >> >> > 1021 periodic scrub scheduled @ >>> 2024-06-17T16:20:33.624394+0000 >>> >> >> > 10.5d 5222202 5222202 0 0 8388608 >>> 6031557641 >>> >> >> > 9520790 1928 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'179508607 79875:399919359 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-11T17:10:33.165278+0000 2024-05-30T05:53:10.772736+0000 >>> >> >> > 575 periodic scrub scheduled @ >>> 2024-06-17T11:34:40.016966+0000 >>> >> >> > 10.5e 5212770 4506643 0 0 0 >>> 5830165136 >>> >> >> > 9388228 1938 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180002484 79875:391844220 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T22:32:49.026280+0000 2024-05-24T15:31:50.362016+0000 >>> >> >> > 523 periodic scrub scheduled @ >>> 2024-06-18T06:19:36.998416+0000 >>> >> >> > 10.5f 5221504 5221504 0 0 4194394 >>> 5807975316 >>> >> >> > 9387383 1879 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181161493 79875:408530090 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T15:10:33.261745+0000 2024-05-25T06:37:01.629711+0000 >>> >> >> > 845 periodic scrub scheduled @ >>> 2024-06-18T01:25:00.005508+0000 >>> >> >> > 10.60 5217784 5186615 0 0 4194304 >>> 6208553838 >>> >> >> > 9749044 1962 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181569072 79875:392211608 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T21:44:35.372861+0000 2024-06-11T21:44:35.372861+0000 >>> >> >> > 1505 periodic scrub scheduled @ >>> 2024-06-17T19:37:54.401372+0000 >>> >> >> > 10.61 5221292 4586636 0 0 8388608 >>> 5766320671 >>> >> >> > 9280184 1910 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180945714 79875:344272511 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T16:51:25.519891+0000 2024-05-24T17:37:21.814566+0000 >>> >> >> > 566 periodic scrub scheduled @ >>> 2024-06-18T02:56:43.473028+0000 >>> >> >> > 10.62 5218498 4699002 0 0 4194466 >>> 6027128219 >>> >> >> > 9555111 1884 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180387699 79875:341929273 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T03:10:08.547987+0000 2024-06-01T02:47:43.109035+0000 >>> >> >> > 509 periodic scrub scheduled @ >>> 2024-06-17T23:58:36.771468+0000 >>> >> >> > 10.63 5219101 5219101 0 0 0 >>> 6040382110 >>> >> >> > 9500040 1885 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181623402 79875:389756489 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-02T23:50:14.947543+0000 2024-06-02T23:50:14.947543+0000 >>> >> >> > 13925 queued for scrub >>> >> >> > 10.64 5217025 5217025 0 0 2277792 >>> 6222337345 >>> >> >> > 9701950 1866 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181853168 79875:406174284 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-02T04:57:41.872723+0000 2024-06-02T04:57:41.872723+0000 >>> >> >> > 6394 queued for scrub >>> >> >> > 10.65 5217908 4590550 0 0 4194326 >>> 6223151288 >>> >> >> > 9709345 1916 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181196382 79875:340107600 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T15:53:28.725001+0000 2024-06-11T15:53:28.725001+0000 >>> >> >> > 1343 periodic scrub scheduled @ >>> 2024-06-17T11:59:39.917825+0000 >>> >> >> > 10.66 5218911 4893654 0 0 4194304 >>> 5815703055 >>> >> >> > 9338987 1939 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'183062531 79875:395193073 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-12T01:10:56.919233+0000 2024-05-25T04:50:04.300293+0000 >>> >> >> > 590 periodic scrub scheduled @ >>> 2024-06-18T11:39:55.201903+0000 >>> >> >> > 10.67 5217539 4587208 0 0 4194304 >>> 6095526883 >>> >> >> > 9640323 1955 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181875315 79875:342878178 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T07:58:41.522730+0000 2024-06-01T07:20:59.951966+0000 >>> >> >> > 553 periodic scrub scheduled @ >>> 2024-06-18T05:40:51.996147+0000 >>> >> >> > 10.68 5215852 4505902 0 0 8388608 >>> 6093858586 >>> >> >> > 9596421 1926 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'179490643 79875:379632203 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T16:02:14.995195+0000 2024-05-30T22:02:42.335910+0000 >>> >> >> > 526 periodic scrub scheduled @ >>> 2024-06-17T02:53:33.980471+0000 >>> >> >> > 10.69 5216596 5216596 0 0 4194304 >>> 5833268044 >>> >> >> > 9400414 1899 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181876551 79875:387059449 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T17:00:57.111851+0000 2024-05-23T07:05:05.989427+0000 >>> >> >> > 572 periodic scrub scheduled @ >>> 2024-06-17T18:52:42.850336+0000 >>> >> >> > 10.6a 5214106 4497370 0 0 0 >>> 5745504618 >>> >> >> > 9296938 1869 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180109669 79875:381871149 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T00:27:45.155477+0000 2024-05-25T09:22:19.570011+0000 >>> >> >> > 494 periodic scrub scheduled @ >>> 2024-06-18T07:00:30.017009+0000 >>> >> >> > 10.6b 5219339 5219339 0 0 0 >>> 5801390847 >>> >> >> > 9370084 1957 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180041524 79875:342270608 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-12T03:01:39.358464+0000 2024-05-23T01:04:22.522035+0000 >>> >> >> > 488 periodic scrub scheduled @ >>> 2024-06-17T19:03:41.161095+0000 >>> >> >> > 10.6c 5217917 5217917 0 0 554 >>> 6100979305 >>> >> >> > 9612250 1955 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180968918 79875:341488318 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T18:47:35.969345+0000 2024-05-31T11:54:59.559316+0000 >>> >> >> > 588 periodic scrub scheduled @ >>> 2024-06-18T06:02:38.982765+0000 >>> >> >> > 10.6d 5221048 4550388 0 0 4194304 >>> 6024067862 >>> >> >> > 9621947 1927 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'179373269 79875:388095690 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T03:26:37.931310+0000 2024-05-31T04:10:41.991457+0000 >>> >> >> > 502 periodic scrub scheduled @ >>> 2024-06-18T08:52:34.478975+0000 >>> >> >> > 10.6e 5215550 4888179 0 0 8388608 >>> 6110450001 >>> >> >> > 9666892 1906 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'180885974 79875:380260263 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-11T20:18:23.657558+0000 2024-06-11T20:18:23.657558+0000 >>> >> >> > 1218 periodic scrub scheduled @ >>> 2024-06-18T07:12:57.858451+0000 >>> >> >> > 10.6f 5215850 4566820 0 0 0 >>> 6107146788 >>> >> >> > 9634911 1923 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181752773 79875:396708021 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-10T17:32:51.735004+0000 2024-06-10T17:32:51.735004+0000 >>> >> >> > 10119 periodic scrub scheduled @ >>> 2024-06-17T02:47:23.675814+0000 >>> >> >> > 10.70 5218211 4672319 0 0 4194304 >>> 5793441438 >>> >> >> > 9371563 1881 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180405012 79875:404578822 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-03T04:46:35.130293+0000 2024-05-22T07:50:20.898441+0000 >>> >> >> > 969 queued for deep scrub >>> >> >> > 10.71 5218299 4599312 0 0 4194304 >>> 5807950891 >>> >> >> > 9349720 1906 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180690682 79875:344108361 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-12T03:18:16.645927+0000 2024-05-23T09:32:56.055703+0000 >>> >> >> > 487 periodic scrub scheduled @ >>> 2024-06-17T13:47:03.236787+0000 >>> >> >> > 10.72 5223470 4922347 0 0 8388608 >>> 6416468325 >>> >> >> > 9922777 1958 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'180797342 79875:342356916 [0,1,2]p0 [1,2]p1 >>> >> >> > 2024-06-12T05:42:47.545762+0000 2024-06-12T05:42:47.545762+0000 >>> >> >> > 1307 periodic scrub scheduled @ >>> 2024-06-18T13:20:41.826465+0000 >>> >> >> > 10.73 5218301 4520651 0 0 0 >>> 6293832301 >>> >> >> > 9738943 1953 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'182246986 79875:403792984 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-11T20:59:02.079157+0000 2024-06-11T20:59:02.079157+0000 >>> >> >> > 1399 periodic scrub scheduled @ >>> 2024-06-17T01:00:32.819334+0000 >>> >> >> > 10.74 5219061 4595112 0 0 0 >>> 5974775781 >>> >> >> > 9469653 1943 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181947609 79875:420571221 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-03T02:17:19.687482+0000 2024-05-28T08:43:38.825954+0000 >>> >> >> > 927 queued for scrub >>> >> >> > 10.75 5220994 4596634 0 0 25165824 >>> 5822315216 >>> >> >> > 9362548 1877 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'180376305 79875:338847231 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-03T05:52:28.743255+0000 2024-05-24T05:11:00.211501+0000 >>> >> >> > 981 queued for scrub >>> >> >> > 10.76 5222111 4892295 0 0 0 >>> 6067998823 >>> >> >> > 9574254 1935 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 2h 79875'183871180 79875:409218100 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-03T04:14:04.513004+0000 2024-06-03T04:14:04.513004+0000 >>> >> >> > 6987 queued for scrub >>> >> >> > 10.77 5219020 4574683 0 0 0 >>> 5889074277 >>> >> >> > 9460824 1896 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181614825 79875:342558162 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-12T08:07:38.169523+0000 2024-05-27T15:15:35.587972+0000 >>> >> >> > 536 periodic scrub scheduled @ >>> 2024-06-17T18:29:44.366900+0000 >>> >> >> > 10.78 5220442 4693235 0 0 8388608 >>> 5879566566 >>> >> >> > 9395501 1923 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'179031456 79875:391462415 [1,0,2]p1 [1,2]p1 >>> >> >> > 2024-06-02T09:32:42.834344+0000 2024-05-27T10:41:23.822124+0000 >>> >> >> > 994 queued for scrub >>> >> >> > 10.79 5219734 4702117 0 0 4194304 >>> 6129933545 >>> >> >> > 9646669 1926 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'182574416 79875:402249812 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-12T05:52:02.959814+0000 2024-06-01T00:53:49.548055+0000 >>> >> >> > 555 periodic scrub scheduled @ >>> 2024-06-17T17:48:00.189475+0000 >>> >> >> > 10.7a 5215303 4526846 0 0 0 >>> 6316171107 >>> >> >> > 9817029 1932 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181765231 79875:395877057 [2,0,1]p2 [2,1]p2 >>> >> >> > 2024-06-12T02:23:42.692096+0000 2024-06-12T02:23:42.692096+0000 >>> >> >> > 1457 periodic scrub scheduled @ >>> 2024-06-17T22:13:46.037000+0000 >>> >> >> > 10.7b 5217213 4606816 0 0 4194304 >>> 6226859283 >>> >> >> > 9718621 1952 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'181413541 79875:343968658 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T23:52:28.414752+0000 2024-06-11T23:52:28.414752+0000 >>> >> >> > 1308 periodic scrub scheduled @ >>> 2024-06-17T18:52:39.473049+0000 >>> >> >> > 10.7c 5215075 5215075 0 0 4194304 >>> 6013877208 >>> >> >> > 9602345 1867 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'180991860 79875:341775615 [0,2,1]p0 [1,2]p1 >>> >> >> > 2024-06-11T23:30:40.614412+0000 2024-05-31T14:10:30.542603+0000 >>> >> >> > 503 periodic scrub scheduled @ >>> 2024-06-17T02:30:15.591036+0000 >>> >> >> > 10.7d 5220700 4569720 0 0 5837609 >>> 6258736025 >>> >> >> > 9764836 1897 3000 >>> >> >> active+undersized+degraded+remapped+backfilling >>> >> >> > 3h 79875'179741885 79875:401448508 [2,1,0]p2 [2,1]p2 >>> >> >> > 2024-06-12T04:45:05.030905+0000 2024-06-12T04:45:05.030905+0000 >>> >> >> > 1565 periodic scrub scheduled @ >>> 2024-06-17T04:54:42.039122+0000 >>> >> >> > 10.7e 5217113 5217113 0 0 16777216 >>> 6072081676 >>> >> >> > 9598390 1902 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'181040361 79875:391032348 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T17:37:26.589968+0000 2024-06-11T17:37:26.589968+0000 >>> >> >> > 1613 periodic scrub scheduled @ >>> 2024-06-17T05:06:05.391340+0000 >>> >> >> > 10.7f 5218719 5218719 0 0 9909061 >>> 6130469539 >>> >> >> > 9657824 1943 3000 >>> >> >> active+undersized+degraded+remapped+backfill_wait >>> >> >> > 3h 79875'182283514 79875:408550888 [1,2,0]p1 [1,2]p1 >>> >> >> > 2024-06-11T13:05:03.552526+0000 2024-06-11T13:05:03.552526+0000 >>> >> >> > 1796 periodic scrub scheduled @ >>> 2024-06-17T09:57:04.545523+0000 >>> >> >> > >>> >> >> > * NOTE: Omap statistics are gathered during deep scrub and may be >>> >> >> > inaccurate soon afterwards depending on utilization. See >>> >> >> > >>> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics >>> >> for >>> >> >> > further details. >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > sudo ceph osd df >>> >> >> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>> >> META >>> >> >> > AVAIL %USE VAR PGS STATUS >>> >> >> > 0 ssd 3.63869 1.00000 3.6 TiB 82 GiB 50 MiB 40 >>> GiB >>> >> 42 >>> >> >> GiB >>> >> >> > 3.6 TiB 2.21 0.03 0 up >>> >> >> > 1 ssd 3.63869 1.00000 3.6 TiB 2.0 TiB 16 GiB 1021 >>> GiB >>> >> 1.0 >>> >> >> TiB >>> >> >> > 1.6 TiB 55.41 0.75 161 up >>> >> >> > 2 ssd 3.63869 1.00000 3.6 TiB 3.0 TiB 16 GiB 1.1 >>> TiB >>> >> 1.9 >>> >> >> TiB >>> >> >> > 676 GiB 81.86 1.11 161 up >>> >> >> > >>> >> >> > >>> >> >> > sudo ceph df detail >>> >> >> > --- RAW STORAGE --- >>> >> >> > CLASS SIZE AVAIL USED RAW USED %RAW USED >>> >> >> > hdd 1.2 PiB 308 TiB 871 TiB 871 TiB 73.86 >>> >> >> > ssd 11 TiB 5.8 TiB 5.1 TiB 5.1 TiB 46.51 >>> >> >> > TOTAL 1.2 PiB 314 TiB 876 TiB 876 TiB 73.61 >>> >> >> > >>> >> >> > --- POOLS --- >>> >> >> > POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED >>> >> (DATA) >>> >> >> > (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED >>> >> COMPR >>> >> >> > UNDER COMPR >>> >> >> > .mgr 1 1 827 MiB 827 MiB 0 B 208 1.6 GiB >>> 1.6 >>> >> GiB >>> >> >> > 0 B 0.11 734 GiB N/A 50 GiB N/A >>> 0 >>> >> B >>> >> >> > 0 B >>> >> >> > metadata 10 128 1.0 TiB 647 MiB 1.0 TiB 667.90M 2.1 TiB >>> 1.3 >>> >> GiB >>> >> >> > 2.1 TiB 59.45 719 GiB N/A 3.5 TiB N/A >>> >> 0 B >>> >> >> > 0 B >>> >> >> > data 11 2048 288 TiB 288 TiB 0 B 566.97M 863 TiB >>> 863 >>> >> TiB >>> >> >> > 0 B 81.10 67 TiB N/A N/A N/A >>> 3.4 >>> >> TiB >>> >> >> > 6.8 TiB >>> >> >> > ssd-data 18 32 14 GiB 14 GiB 0 B 265.17k 29 GiB >>> 29 >>> >> GiB >>> >> >> > 0 B 1.90 734 GiB N/A N/A N/A >>> 0 >>> >> B >>> >> >> > 0 B >>> >> >> > >>> >> >> > >>> >> >> > sudo ceph balancer status >>> >> >> > { >>> >> >> > "active": true, >>> >> >> > "last_optimize_duration": "0:00:00.000170", >>> >> >> > "last_optimize_started": "Wed Jun 12 13:14:49 2024", >>> >> >> > "mode": "upmap", >>> >> >> > "no_optimization_needed": true, >>> >> >> > "optimize_result": "Some objects (0.172816) are degraded; try >>> >> again >>> >> >> > later", >>> >> >> > "plans": [] >>> >> >> > } >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > [image: ariadne.ai Logo] Lars Köppel >>> >> >> > Developer >>> >> >> > Email: lars.koeppel@xxxxxxxxxx >>> >> >> > Phone: +49 6221 5993580 <+4962215993580> >>> >> >> > ariadne.ai (Germany) GmbH >>> >> >> > Häusserstraße 3, 69115 Heidelberg >>> >> >> > Amtsgericht Mannheim, HRB 744040 >>> >> >> > Geschäftsführer: Dr. Fabian Svara >>> >> >> > https://ariadne.ai >>> >> >> > >>> >> >> > >>> >> >> > On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri < >>> >> anthony.datri@xxxxxxxxx> >>> >> >> > wrote: >>> >> >> > >>> >> >> >> If you have: >>> >> >> >> >>> >> >> >> * pg_num too low (defaults are too low) >>> >> >> >> * pg_num not a power of 2 >>> >> >> >> * pg_num != number of OSDs in the pool >>> >> >> >> * balancer not enabled >>> >> >> >> >>> >> >> >> any of those might result in imbalance. >>> >> >> >> >>> >> >> >> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote: >>> >> >> >> > >>> >> >> >> > I don't have any good explanation at this point. Can you >>> share some >>> >> >> more >>> >> >> >> information like: >>> >> >> >> > >>> >> >> >> > ceph pg ls-by-pool <cephfs_metadata> >>> >> >> >> > ceph osd df (for the relevant OSDs) >>> >> >> >> > ceph df >>> >> >> >> > >>> >> >> >> > Thanks, >>> >> >> >> > Eugen >>> >> >> >> > >>> >> >> >> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: >>> >> >> >> > >>> >> >> >> >> Since my last update the size of the largest OSD increased >>> by 0.4 >>> >> TiB >>> >> >> >> while >>> >> >> >> >> the smallest one only increased by 0.1 TiB. How is this >>> possible? >>> >> >> >> >> >>> >> >> >> >> Because the metadata pool reported to have only 900MB space >>> left, >>> >> I >>> >> >> >> stopped >>> >> >> >> >> the hot-standby MDS. This gave me 8GB back but these filled >>> up in >>> >> the >>> >> >> >> last >>> >> >> >> >> 2h. >>> >> >> >> >> I think I have to zap the next OSD because the filesystem is >>> >> getting >>> >> >> >> read >>> >> >> >> >> only... >>> >> >> >> >> >>> >> >> >> >> How is it possible that an OSD has over 1 TiB less data on it >>> >> after a >>> >> >> >> >> rebuild? And how is it possible to have so different sizes of >>> >> OSDs? >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> [image: ariadne.ai Logo] Lars Köppel >>> >> >> >> >> Developer >>> >> >> >> >> Email: lars.koeppel@xxxxxxxxxx >>> >> >> >> >> Phone: +49 6221 5993580 <+4962215993580> >>> >> >> >> >> ariadne.ai (Germany) GmbH >>> >> >> >> >> Häusserstraße 3, 69115 Heidelberg >>> >> >> >> >> Amtsgericht Mannheim, HRB 744040 >>> >> >> >> >> Geschäftsführer: Dr. Fabian Svara >>> >> >> >> >> https://ariadne.ai >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel < >>> >> lars.koeppel@xxxxxxxxxx >>> >> >> > >>> >> >> >> wrote: >>> >> >> >> >> >>> >> >> >> >>> Only in warning mode. And there were no PG splits or merges >>> in >>> >> the >>> >> >> >> last 2 >>> >> >> >> >>> month. >>> >> >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> [image: ariadne.ai Logo] Lars Köppel >>> >> >> >> >>> Developer >>> >> >> >> >>> Email: lars.koeppel@xxxxxxxxxx >>> >> >> >> >>> Phone: +49 6221 5993580 <+4962215993580> >>> >> >> >> >>> ariadne.ai (Germany) GmbH >>> >> >> >> >>> Häusserstraße 3, 69115 Heidelberg >>> >> >> >> >>> Amtsgericht Mannheim, HRB 744040 >>> >> >> >> >>> Geschäftsführer: Dr. Fabian Svara >>> >> >> >> >>> https://ariadne.ai >>> >> >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> >>> >> wrote: >>> >> >> >> >>> >>> >> >> >> >>>> I don't think scrubs can cause this. Do you have autoscaler >>> >> >> enabled? >>> >> >> >> >>>> >>> >> >> >> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: >>> >> >> >> >>>> >>> >> >> >> >>>> > Hi, >>> >> >> >> >>>> > >>> >> >> >> >>>> > thank you for your response. >>> >> >> >> >>>> > >>> >> >> >> >>>> > I don't think this thread covers my problem, because the >>> OSDs >>> >> for >>> >> >> >> the >>> >> >> >> >>>> > metadata pool fill up at different rates. So I would >>> think >>> >> this >>> >> >> is >>> >> >> >> no >>> >> >> >> >>>> > direct problem with the journal. >>> >> >> >> >>>> > Because we had earlier problems with the journal I >>> changed >>> >> some >>> >> >> >> >>>> > settings(see below). I already restarted all MDS multiple >>> >> times >>> >> >> but >>> >> >> >> no >>> >> >> >> >>>> > change here. >>> >> >> >> >>>> > >>> >> >> >> >>>> > The health warnings regarding cache pressure resolve >>> normally >>> >> >> after >>> >> >> >> a >>> >> >> >> >>>> > short period of time, when the heavy load on the client >>> ends. >>> >> >> >> Sometimes >>> >> >> >> >>>> it >>> >> >> >> >>>> > stays a bit longer because an rsync is running and >>> copying >>> >> data >>> >> >> on >>> >> >> >> the >>> >> >> >> >>>> > cluster(rsync is not good at releasing the caps). >>> >> >> >> >>>> > >>> >> >> >> >>>> > Could it be a problem if scrubs run most of the time in >>> the >>> >> >> >> background? >>> >> >> >> >>>> Can >>> >> >> >> >>>> > this block any other tasks or generate new data itself? >>> >> >> >> >>>> > >>> >> >> >> >>>> > Best regards, >>> >> >> >> >>>> > Lars >>> >> >> >> >>>> > >>> >> >> >> >>>> > >>> >> >> >> >>>> > global basic >>> mds_cache_memory_limit >>> >> >> >> >>>> > 17179869184 >>> >> >> >> >>>> > global advanced >>> mds_max_caps_per_client >>> >> >> >> >>>> > 16384 >>> >> >> >> >>>> > global advanced >>> >> >> >> >>>> mds_recall_global_max_decay_threshold >>> >> >> >> >>>> > 262144 >>> >> >> >> >>>> > global advanced >>> >> mds_recall_max_decay_rate >>> >> >> >> >>>> > 1.000000 >>> >> >> >> >>>> > global advanced >>> >> >> mds_recall_max_decay_threshold >>> >> >> >> >>>> > 262144 >>> >> >> >> >>>> > mds advanced >>> mds_cache_trim_threshold >>> >> >> >> >>>> > 131072 >>> >> >> >> >>>> > mds advanced mds_heartbeat_grace >>> >> >> >> >>>> > 120.000000 >>> >> >> >> >>>> > mds advanced >>> >> mds_heartbeat_reset_grace >>> >> >> >> >>>> > 7400 >>> >> >> >> >>>> > mds advanced mds_tick_interval >>> >> >> >> >>>> > 3.000000 >>> >> >> >> >>>> > >>> >> >> >> >>>> > >>> >> >> >> >>>> > [image: ariadne.ai Logo] Lars Köppel >>> >> >> >> >>>> > Developer >>> >> >> >> >>>> > Email: lars.koeppel@xxxxxxxxxx >>> >> >> >> >>>> > Phone: +49 6221 5993580 <+4962215993580> >>> >> >> >> >>>> > ariadne.ai (Germany) GmbH >>> >> >> >> >>>> > Häusserstraße 3, 69115 Heidelberg >>> >> >> >> >>>> > Amtsgericht Mannheim, HRB 744040 >>> >> >> >> >>>> > Geschäftsführer: Dr. Fabian Svara >>> >> >> >> >>>> > https://ariadne.ai >>> >> >> >> >>>> > >>> >> >> >> >>>> > >>> >> >> >> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block < >>> eblock@xxxxxx> >>> >> >> wrote: >>> >> >> >> >>>> > >>> >> >> >> >>>> >> Hi, >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> can you check if this thread [1] applies to your >>> situation? >>> >> You >>> >> >> >> don't >>> >> >> >> >>>> >> have multi-active MDS enabled, but maybe it's still some >>> >> journal >>> >> >> >> >>>> >> trimming, or maybe misbehaving clients? In your first >>> post >>> >> there >>> >> >> >> were >>> >> >> >> >>>> >> health warnings regarding cache pressure and cache >>> size. Are >>> >> >> those >>> >> >> >> >>>> >> resolved? >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> [1] >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >>> >> >> >> >>> >> >> >>> >> >>> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> > Hello everyone, >>> >> >> >> >>>> >> > >>> >> >> >> >>>> >> > short update to this problem. >>> >> >> >> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the >>> >> expected >>> >> >> >> size >>> >> >> >> >>>> >> ~50%). >>> >> >> >> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. >>> They >>> >> >> >> jumped up >>> >> >> >> >>>> and >>> >> >> >> >>>> >> > down a lot but the higher one has now also reached >>> >> 'nearfull' >>> >> >> >> >>>> status. How >>> >> >> >> >>>> >> > is this possible? What is going on? >>> >> >> >> >>>> >> > >>> >> >> >> >>>> >> > Does anyone have a solution how to fix this without >>> zapping >>> >> >> the >>> >> >> >> OSD? >>> >> >> >> >>>> >> > >>> >> >> >> >>>> >> > Best regards, >>> >> >> >> >>>> >> > Lars >>> >> >> >> >>>> >> > >>> >> >> >> >>>> >> > >>> >> >> >> >>>> >> > [image: ariadne.ai Logo] Lars Köppel >>> >> >> >> >>>> >> > Developer >>> >> >> >> >>>> >> > Email: lars.koeppel@xxxxxxxxxx >>> >> >> >> >>>> >> > Phone: +49 6221 5993580 <+4962215993580> >>> >> >> >> >>>> >> > ariadne.ai (Germany) GmbH >>> >> >> >> >>>> >> > Häusserstraße 3, 69115 Heidelberg >>> >> >> >> >>>> >> > Amtsgericht Mannheim, HRB 744040 >>> >> >> >> >>>> >> > Geschäftsführer: Dr. Fabian Svara >>> >> >> >> >>>> >> > https://ariadne.ai >>> >> >> >> >>>> >> > _______________________________________________ >>> >> >> >> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx >>> >> >> >> >>>> >> > To unsubscribe send an email to >>> ceph-users-leave@xxxxxxx >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> _______________________________________________ >>> >> >> >> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx >>> >> >> >> >>>> >> To unsubscribe send an email to >>> ceph-users-leave@xxxxxxx >>> >> >> >> >>>> >> >>> >> >> >> >>>> >>> >> >> >> >>>> >>> >> >> >> >>>> >>> >> >> >> >>>> >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > _______________________________________________ >>> >> >> >> > ceph-users mailing list -- ceph-users@xxxxxxx >>> >> >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >> >>> >>> >>> >>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |