We have been using snapshots for a long time. The only change in usage is that we are currently deleting many small files from the system. Because this is slow (~150 requests/s) this is running for the last few weeks. Could such a load result in a problem with the MDS? I have to ask for permission to order more drives. This could take some time. [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel@xxxxxxxxxx Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai On Thu, Jun 13, 2024 at 12:55 PM Eugen Block <eblock@xxxxxx> wrote: > Downgrading isn't supported, I don't think that would be a good idea. > I also don't see anything obvious standing out in the pg output. Any > chance you can add more OSDs to the metadata pool to see if it stops > at some point? Did the cluster usage change in any way? For example > cephfs snapshots which haven't been used before or something like that? > > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > > > I updated from 17.2.6 to 17.2.7 and a few hours later to 18.2.2. > > Would it be an option to go back to 17.2.6? > > > > > > [image: ariadne.ai Logo] Lars Köppel > > Developer > > Email: lars.koeppel@xxxxxxxxxx > > Phone: +49 6221 5993580 <+4962215993580> > > ariadne.ai (Germany) GmbH > > Häusserstraße 3, 69115 Heidelberg > > Amtsgericht Mannheim, HRB 744040 > > Geschäftsführer: Dr. Fabian Svara > > https://ariadne.ai > > > > > > On Wed, Jun 12, 2024 at 5:30 PM Eugen Block <eblock@xxxxxx> wrote: > > > >> Which version did you upgrade from to 18.2.2? > >> I can’t pin it down to a specific issue, but somewhere in the back of > >> my mind is something related to a new omap format or something. But > >> I’m really not sure at all. > >> > >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> > >> > I am happy to help you with as much information as possible. I > probably > >> > just don't know where to look for it. > >> > Below are the requested information. The cluster is rebuilding the > >> > zapped OSD at the moment. This will probably take the next few days. > >> > > >> > > >> > sudo ceph pg ls-by-pool metadata > >> > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* > >> > OMAP_KEYS* LOG LOG_DUPS STATE > >> > SINCE VERSION REPORTED UP ACTING > >> > SCRUB_STAMP DEEP_SCRUB_STAMP > >> > LAST_SCRUB_DURATION SCRUB_SCHEDULING > >> > 10.0 5217325 4994695 0 0 4194304 5880891340 > >> > 9393865 1885 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'180849582 79875:391519635 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T09:08:09.829362+0000 2024-05-28T05:52:59.321589+0000 > >> > 627 periodic scrub scheduled @ 2024-06-17T08:21:31.808348+0000 > >> > 10.1 5214785 5193424 0 0 0 5843682713 > >> > 9410150 1912 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180914288 79875:342746928 [2,1,0]p2 [2,1]p2 > >> > 2024-06-01T15:56:28.927288+0000 2024-05-27T03:31:37.682966+0000 > >> > 966 queued for scrub > >> > 10.2 5218432 5187168 0 0 0 6402011266 > >> > 9812513 1874 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180970531 79875:341340204 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T13:40:58.994256+0000 2024-06-11T13:40:58.994256+0000 > >> > 1942 periodic scrub scheduled @ 2024-06-17T06:07:15.329675+0000 > >> > 10.3 5217413 5217413 0 0 8388788 5766005023 > >> > 9271787 1923 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181012233 79875:388295881 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T00:35:56.965547+0000 2024-05-23T19:54:56.121729+0000 > >> > 492 periodic scrub scheduled @ 2024-06-18T06:39:31.103864+0000 > >> > 10.4 5220069 5220069 0 0 12583466 6027548724 > >> > 9537290 1959 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181576075 79875:405295868 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T17:47:22.923514+0000 2024-05-31T02:06:55.339574+0000 > >> > 581 periodic scrub scheduled @ 2024-06-17T00:59:37.214420+0000 > >> > 10.5 5216162 5211999 0 0 4194304 5941347251 > >> > 9542764 1930 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180455793 79875:338418517 [2,1,0]p2 [2,1]p2 > >> > 2024-06-11T22:50:16.170708+0000 2024-05-30T23:49:54.316379+0000 > >> > 528 periodic scrub scheduled @ 2024-06-17T04:39:25.905185+0000 > >> > 10.6 5216100 4980459 0 0 4521984 6428088514 > >> > 9850762 1911 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'184045876 79875:396809795 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T22:24:05.102716+0000 2024-06-11T22:24:05.102716+0000 > >> > 1082 periodic scrub scheduled @ 2024-06-17T07:58:44.289885+0000 > >> > 10.7 5218232 5218232 0 0 4194304 6377065363 > >> > 9849360 1919 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182672562 79875:342449062 [1,0,2]p1 [1,2]p1 > >> > 2024-06-11T06:22:15.689422+0000 2024-06-11T06:22:15.689422+0000 > >> > 8225 periodic scrub scheduled @ 2024-06-17T13:05:59.225052+0000 > >> > 10.8 5219620 5182816 0 0 0 6167304290 > >> > 9691796 1896 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179628377 79875:378022884 [2,1,0]p2 [2,1]p2 > >> > 2024-06-11T22:06:01.386763+0000 2024-06-11T22:06:01.386763+0000 > >> > 1286 periodic scrub scheduled @ 2024-06-17T07:54:54.133093+0000 > >> > 10.9 5219448 5164591 0 0 8388698 5796048346 > >> > 9338312 1868 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181739392 79875:387412389 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T05:21:00.586747+0000 2024-05-26T11:10:59.780673+0000 > >> > 539 periodic scrub scheduled @ 2024-06-18T15:32:59.155092+0000 > >> > 10.a 5219861 5163635 0 0 12582912 5841839055 > >> > 9387200 1916 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180205688 79875:379381294 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T12:35:05.571200+0000 2024-05-22T11:07:16.041773+0000 > >> > 1093 periodic deep scrub scheduled @ > >> 2024-06-17T05:21:40.136463+0000 > >> > 10.b 5217949 5217949 0 0 16777216 5935863260 > >> > 9462127 1881 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181655745 79875:343806807 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T22:41:28.976920+0000 2024-05-26T08:43:29.217457+0000 > >> > 520 periodic scrub scheduled @ 2024-06-17T17:44:32.764093+0000 > >> > 10.c 5221697 5217118 0 0 4194304 6015217841 > >> > 9574445 1928 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180892826 79875:341490398 [2,1,0]p2 [2,1]p2 > >> > 2024-06-11T09:20:58.443473+0000 2024-05-30T00:13:50.306507+0000 > >> > 768 periodic scrub scheduled @ 2024-06-16T19:41:21.977436+0000 > >> > 10.d 5217727 4908764 0 0 0 5825598519 > >> > 9349877 1930 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'179050347 79875:387455993 [0,1,2]p0 [1,2]p1 > >> > 2024-06-02T07:27:27.873631+0000 2024-05-20T03:58:30.225170+0000 > >> > 952 queued for deep scrub > >> > 10.e 5215040 5210572 0 0 4194304 5790634469 > >> > 9327651 1905 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180049044 79875:377196119 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T05:03:08.521423+0000 2024-05-22T13:47:46.753131+0000 > >> > 527 periodic scrub scheduled @ 2024-06-17T17:30:36.079456+0000 > >> > 10.f 5217202 4885952 0 0 8388630 6005274626 > >> > 9589178 1917 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'181953069 79875:396402453 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T12:20:49.661741+0000 2024-06-11T12:20:49.661741+0000 > >> > 3949 periodic scrub scheduled @ 2024-06-17T18:37:10.974112+0000 > >> > 10.10 5214985 5157391 0 0 4194394 6275024632 > >> > 9721505 1922 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181801708 79875:395055895 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T01:49:47.831191+0000 2024-06-11T01:49:47.831191+0000 > >> > 9579 periodic scrub scheduled @ 2024-06-16T11:40:28.305905+0000 > >> > 10.11 5219236 4983134 0 0 4194304 5787580073 > >> > 9316893 1901 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'181011816 79875:343488767 [1,0,2]p1 [1,2]p1 > >> > 2024-06-03T00:06:14.862209+0000 2024-05-27T00:05:56.468361+0000 > >> > 960 queued for scrub > >> > 10.12 5214103 5182770 0 0 90 5970750188 > >> > 9505613 1891 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180210613 79875:341042616 [0,1,2]p0 [1,2]p1 > >> > 2024-06-12T06:39:48.252419+0000 2024-05-28T01:32:03.381577+0000 > >> > 527 periodic scrub scheduled @ 2024-06-18T11:23:33.994438+0000 > >> > 10.13 5214580 5214580 0 0 8388642 5793432968 > >> > 9335940 1957 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182394614 79875:393423214 [1,2,0]p1 [1,2]p1 > >> > 2024-06-10T17:52:04.690203+0000 2024-05-22T03:39:02.453336+0000 > >> > 1151 periodic deep scrub scheduled @ > >> 2024-06-16T00:28:37.682765+0000 > >> > 10.14 5218591 5218591 0 0 5053325 6046268958 > >> > 9582481 1915 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'184576050 79875:425213800 [2,0,1]p2 [2,1]p2 > >> > 2024-06-03T05:36:07.656965+0000 2024-05-28T22:46:16.814318+0000 > >> > 985 queued for scrub > >> > 10.15 5215919 4907184 0 0 11939335 5752284246 > >> > 9285870 1889 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'179713036 79875:337938973 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T14:06:50.138992+0000 2024-05-24T07:52:22.522455+0000 > >> > 786 periodic scrub scheduled @ 2024-06-17T13:00:48.869790+0000 > >> > 10.16 5221091 5221091 0 0 8388608 5966308565 > >> > 9487816 1916 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'183445984 79875:399413009 [0,2,1]p0 [1,2]p1 > >> > 2024-06-03T02:01:53.649417+0000 2024-06-03T02:01:53.649417+0000 > >> > 6935 queued for scrub > >> > 10.17 5218069 5194732 0 0 4194304 6231638001 > >> > 9735915 1890 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182390128 79875:340719915 [2,0,1]p2 [2,1]p2 > >> > 2024-06-12T06:31:01.523916+0000 2024-06-12T06:31:01.523916+0000 > >> > 1272 periodic scrub scheduled @ 2024-06-17T16:33:45.723673+0000 > >> > 10.18 5214253 5159897 0 0 0 6128506469 > >> > 9673261 1886 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179754208 79875:389150273 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T02:53:31.027154+0000 2024-06-12T02:53:31.027154+0000 > >> > 1308 periodic scrub scheduled @ 2024-06-17T22:19:41.359070+0000 > >> > 10.19 5220131 5183342 0 0 0 5869775360 > >> > 9460583 1953 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181750923 79875:393162989 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T19:21:24.106504+0000 2024-05-22T15:42:01.721091+0000 > >> > 522 periodic scrub scheduled @ 2024-06-17T20:09:55.173174+0000 > >> > 10.1a 5217968 5186700 0 0 4194304 6183652625 > >> > 9733847 1896 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181056240 79875:394241128 [0,2,1]p0 [1,2]p1 > >> > 2024-06-02T15:37:49.730965+0000 2024-06-02T15:37:49.730965+0000 > >> > 7347 queued for scrub > >> > 10.1b 5221259 5161879 0 0 0 5989030303 > >> > 9561035 1875 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181052305 79875:343144823 [0,1,2]p0 [1,2]p1 > >> > 2024-06-12T06:01:07.213894+0000 2024-06-02T00:23:51.063901+0000 > >> > 543 periodic scrub scheduled @ 2024-06-17T15:51:21.700275+0000 > >> > 10.1c 5217034 5217034 0 0 554 5979861913 > >> > 9504442 1875 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180913313 79875:341328275 [1,2,0]p1 [1,2]p1 > >> > 2024-06-01T20:02:47.025129+0000 2024-06-01T20:02:47.025129+0000 > >> > 6568 queued for scrub > >> > 10.1d 5219691 4997027 0 0 12582912 6081321658 > >> > 9599164 1918 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'179638837 79875:391808225 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T20:27:02.506715+0000 2024-05-31T22:58:06.578167+0000 > >> > 517 periodic scrub scheduled @ 2024-06-16T21:05:59.442200+0000 > >> > 10.1e 5213369 5208776 0 0 4194304 6295629499 > >> > 9765317 1941 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181145210 79875:391326487 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T07:49:28.432084+0000 2024-06-12T07:49:28.432084+0000 > >> > 1990 periodic scrub scheduled @ 2024-06-18T01:04:07.677468+0000 > >> > 10.1f 5218399 5000094 0 0 8388608 5821063844 > >> > 9421222 1896 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'180960296 79875:400635456 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T13:53:40.278050+0000 2024-05-22T19:39:02.345663+0000 > >> > 761 periodic scrub scheduled @ 2024-06-16T22:24:39.094932+0000 > >> > 10.20 5216437 5185379 0 0 0 6280994122 > >> > 9719574 1945 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181610117 79875:391874431 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T12:58:57.754648+0000 2024-06-11T12:58:57.754648+0000 > >> > 2288 periodic scrub scheduled @ 2024-06-17T23:33:35.937406+0000 > >> > 10.21 5217349 5217349 0 0 8388698 6348931429 > >> > 9822197 1867 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181724357 79875:343883762 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T18:28:34.044546+0000 2024-06-11T18:28:34.044546+0000 > >> > 1817 periodic scrub scheduled @ 2024-06-17T12:18:47.312213+0000 > >> > 10.22 5215544 4943475 0 0 4194304 5982815232 > >> > 9482159 1899 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'180230331 79875:340839853 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T15:21:21.994674+0000 2024-05-30T08:01:03.301650+0000 > >> > 649 periodic scrub scheduled @ 2024-06-17T01:33:20.228420+0000 > >> > 10.23 5217758 5217758 0 0 146 5804086623 > >> > 9336298 1920 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180948432 79875:388158699 [1,2,0]p1 [1,2]p1 > >> > 2024-06-03T05:19:32.311846+0000 2024-05-27T08:13:15.250058+0000 > >> > 971 queued for scrub > >> > 10.24 5221505 5221505 0 0 0 6360995644 > >> > 9816333 1925 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182198758 79875:406847098 [0,2,1]p0 [1,2]p1 > >> > 2024-06-12T04:00:44.315682+0000 2024-06-12T04:00:44.315682+0000 > >> > 2046 periodic scrub scheduled @ 2024-06-18T01:00:22.708069+0000 > >> > 10.25 5214911 5214911 0 0 12583002 5726320657 > >> > 9264352 1910 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180132024 79875:338167995 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T14:43:47.076906+0000 2024-05-23T12:16:34.271856+0000 > >> > 715 periodic scrub scheduled @ 2024-06-17T14:42:04.951046+0000 > >> > 10.26 5217588 4973667 0 0 0 6040775867 > >> > 9598406 1927 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'188144536 79875:400277134 [2,0,1]p2 [2,1]p2 > >> > 2024-06-02T06:55:09.016528+0000 2024-06-02T06:55:09.016528+0000 > >> > 7047 queued for scrub > >> > 10.27 5218490 5218490 0 0 8388630 6265942186 > >> > 9818746 1941 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182400328 79875:342052414 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T01:01:06.674323+0000 2024-06-12T01:01:06.674323+0000 > >> > 1500 periodic scrub scheduled @ 2024-06-18T09:04:18.289464+0000 > >> > 10.28 5221153 5221153 0 0 0 6049095371 > >> > 9597182 1913 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179438777 79875:378265375 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T04:09:58.896192+0000 2024-06-01T04:32:46.011974+0000 > >> > 553 periodic scrub scheduled @ 2024-06-18T00:53:21.846375+0000 > >> > 10.29 5214066 5214066 0 0 4194304 5767555623 > >> > 9316899 1937 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181568997 79875:387683159 [0,2,1]p0 [1,2]p1 > >> > 2024-06-10T22:48:57.757493+0000 2024-05-21T07:14:53.348341+0000 > >> > 938 periodic deep scrub scheduled @ > >> 2024-06-16T06:24:55.108695+0000 > >> > 10.2a 5216314 4977811 0 0 4194304 6012567810 > >> > 9563804 1951 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'181140734 79875:381017643 [2,1,0]p2 [2,1]p2 > >> > 2024-06-11T09:57:29.097708+0000 2024-05-30T03:47:22.798044+0000 > >> > 1169 periodic scrub scheduled @ 2024-06-16T10:08:48.053531+0000 > >> > 10.2b 5219620 5219620 0 0 8388608 5829917912 > >> > 9412426 1889 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180465782 79875:342263856 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T08:30:08.941782+0000 2024-05-24T22:00:29.535205+0000 > >> > 636 periodic scrub scheduled @ 2024-06-17T09:00:09.155675+0000 > >> > 10.2c 5214087 5214087 0 0 0 5874182922 > >> > 9375708 1871 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180616041 79875:341244736 [1,2,0]p1 [1,2]p1 > >> > 2024-06-10T23:10:08.184410+0000 2024-05-27T17:41:07.583255+0000 > >> > 1268 periodic scrub scheduled @ 2024-06-16T00:28:52.710385+0000 > >> > 10.2d 5213968 4984822 0 0 4337824 5835754026 > >> > 9417033 1947 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'178824607 79875:386427308 [2,0,1]p2 [2,1]p2 > >> > 2024-06-12T09:27:43.774331+0000 2024-05-27T05:33:45.293266+0000 > >> > 494 periodic scrub scheduled @ 2024-06-18T10:19:18.219366+0000 > >> > 10.2e 5216263 4974456 0 0 12582912 6025714323 > >> > 9553880 1940 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'180639631 79875:378284872 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T01:19:42.477063+0000 2024-05-31T08:42:41.293335+0000 > >> > 526 periodic scrub scheduled @ 2024-06-17T09:53:24.981953+0000 > >> > 10.2f 5220057 5220057 0 0 13839831 5804027574 > >> > 9370862 1882 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180961915 79875:394513378 [1,0,2]p1 [1,2]p1 > >> > 2024-06-11T19:58:05.145147+0000 2024-05-24T20:18:32.301831+0000 > >> > 535 periodic scrub scheduled @ 2024-06-18T03:16:19.773648+0000 > >> > 10.30 5217090 4955469 0 0 12582934 6438521431 > >> > 9924825 1869 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 2h 79875'181488488 79875:404940886 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T07:16:18.095439+0000 2024-06-12T07:16:18.095439+0000 > >> > 2190 periodic scrub scheduled @ 2024-06-18T10:42:55.574570+0000 > >> > 10.31 5218176 5218176 0 0 4194304 6215016361 > >> > 9752698 1897 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181622879 79875:343554151 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T09:00:15.941065+0000 2024-06-12T09:00:15.941065+0000 > >> > 1806 periodic scrub scheduled @ 2024-06-18T02:46:00.756777+0000 > >> > 10.32 5223171 4704732 0 0 8388608 6205669256 > >> > 9800320 1920 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180911263 79875:342754128 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T19:49:09.010583+0000 2024-06-11T19:49:09.010583+0000 > >> > 1664 periodic scrub scheduled @ 2024-06-17T07:07:21.062789+0000 > >> > 10.33 5215341 4525338 0 0 4194304 6066965588 > >> > 9635403 1888 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181467497 79875:402958597 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T01:59:23.933458+0000 2024-05-31T17:01:38.620377+0000 > >> > 512 periodic scrub scheduled @ 2024-06-17T22:07:45.026120+0000 > >> > 10.34 5217342 5217342 0 0 8388608 5999920961 > >> > 9570527 1956 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182013407 79875:419219483 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T09:10:17.390128+0000 2024-06-01T15:40:23.241769+0000 > >> > 601 periodic scrub scheduled @ 2024-06-18T09:56:03.061859+0000 > >> > 10.35 5222114 5222114 0 0 8388608 6021420075 > >> > 9571300 1952 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180948342 79875:338598678 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T22:59:17.017317+0000 2024-05-30T15:59:15.553266+0000 > >> > 539 periodic scrub scheduled @ 2024-06-17T02:41:31.718321+0000 > >> > 10.36 5223657 5223657 0 0 4331271 5844578100 > >> > 9347405 1903 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'183363403 79875:408484683 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T18:37:46.653097+0000 2024-05-25T14:13:51.648712+0000 > >> > 553 periodic scrub scheduled @ 2024-06-17T12:58:00.134657+0000 > >> > 10.37 5219248 5219248 0 0 4194304 6072826085 > >> > 9579049 1936 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182091890 79875:341545087 [1,0,2]p1 [1,2]p1 > >> > 2024-06-11T14:19:40.744743+0000 2024-05-29T18:40:12.602405+0000 > >> > 771 periodic scrub scheduled @ 2024-06-16T18:24:26.104867+0000 > >> > 10.38 5214022 5214022 0 0 4194858 5780805541 > >> > 9299038 1951 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'178809936 79875:390113817 [1,0,2]p1 [1,2]p1 > >> > 2024-06-02T00:55:22.046433+0000 2024-05-25T16:49:10.279003+0000 > >> > 931 queued for scrub > >> > 10.39 5220692 4542094 0 0 0 6401799077 > >> > 9864164 1872 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'182881827 79875:402437400 [2,0,1]p2 [2,1]p2 > >> > 2024-06-12T01:42:35.239155+0000 2024-06-12T01:42:35.239155+0000 > >> > 1370 periodic scrub scheduled @ 2024-06-17T10:51:19.580502+0000 > >> > 10.3a 5215648 4552651 0 0 0 6194465524 > >> > 9713558 1898 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181458295 79875:394916652 [2,1,0]p2 [2,1]p2 > >> > 2024-06-11T21:19:29.020889+0000 2024-06-11T21:19:29.020889+0000 > >> > 1227 periodic scrub scheduled @ 2024-06-17T07:19:46.503819+0000 > >> > 10.3b 5218220 5218220 0 0 8388608 5951035110 > >> > 9490698 1878 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180815984 79875:342438023 [2,1,0]p2 [2,1]p2 > >> > 2024-06-03T05:03:19.483049+0000 2024-05-28T18:50:55.369009+0000 > >> > 1004 queued for scrub > >> > 10.3c 5221615 5221615 0 0 12582912 6260894058 > >> > 9727358 1938 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181561627 79875:342153262 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T19:12:06.206008+0000 2024-06-11T19:12:06.206008+0000 > >> > 1471 periodic scrub scheduled @ 2024-06-17T02:09:19.786893+0000 > >> > 10.3d 5213771 5213771 0 0 0 6215975311 > >> > 9699615 1946 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179354493 79875:399858588 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T04:05:10.877703+0000 2024-06-11T04:05:10.877703+0000 > >> > 8123 periodic scrub scheduled @ 2024-06-17T08:03:22.770035+0000 > >> > 10.3e 5219446 4659577 0 0 4194304 5852588566 > >> > 9370152 1933 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180035882 79875:391595513 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T20:35:43.624903+0000 2024-05-22T21:40:55.606650+0000 > >> > 520 periodic scrub scheduled @ 2024-06-17T00:54:06.347642+0000 > >> > 10.3f 5219729 4889443 0 0 16777216 5925457842 > >> > 9484458 1948 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'181330738 79875:409113809 [0,2,1]p0 [1,2]p1 > >> > 2024-06-12T05:12:01.852109+0000 2024-05-27T20:03:22.704122+0000 > >> > 533 periodic scrub scheduled @ 2024-06-17T13:07:19.370647+0000 > >> > 10.40 5222241 4445164 0 0 0 6312545191 > >> > 9773697 1873 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181341418 79875:393186178 [2,1,0]p2 [2,1]p2 > >> > 2024-06-10T20:00:37.561045+0000 2024-06-10T20:00:37.561045+0000 > >> > 7712 periodic scrub scheduled @ 2024-06-16T05:59:45.317316+0000 > >> > 10.41 5215195 5183508 0 0 4194326 6000770355 > >> > 9529537 1938 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181494334 79875:343712275 [0,2,1]p0 [1,2]p1 > >> > 2024-06-12T04:18:58.693597+0000 2024-06-01T11:05:59.370690+0000 > >> > 539 periodic scrub scheduled @ 2024-06-17T15:34:34.997281+0000 > >> > 10.42 5219564 4681323 0 0 12582912 5822745109 > >> > 9331625 1916 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'179811520 79875:340958258 [1,2,0]p1 [1,2]p1 > >> > 2024-06-02T19:58:06.439770+0000 2024-05-21T18:28:28.794738+0000 > >> > 11044 queued for deep scrub > >> > 10.43 5219412 4599458 0 0 594 5737440613 > >> > 9239869 1950 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180634536 79875:388879991 [1,0,2]p1 [1,2]p1 > >> > 2024-06-01T18:13:17.068416+0000 2024-05-20T16:32:20.302230+0000 > >> > 1002 queued for deep scrub > >> > 10.44 5216008 5216008 0 0 4194304 6270195298 > >> > 9786806 1944 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182088681 79875:406175480 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T00:19:01.240405+0000 2024-06-12T00:19:01.240405+0000 > >> > 1593 periodic scrub scheduled @ 2024-06-17T12:19:53.888923+0000 > >> > 10.45 5216238 4694313 0 0 0 5714807184 > >> > 9223666 1944 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'179709878 79875:338307727 [1,0,2]p1 [1,2]p1 > >> > 2024-06-02T00:39:50.657116+0000 2024-05-21T02:52:14.376959+0000 > >> > 958 queued for deep scrub > >> > 10.46 5217950 5217950 0 0 9729074 5945415128 > >> > 9499020 1915 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'183726716 79875:395024265 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T14:56:28.875949+0000 2024-05-29T04:19:53.409918+0000 > >> > 724 periodic scrub scheduled @ 2024-06-16T17:42:09.428674+0000 > >> > 10.47 5219125 5219125 0 0 4194304 5775128028 > >> > 9299099 1864 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181328922 79875:340510799 [1,0,2]p1 [1,2]p1 > >> > 2024-06-11T17:58:16.753327+0000 2024-05-23T14:24:25.876796+0000 > >> > 652 periodic scrub scheduled @ 2024-06-17T20:23:31.713805+0000 > >> > 10.48 5216471 5185269 0 0 4194326 6352053972 > >> > 9843580 1929 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179547710 79875:378125137 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T23:22:17.435614+0000 2024-06-11T23:22:17.435614+0000 > >> > 1379 periodic scrub scheduled @ 2024-06-17T19:10:56.456654+0000 > >> > 10.49 5217119 4580870 0 0 4194304 5971558944 > >> > 9526047 1879 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'182489297 79875:389298429 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T08:57:42.791265+0000 2024-05-29T07:13:57.867295+0000 > >> > 959 periodic scrub scheduled @ 2024-06-17T06:44:34.852894+0000 > >> > 10.4a 5217577 4909174 0 0 4194304 5960086599 > >> > 9511657 1910 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'180982438 79875:381432369 [0,1,2]p0 [1,2]p1 > >> > 2024-06-12T02:31:42.576663+0000 2024-06-01T09:08:07.860775+0000 > >> > 480 periodic scrub scheduled @ 2024-06-18T01:57:37.976185+0000 > >> > 10.4b 5220270 5220270 0 0 159808 6313986186 > >> > 9763672 1934 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181279214 79875:342509944 [2,1,0]p2 [2,1]p2 > >> > 2024-06-11T11:03:38.997373+0000 2024-06-11T11:03:38.997373+0000 > >> > 3970 periodic scrub scheduled @ 2024-06-17T17:04:16.317699+0000 > >> > 10.4c 5217864 5217864 0 0 4194304 6104537460 > >> > 9679170 1948 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181081348 79875:341937760 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T06:09:49.667680+0000 2024-06-01T13:06:12.938513+0000 > >> > 520 periodic scrub scheduled @ 2024-06-18T11:39:03.277402+0000 > >> > 10.4d 5217604 4905278 0 0 8388608 6250536027 > >> > 9765571 1917 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'179562016 79875:387768172 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T13:38:28.479053+0000 2024-06-11T13:38:28.479053+0000 > >> > 2004 periodic scrub scheduled @ 2024-06-17T15:12:47.615309+0000 > >> > 10.4e 5220202 5220202 0 0 12215022 5781813383 > >> > 9291138 1940 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180360470 79875:377712114 [1,0,2]p1 [1,2]p1 > >> > 2024-06-01T22:06:12.347760+0000 2024-05-25T19:34:49.805104+0000 > >> > 982 queued for scrub > >> > 10.4f 5215243 4561466 0 0 0 6099447266 > >> > 9605810 1930 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181794065 79875:396660094 [2,1,0]p2 [2,1]p2 > >> > 2024-06-02T13:35:21.451128+0000 2024-06-02T13:35:21.451128+0000 > >> > 7316 queued for scrub > >> > 10.50 5215219 5184003 0 0 0 6151789627 > >> > 9674779 1871 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181695151 79875:405060298 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T16:42:00.221013+0000 2024-06-11T16:42:00.221013+0000 > >> > 2385 periodic scrub scheduled @ 2024-06-17T10:54:13.580253+0000 > >> > 10.51 5218666 5218666 0 0 8388608 5774655330 > >> > 9331373 1920 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180711664 79875:342978462 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T14:31:51.402844+0000 2024-05-24T11:26:15.539677+0000 > >> > 730 periodic scrub scheduled @ 2024-06-17T20:48:06.862863+0000 > >> > 10.52 5218435 5187050 0 0 4194304 6375954134 > >> > 9843553 1935 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180834874 79875:341502004 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T08:41:43.586833+0000 2024-06-11T08:41:43.586833+0000 > >> > 8368 periodic scrub scheduled @ 2024-06-16T08:59:08.248156+0000 > >> > 10.53 5216822 5216822 0 0 10001995 5786896551 > >> > 9368323 1928 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181337168 79875:402047940 [0,2,1]p0 [1,2]p1 > >> > 2024-06-12T04:54:21.147742+0000 2024-05-23T23:02:36.154347+0000 > >> > 557 periodic scrub scheduled @ 2024-06-17T11:39:03.144049+0000 > >> > 10.54 5213949 4567626 0 0 8388608 6095478377 > >> > 9664814 1944 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'182179986 79875:421118572 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T09:19:30.470319+0000 2024-06-01T17:56:32.089348+0000 > >> > 551 periodic scrub scheduled @ 2024-06-18T09:51:50.591138+0000 > >> > 10.55 5217389 5217389 0 0 4194948 5905845521 > >> > 9403344 1950 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'183481821 79875:341223864 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T15:31:05.524698+0000 2024-05-29T11:50:10.322128+0000 > >> > 584 periodic scrub scheduled @ 2024-06-17T04:28:55.809357+0000 > >> > 10.56 5219142 5219142 0 0 0 6311549101 > >> > 9773102 1897 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'184062267 79875:408792597 [1,2,0]p1 [1,2]p1 > >> > 2024-06-10T22:17:28.523159+0000 2024-06-10T22:17:28.523159+0000 > >> > 8211 periodic scrub scheduled @ 2024-06-17T03:16:39.913869+0000 > >> > 10.57 5219035 4894148 0 0 180 5862209193 > >> > 9396761 1895 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'181886155 79875:340148291 [0,1,2]p0 [1,2]p1 > >> > 2024-06-10T22:33:19.364284+0000 2024-05-28T12:05:26.762682+0000 > >> > 951 periodic scrub scheduled @ 2024-06-16T00:55:55.641233+0000 > >> > 10.58 5218318 5185905 0 0 9878587 5911568798 > >> > 9459707 1875 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179109213 79875:390200207 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T13:08:37.380203+0000 2024-05-30T10:19:35.091370+0000 > >> > 578 periodic scrub scheduled @ 2024-06-17T15:53:56.734391+0000 > >> > 10.59 5218315 5185838 0 0 4194304 5727052522 > >> > 9214424 1875 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181700141 79875:400297310 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T11:15:00.221765+0000 2024-05-21T11:28:39.056382+0000 > >> > 682 periodic deep scrub scheduled @ > >> 2024-06-16T13:00:43.750185+0000 > >> > 10.5a 5218301 5218301 0 0 4194304 5833097049 > >> > 9365354 1942 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180699909 79875:393344045 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T01:50:51.954645+0000 2024-05-25T00:11:49.917556+0000 > >> > 497 periodic scrub scheduled @ 2024-06-17T12:05:07.871109+0000 > >> > 10.5b 5215134 5215134 0 0 4194802 6070043467 > >> > 9585134 1892 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181149500 79875:342539489 [2,0,1]p2 [2,1]p2 > >> > 2024-06-12T08:19:32.265112+0000 2024-06-02T11:33:23.094569+0000 > >> > 714 periodic scrub scheduled @ 2024-06-17T23:10:14.905316+0000 > >> > 10.5c 5221503 5221503 0 0 4194304 5989312430 > >> > 9524556 1939 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180748682 79875:340750417 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T09:37:59.483197+0000 2024-05-28T03:41:58.475567+0000 > >> > 1021 periodic scrub scheduled @ 2024-06-17T16:20:33.624394+0000 > >> > 10.5d 5222202 5222202 0 0 8388608 6031557641 > >> > 9520790 1928 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'179508607 79875:399919359 [1,0,2]p1 [1,2]p1 > >> > 2024-06-11T17:10:33.165278+0000 2024-05-30T05:53:10.772736+0000 > >> > 575 periodic scrub scheduled @ 2024-06-17T11:34:40.016966+0000 > >> > 10.5e 5212770 4506643 0 0 0 5830165136 > >> > 9388228 1938 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180002484 79875:391844220 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T22:32:49.026280+0000 2024-05-24T15:31:50.362016+0000 > >> > 523 periodic scrub scheduled @ 2024-06-18T06:19:36.998416+0000 > >> > 10.5f 5221504 5221504 0 0 4194394 5807975316 > >> > 9387383 1879 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181161493 79875:408530090 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T15:10:33.261745+0000 2024-05-25T06:37:01.629711+0000 > >> > 845 periodic scrub scheduled @ 2024-06-18T01:25:00.005508+0000 > >> > 10.60 5217784 5186615 0 0 4194304 6208553838 > >> > 9749044 1962 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181569072 79875:392211608 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T21:44:35.372861+0000 2024-06-11T21:44:35.372861+0000 > >> > 1505 periodic scrub scheduled @ 2024-06-17T19:37:54.401372+0000 > >> > 10.61 5221292 4586636 0 0 8388608 5766320671 > >> > 9280184 1910 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180945714 79875:344272511 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T16:51:25.519891+0000 2024-05-24T17:37:21.814566+0000 > >> > 566 periodic scrub scheduled @ 2024-06-18T02:56:43.473028+0000 > >> > 10.62 5218498 4699002 0 0 4194466 6027128219 > >> > 9555111 1884 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180387699 79875:341929273 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T03:10:08.547987+0000 2024-06-01T02:47:43.109035+0000 > >> > 509 periodic scrub scheduled @ 2024-06-17T23:58:36.771468+0000 > >> > 10.63 5219101 5219101 0 0 0 6040382110 > >> > 9500040 1885 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181623402 79875:389756489 [0,1,2]p0 [1,2]p1 > >> > 2024-06-02T23:50:14.947543+0000 2024-06-02T23:50:14.947543+0000 > >> > 13925 queued for scrub > >> > 10.64 5217025 5217025 0 0 2277792 6222337345 > >> > 9701950 1866 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181853168 79875:406174284 [1,2,0]p1 [1,2]p1 > >> > 2024-06-02T04:57:41.872723+0000 2024-06-02T04:57:41.872723+0000 > >> > 6394 queued for scrub > >> > 10.65 5217908 4590550 0 0 4194326 6223151288 > >> > 9709345 1916 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181196382 79875:340107600 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T15:53:28.725001+0000 2024-06-11T15:53:28.725001+0000 > >> > 1343 periodic scrub scheduled @ 2024-06-17T11:59:39.917825+0000 > >> > 10.66 5218911 4893654 0 0 4194304 5815703055 > >> > 9338987 1939 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'183062531 79875:395193073 [1,0,2]p1 [1,2]p1 > >> > 2024-06-12T01:10:56.919233+0000 2024-05-25T04:50:04.300293+0000 > >> > 590 periodic scrub scheduled @ 2024-06-18T11:39:55.201903+0000 > >> > 10.67 5217539 4587208 0 0 4194304 6095526883 > >> > 9640323 1955 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181875315 79875:342878178 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T07:58:41.522730+0000 2024-06-01T07:20:59.951966+0000 > >> > 553 periodic scrub scheduled @ 2024-06-18T05:40:51.996147+0000 > >> > 10.68 5215852 4505902 0 0 8388608 6093858586 > >> > 9596421 1926 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'179490643 79875:379632203 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T16:02:14.995195+0000 2024-05-30T22:02:42.335910+0000 > >> > 526 periodic scrub scheduled @ 2024-06-17T02:53:33.980471+0000 > >> > 10.69 5216596 5216596 0 0 4194304 5833268044 > >> > 9400414 1899 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181876551 79875:387059449 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T17:00:57.111851+0000 2024-05-23T07:05:05.989427+0000 > >> > 572 periodic scrub scheduled @ 2024-06-17T18:52:42.850336+0000 > >> > 10.6a 5214106 4497370 0 0 0 5745504618 > >> > 9296938 1869 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180109669 79875:381871149 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T00:27:45.155477+0000 2024-05-25T09:22:19.570011+0000 > >> > 494 periodic scrub scheduled @ 2024-06-18T07:00:30.017009+0000 > >> > 10.6b 5219339 5219339 0 0 0 5801390847 > >> > 9370084 1957 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180041524 79875:342270608 [0,1,2]p0 [1,2]p1 > >> > 2024-06-12T03:01:39.358464+0000 2024-05-23T01:04:22.522035+0000 > >> > 488 periodic scrub scheduled @ 2024-06-17T19:03:41.161095+0000 > >> > 10.6c 5217917 5217917 0 0 554 6100979305 > >> > 9612250 1955 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180968918 79875:341488318 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T18:47:35.969345+0000 2024-05-31T11:54:59.559316+0000 > >> > 588 periodic scrub scheduled @ 2024-06-18T06:02:38.982765+0000 > >> > 10.6d 5221048 4550388 0 0 4194304 6024067862 > >> > 9621947 1927 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'179373269 79875:388095690 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T03:26:37.931310+0000 2024-05-31T04:10:41.991457+0000 > >> > 502 periodic scrub scheduled @ 2024-06-18T08:52:34.478975+0000 > >> > 10.6e 5215550 4888179 0 0 8388608 6110450001 > >> > 9666892 1906 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'180885974 79875:380260263 [0,1,2]p0 [1,2]p1 > >> > 2024-06-11T20:18:23.657558+0000 2024-06-11T20:18:23.657558+0000 > >> > 1218 periodic scrub scheduled @ 2024-06-18T07:12:57.858451+0000 > >> > 10.6f 5215850 4566820 0 0 0 6107146788 > >> > 9634911 1923 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181752773 79875:396708021 [2,1,0]p2 [2,1]p2 > >> > 2024-06-10T17:32:51.735004+0000 2024-06-10T17:32:51.735004+0000 > >> > 10119 periodic scrub scheduled @ 2024-06-17T02:47:23.675814+0000 > >> > 10.70 5218211 4672319 0 0 4194304 5793441438 > >> > 9371563 1881 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180405012 79875:404578822 [1,2,0]p1 [1,2]p1 > >> > 2024-06-03T04:46:35.130293+0000 2024-05-22T07:50:20.898441+0000 > >> > 969 queued for deep scrub > >> > 10.71 5218299 4599312 0 0 4194304 5807950891 > >> > 9349720 1906 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180690682 79875:344108361 [0,2,1]p0 [1,2]p1 > >> > 2024-06-12T03:18:16.645927+0000 2024-05-23T09:32:56.055703+0000 > >> > 487 periodic scrub scheduled @ 2024-06-17T13:47:03.236787+0000 > >> > 10.72 5223470 4922347 0 0 8388608 6416468325 > >> > 9922777 1958 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'180797342 79875:342356916 [0,1,2]p0 [1,2]p1 > >> > 2024-06-12T05:42:47.545762+0000 2024-06-12T05:42:47.545762+0000 > >> > 1307 periodic scrub scheduled @ 2024-06-18T13:20:41.826465+0000 > >> > 10.73 5218301 4520651 0 0 0 6293832301 > >> > 9738943 1953 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'182246986 79875:403792984 [2,0,1]p2 [2,1]p2 > >> > 2024-06-11T20:59:02.079157+0000 2024-06-11T20:59:02.079157+0000 > >> > 1399 periodic scrub scheduled @ 2024-06-17T01:00:32.819334+0000 > >> > 10.74 5219061 4595112 0 0 0 5974775781 > >> > 9469653 1943 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181947609 79875:420571221 [2,1,0]p2 [2,1]p2 > >> > 2024-06-03T02:17:19.687482+0000 2024-05-28T08:43:38.825954+0000 > >> > 927 queued for scrub > >> > 10.75 5220994 4596634 0 0 25165824 5822315216 > >> > 9362548 1877 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'180376305 79875:338847231 [2,0,1]p2 [2,1]p2 > >> > 2024-06-03T05:52:28.743255+0000 2024-05-24T05:11:00.211501+0000 > >> > 981 queued for scrub > >> > 10.76 5222111 4892295 0 0 0 6067998823 > >> > 9574254 1935 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 2h 79875'183871180 79875:409218100 [1,0,2]p1 [1,2]p1 > >> > 2024-06-03T04:14:04.513004+0000 2024-06-03T04:14:04.513004+0000 > >> > 6987 queued for scrub > >> > 10.77 5219020 4574683 0 0 0 5889074277 > >> > 9460824 1896 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181614825 79875:342558162 [2,0,1]p2 [2,1]p2 > >> > 2024-06-12T08:07:38.169523+0000 2024-05-27T15:15:35.587972+0000 > >> > 536 periodic scrub scheduled @ 2024-06-17T18:29:44.366900+0000 > >> > 10.78 5220442 4693235 0 0 8388608 5879566566 > >> > 9395501 1923 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'179031456 79875:391462415 [1,0,2]p1 [1,2]p1 > >> > 2024-06-02T09:32:42.834344+0000 2024-05-27T10:41:23.822124+0000 > >> > 994 queued for scrub > >> > 10.79 5219734 4702117 0 0 4194304 6129933545 > >> > 9646669 1926 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'182574416 79875:402249812 [1,2,0]p1 [1,2]p1 > >> > 2024-06-12T05:52:02.959814+0000 2024-06-01T00:53:49.548055+0000 > >> > 555 periodic scrub scheduled @ 2024-06-17T17:48:00.189475+0000 > >> > 10.7a 5215303 4526846 0 0 0 6316171107 > >> > 9817029 1932 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181765231 79875:395877057 [2,0,1]p2 [2,1]p2 > >> > 2024-06-12T02:23:42.692096+0000 2024-06-12T02:23:42.692096+0000 > >> > 1457 periodic scrub scheduled @ 2024-06-17T22:13:46.037000+0000 > >> > 10.7b 5217213 4606816 0 0 4194304 6226859283 > >> > 9718621 1952 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'181413541 79875:343968658 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T23:52:28.414752+0000 2024-06-11T23:52:28.414752+0000 > >> > 1308 periodic scrub scheduled @ 2024-06-17T18:52:39.473049+0000 > >> > 10.7c 5215075 5215075 0 0 4194304 6013877208 > >> > 9602345 1867 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'180991860 79875:341775615 [0,2,1]p0 [1,2]p1 > >> > 2024-06-11T23:30:40.614412+0000 2024-05-31T14:10:30.542603+0000 > >> > 503 periodic scrub scheduled @ 2024-06-17T02:30:15.591036+0000 > >> > 10.7d 5220700 4569720 0 0 5837609 6258736025 > >> > 9764836 1897 3000 > >> active+undersized+degraded+remapped+backfilling > >> > 3h 79875'179741885 79875:401448508 [2,1,0]p2 [2,1]p2 > >> > 2024-06-12T04:45:05.030905+0000 2024-06-12T04:45:05.030905+0000 > >> > 1565 periodic scrub scheduled @ 2024-06-17T04:54:42.039122+0000 > >> > 10.7e 5217113 5217113 0 0 16777216 6072081676 > >> > 9598390 1902 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'181040361 79875:391032348 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T17:37:26.589968+0000 2024-06-11T17:37:26.589968+0000 > >> > 1613 periodic scrub scheduled @ 2024-06-17T05:06:05.391340+0000 > >> > 10.7f 5218719 5218719 0 0 9909061 6130469539 > >> > 9657824 1943 3000 > >> active+undersized+degraded+remapped+backfill_wait > >> > 3h 79875'182283514 79875:408550888 [1,2,0]p1 [1,2]p1 > >> > 2024-06-11T13:05:03.552526+0000 2024-06-11T13:05:03.552526+0000 > >> > 1796 periodic scrub scheduled @ 2024-06-17T09:57:04.545523+0000 > >> > > >> > * NOTE: Omap statistics are gathered during deep scrub and may be > >> > inaccurate soon afterwards depending on utilization. See > >> > http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics > for > >> > further details. > >> > > >> > > >> > > >> > sudo ceph osd df > >> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META > >> > AVAIL %USE VAR PGS STATUS > >> > 0 ssd 3.63869 1.00000 3.6 TiB 82 GiB 50 MiB 40 GiB > 42 > >> GiB > >> > 3.6 TiB 2.21 0.03 0 up > >> > 1 ssd 3.63869 1.00000 3.6 TiB 2.0 TiB 16 GiB 1021 GiB > 1.0 > >> TiB > >> > 1.6 TiB 55.41 0.75 161 up > >> > 2 ssd 3.63869 1.00000 3.6 TiB 3.0 TiB 16 GiB 1.1 TiB > 1.9 > >> TiB > >> > 676 GiB 81.86 1.11 161 up > >> > > >> > > >> > sudo ceph df detail > >> > --- RAW STORAGE --- > >> > CLASS SIZE AVAIL USED RAW USED %RAW USED > >> > hdd 1.2 PiB 308 TiB 871 TiB 871 TiB 73.86 > >> > ssd 11 TiB 5.8 TiB 5.1 TiB 5.1 TiB 46.51 > >> > TOTAL 1.2 PiB 314 TiB 876 TiB 876 TiB 73.61 > >> > > >> > --- POOLS --- > >> > POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED > (DATA) > >> > (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED > COMPR > >> > UNDER COMPR > >> > .mgr 1 1 827 MiB 827 MiB 0 B 208 1.6 GiB 1.6 > GiB > >> > 0 B 0.11 734 GiB N/A 50 GiB N/A 0 > B > >> > 0 B > >> > metadata 10 128 1.0 TiB 647 MiB 1.0 TiB 667.90M 2.1 TiB 1.3 > GiB > >> > 2.1 TiB 59.45 719 GiB N/A 3.5 TiB N/A > 0 B > >> > 0 B > >> > data 11 2048 288 TiB 288 TiB 0 B 566.97M 863 TiB 863 > TiB > >> > 0 B 81.10 67 TiB N/A N/A N/A 3.4 > TiB > >> > 6.8 TiB > >> > ssd-data 18 32 14 GiB 14 GiB 0 B 265.17k 29 GiB 29 > GiB > >> > 0 B 1.90 734 GiB N/A N/A N/A 0 > B > >> > 0 B > >> > > >> > > >> > sudo ceph balancer status > >> > { > >> > "active": true, > >> > "last_optimize_duration": "0:00:00.000170", > >> > "last_optimize_started": "Wed Jun 12 13:14:49 2024", > >> > "mode": "upmap", > >> > "no_optimization_needed": true, > >> > "optimize_result": "Some objects (0.172816) are degraded; try > again > >> > later", > >> > "plans": [] > >> > } > >> > > >> > > >> > > >> > [image: ariadne.ai Logo] Lars Köppel > >> > Developer > >> > Email: lars.koeppel@xxxxxxxxxx > >> > Phone: +49 6221 5993580 <+4962215993580> > >> > ariadne.ai (Germany) GmbH > >> > Häusserstraße 3, 69115 Heidelberg > >> > Amtsgericht Mannheim, HRB 744040 > >> > Geschäftsführer: Dr. Fabian Svara > >> > https://ariadne.ai > >> > > >> > > >> > On Wed, Jun 12, 2024 at 2:53 PM Anthony D'Atri < > anthony.datri@xxxxxxxxx> > >> > wrote: > >> > > >> >> If you have: > >> >> > >> >> * pg_num too low (defaults are too low) > >> >> * pg_num not a power of 2 > >> >> * pg_num != number of OSDs in the pool > >> >> * balancer not enabled > >> >> > >> >> any of those might result in imbalance. > >> >> > >> >> > On Jun 12, 2024, at 07:33, Eugen Block <eblock@xxxxxx> wrote: > >> >> > > >> >> > I don't have any good explanation at this point. Can you share some > >> more > >> >> information like: > >> >> > > >> >> > ceph pg ls-by-pool <cephfs_metadata> > >> >> > ceph osd df (for the relevant OSDs) > >> >> > ceph df > >> >> > > >> >> > Thanks, > >> >> > Eugen > >> >> > > >> >> > Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> >> > > >> >> >> Since my last update the size of the largest OSD increased by 0.4 > TiB > >> >> while > >> >> >> the smallest one only increased by 0.1 TiB. How is this possible? > >> >> >> > >> >> >> Because the metadata pool reported to have only 900MB space left, > I > >> >> stopped > >> >> >> the hot-standby MDS. This gave me 8GB back but these filled up in > the > >> >> last > >> >> >> 2h. > >> >> >> I think I have to zap the next OSD because the filesystem is > getting > >> >> read > >> >> >> only... > >> >> >> > >> >> >> How is it possible that an OSD has over 1 TiB less data on it > after a > >> >> >> rebuild? And how is it possible to have so different sizes of > OSDs? > >> >> >> > >> >> >> > >> >> >> [image: ariadne.ai Logo] Lars Köppel > >> >> >> Developer > >> >> >> Email: lars.koeppel@xxxxxxxxxx > >> >> >> Phone: +49 6221 5993580 <+4962215993580> > >> >> >> ariadne.ai (Germany) GmbH > >> >> >> Häusserstraße 3, 69115 Heidelberg > >> >> >> Amtsgericht Mannheim, HRB 744040 > >> >> >> Geschäftsführer: Dr. Fabian Svara > >> >> >> https://ariadne.ai > >> >> >> > >> >> >> > >> >> >> On Tue, Jun 11, 2024 at 3:47 PM Lars Köppel < > lars.koeppel@xxxxxxxxxx > >> > > >> >> wrote: > >> >> >> > >> >> >>> Only in warning mode. And there were no PG splits or merges in > the > >> >> last 2 > >> >> >>> month. > >> >> >>> > >> >> >>> > >> >> >>> [image: ariadne.ai Logo] Lars Köppel > >> >> >>> Developer > >> >> >>> Email: lars.koeppel@xxxxxxxxxx > >> >> >>> Phone: +49 6221 5993580 <+4962215993580> > >> >> >>> ariadne.ai (Germany) GmbH > >> >> >>> Häusserstraße 3, 69115 Heidelberg > >> >> >>> Amtsgericht Mannheim, HRB 744040 > >> >> >>> Geschäftsführer: Dr. Fabian Svara > >> >> >>> https://ariadne.ai > >> >> >>> > >> >> >>> > >> >> >>> On Tue, Jun 11, 2024 at 3:32 PM Eugen Block <eblock@xxxxxx> > wrote: > >> >> >>> > >> >> >>>> I don't think scrubs can cause this. Do you have autoscaler > >> enabled? > >> >> >>>> > >> >> >>>> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> >> >>>> > >> >> >>>> > Hi, > >> >> >>>> > > >> >> >>>> > thank you for your response. > >> >> >>>> > > >> >> >>>> > I don't think this thread covers my problem, because the OSDs > for > >> >> the > >> >> >>>> > metadata pool fill up at different rates. So I would think > this > >> is > >> >> no > >> >> >>>> > direct problem with the journal. > >> >> >>>> > Because we had earlier problems with the journal I changed > some > >> >> >>>> > settings(see below). I already restarted all MDS multiple > times > >> but > >> >> no > >> >> >>>> > change here. > >> >> >>>> > > >> >> >>>> > The health warnings regarding cache pressure resolve normally > >> after > >> >> a > >> >> >>>> > short period of time, when the heavy load on the client ends. > >> >> Sometimes > >> >> >>>> it > >> >> >>>> > stays a bit longer because an rsync is running and copying > data > >> on > >> >> the > >> >> >>>> > cluster(rsync is not good at releasing the caps). > >> >> >>>> > > >> >> >>>> > Could it be a problem if scrubs run most of the time in the > >> >> background? > >> >> >>>> Can > >> >> >>>> > this block any other tasks or generate new data itself? > >> >> >>>> > > >> >> >>>> > Best regards, > >> >> >>>> > Lars > >> >> >>>> > > >> >> >>>> > > >> >> >>>> > global basic mds_cache_memory_limit > >> >> >>>> > 17179869184 > >> >> >>>> > global advanced mds_max_caps_per_client > >> >> >>>> > 16384 > >> >> >>>> > global advanced > >> >> >>>> mds_recall_global_max_decay_threshold > >> >> >>>> > 262144 > >> >> >>>> > global advanced > mds_recall_max_decay_rate > >> >> >>>> > 1.000000 > >> >> >>>> > global advanced > >> mds_recall_max_decay_threshold > >> >> >>>> > 262144 > >> >> >>>> > mds advanced mds_cache_trim_threshold > >> >> >>>> > 131072 > >> >> >>>> > mds advanced mds_heartbeat_grace > >> >> >>>> > 120.000000 > >> >> >>>> > mds advanced > mds_heartbeat_reset_grace > >> >> >>>> > 7400 > >> >> >>>> > mds advanced mds_tick_interval > >> >> >>>> > 3.000000 > >> >> >>>> > > >> >> >>>> > > >> >> >>>> > [image: ariadne.ai Logo] Lars Köppel > >> >> >>>> > Developer > >> >> >>>> > Email: lars.koeppel@xxxxxxxxxx > >> >> >>>> > Phone: +49 6221 5993580 <+4962215993580> > >> >> >>>> > ariadne.ai (Germany) GmbH > >> >> >>>> > Häusserstraße 3, 69115 Heidelberg > >> >> >>>> > Amtsgericht Mannheim, HRB 744040 > >> >> >>>> > Geschäftsführer: Dr. Fabian Svara > >> >> >>>> > https://ariadne.ai > >> >> >>>> > > >> >> >>>> > > >> >> >>>> > On Tue, Jun 11, 2024 at 2:05 PM Eugen Block <eblock@xxxxxx> > >> wrote: > >> >> >>>> > > >> >> >>>> >> Hi, > >> >> >>>> >> > >> >> >>>> >> can you check if this thread [1] applies to your situation? > You > >> >> don't > >> >> >>>> >> have multi-active MDS enabled, but maybe it's still some > journal > >> >> >>>> >> trimming, or maybe misbehaving clients? In your first post > there > >> >> were > >> >> >>>> >> health warnings regarding cache pressure and cache size. Are > >> those > >> >> >>>> >> resolved? > >> >> >>>> >> > >> >> >>>> >> [1] > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> > >> >> > >> > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7U27L27FHHPDYGA6VNNVWGLTXCGP7X23/#VOOV235D4TP5TEOJUWHF4AVXIOTHYQQE > >> >> >>>> >> > >> >> >>>> >> Zitat von Lars Köppel <lars.koeppel@xxxxxxxxxx>: > >> >> >>>> >> > >> >> >>>> >> > Hello everyone, > >> >> >>>> >> > > >> >> >>>> >> > short update to this problem. > >> >> >>>> >> > The zapped OSD is rebuilt and it has now 1.9 TiB (the > expected > >> >> size > >> >> >>>> >> ~50%). > >> >> >>>> >> > The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They > >> >> jumped up > >> >> >>>> and > >> >> >>>> >> > down a lot but the higher one has now also reached > 'nearfull' > >> >> >>>> status. How > >> >> >>>> >> > is this possible? What is going on? > >> >> >>>> >> > > >> >> >>>> >> > Does anyone have a solution how to fix this without zapping > >> the > >> >> OSD? > >> >> >>>> >> > > >> >> >>>> >> > Best regards, > >> >> >>>> >> > Lars > >> >> >>>> >> > > >> >> >>>> >> > > >> >> >>>> >> > [image: ariadne.ai Logo] Lars Köppel > >> >> >>>> >> > Developer > >> >> >>>> >> > Email: lars.koeppel@xxxxxxxxxx > >> >> >>>> >> > Phone: +49 6221 5993580 <+4962215993580> > >> >> >>>> >> > ariadne.ai (Germany) GmbH > >> >> >>>> >> > Häusserstraße 3, 69115 Heidelberg > >> >> >>>> >> > Amtsgericht Mannheim, HRB 744040 > >> >> >>>> >> > Geschäftsführer: Dr. Fabian Svara > >> >> >>>> >> > https://ariadne.ai > >> >> >>>> >> > _______________________________________________ > >> >> >>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> >> >>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> _______________________________________________ > >> >> >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx > >> >> >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >> >>>> >> > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >> > >> >> > >> > >> > >> > >> > > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |