Here you go. It doesn't format very well, so I'll summarize what I'm seeing. 5.c has 78051 OMAP_BYTES and 398 OMAP_KEYS 5.16 has 80186950 OMAP_BYTES and 401505 OMAP_KEYS The remaining 30 PGS have zero of both. However, the BYTES for each PG is very much the same at around 8900000 for each. # ceph pg ls-by-pool default.rgw.meta PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 5.0 26240 0 0 0 8909864 0 0 10076 active+clean 10h 8093'54176 8093:5396520 [21,4,12]p21 [21,4,12]p21 2023-07- 31T21:13:20.554485+0000 2023-07-26T03:40:27.457946+0000 5 periodic scrub scheduled @ 2023-08-01T23:55:14.134653+0000 5.1 26065 0 0 0 8840849 0 0 10029 active+clean 10h 8093'56529 8093:4891333 [14,7,23]p14 [14,7,23]p14 2023-07- 31T20:37:34.920128+0000 2023-07-30T10:55:16.529046+0000 5 periodic scrub scheduled @ 2023-08-01T21:12:04.440688+0000 5.2 26406 0 0 0 8943783 0 0 10076 active+clean 20h 8093'56776 8093:5022283 [26,8,25]p26 [26,8,25]p26 2023-07- 31T11:08:00.886979+0000 2023-07-30T06:03:44.341435+0000 5 periodic scrub scheduled @ 2023-08-01T16:02:32.076634+0000 5.3 26250 0 0 0 8932714 0 0 10086 active+clean 20h 8093'56786 8093:5109316 [0,26,32]p0 [0,26,32]p0 2023-07- 31T11:02:35.864452+0000 2023-07-30T04:25:56.495524+0000 5 periodic scrub scheduled @ 2023-08-01T20:18:30.975924+0000 5.4 26071 0 0 0 8874237 0 0 10024 active+clean 6h 8092'53824 8093:5146409 [15,7,34]p15 [15,7,34]p15 2023-08- 01T01:16:48.361184+0000 2023-07-25T15:47:10.627594+0000 5 periodic scrub scheduled @ 2023-08-02T12:32:06.359395+0000 5.5 26160 0 0 0 8870317 0 0 10073 active+clean 12h 8093'56173 8093:4706658 [9,31,16]p9 [9,31,16]p9 2023-07- 31T18:52:26.301525+0000 2023-07-29T08:19:00.537322+0000 5 periodic scrub scheduled @ 2023-08-02T02:11:16.267794+0000 5.6 26186 0 0 0 8904446 0 0 10084 active+clean 44m 8093'57584 8093:5032349 [7,10,38]p7 [7,10,38]p7 2023-08- 01T06:37:45.184419+0000 2023-08-01T06:37:45.184419+0000 313 periodic scrub scheduled @ 2023-08-02T10:01:32.285716+0000 5.7 26292 0 0 0 8908213 0 0 9695 active+clean 87m 8093'56896 8093:4969718 [36,1,13]p36 [36,1,13]p36 2023-08- 01T05:55:06.016287+0000 2023-07-30T21:49:33.028594+0000 5 periodic scrub scheduled @ 2023-08-02T13:54:38.778542+0000 5.8 26323 0 0 0 8911110 0 0 9747 active+clean 3h 8093'56448 8093:4981465 [36,15,2]p36 [36,15,2]p36 2023-08- 01T04:21:06.360778+0000 2023-07-29T14:46:02.363530+0000 5 periodic scrub scheduled @ 2023-08-02T10:36:48.764085+0000 5.9 26035 0 0 0 8829430 0 0 10034 active+clean 20h 8093'56335 8093:4829155 [37,21,24]p37 [37,21,24]p37 2023-07- 31T11:07:39.961751+0000 2023-07-31T11:07:39.961751+0000 309 periodic scrub scheduled @ 2023-08-01T22:42:18.862879+0000 5.a 26052 0 0 0 8859067 0 0 10087 active+clean 27h 8092'56087 8093:5022933 [2,23,10]p2 [2,23,10]p2 2023-07- 31T03:28:44.433360+0000 2023-07-31T03:28:44.433360+0000 248 periodic scrub scheduled @ 2023-08-01T13:36:39.897693+0000 5.b 25759 0 0 0 8739834 0 0 9693 active+clean 15h 8090'56293 8093:4837010 [36,7,28]p36 [36,7,28]p36 2023-07- 31T15:55:00.415967+0000 2023-07-31T15:55:00.415967+0000 323 periodic scrub scheduled @ 2023-08-01T23:41:03.756058+0000 5.c 25927 0 0 0 8788271 78051 398 10051 active+clean 24h 8093'174851 8093:4982667 [5,36,18]p5 [5,36,18]p5 2023-07- 31T07:20:32.208533+0000 2023-07-31T07:20:32.208533+0000 315 periodic scrub scheduled @ 2023-08-01T18:18:15.292651+0000 5.d 25995 0 0 0 8815306 0 0 10070 active+clean 2h 8093'57270 8093:4994478 [32,13,16]p32 [32,13,16]p32 2023-08- 01T04:27:55.863933+0000 2023-08-01T04:27:55.863933+0000 294 periodic scrub scheduled @ 2023-08-02T05:55:30.108279+0000 5.e 26253 0 0 0 8939984 0 0 10018 active+clean 5h 8092'56919 8093:5135033 [37,19,4]p37 [37,19,4]p37 2023-08- 01T01:38:15.740983+0000 2023-07-30T21:55:45.349878+0000 5 periodic scrub scheduled @ 2023-08-02T04:40:09.172157+0000 5.f 26025 0 0 0 8821973 0 0 10020 active+clean 25h 8093'53120 8093:4794909 [14,11,8]p14 [14,11,8]p14 2023-07- 31T06:22:22.849194+0000 2023-07-25T02:42:24.135997+0000 5 periodic deep scrub scheduled @ 2023-08-01T13:50:38.914828+0000 5.10 25999 0 0 0 8821303 0 0 10048 active+clean 3h 8092'55848 8093:5151525 [39,27,14]p39 [39,27,14]p39 2023-08- 01T03:40:44.355521+0000 2023-07-29T23:58:49.567904+0000 5 periodic scrub scheduled @ 2023-08-02T04:44:52.459615+0000 5.11 26200 0 0 0 8897148 0 0 8909 active+clean 23h 8093'56309 8093:4858657 [35,24,23]p35 [35,24,23]p35 2023-07- 31T08:19:21.090885+0000 2023-07-30T07:21:21.135342+0000 5 periodic scrub scheduled @ 2023-08-01T13:09:38.620801+0000 5.12 26043 0 0 0 8803496 0 0 10016 active+clean 33h 8093'52716 8093:5090415 [21,35,3]p21 [21,35,3]p21 2023-07- 30T22:10:31.308788+0000 2023-07-24T14:40:34.453392+0000 5 periodic deep scrub scheduled @ 2023-08-01T09:36:02.058413+0000 5.13 25929 0 0 0 8785411 0 0 10029 active+clean 16h 8090'54629 8093:5096641 [32,17,9]p32 [32,17,9]p32 2023-07- 31T15:19:04.119491+0000 2023-07-27T16:36:53.401620+0000 5 periodic scrub scheduled @ 2023-08-01T22:19:03.573063+0000 5.14 26102 0 0 0 8858274 0 0 10069 active+clean 89m 8093'54671 8093:4958083 [3,29,12]p3 [3,29,12]p3 2023-08- 01T05:53:20.722831+0000 2023-07-27T11:29:26.930179+0000 5 periodic scrub scheduled @ 2023-08-02T14:03:11.521746+0000 5.15 26122 0 0 0 8850708 0 0 9419 active+clean 14h 8093'57119 8093:4854254 [28,8,29]p28 [28,8,29]p28 2023-07- 31T17:04:04.790500+0000 2023-07-31T17:04:04.790500+0000 309 periodic scrub scheduled @ 2023-08-02T04:29:01.728903+0000 5.16 26168 0 0 0 8869093 80186950 401505 10031 active+clean 32h 8093'122396977 8093:127435157 [26,39,9]p26 [26,39,9]p26 2023-07- 30T23:19:12.784044+0000 2023-07-30T23:19:12.784044+0000 258 periodic scrub scheduled @ 2023-08-01T10:29:11.462563+0000 5.17 25504 0 0 0 8634818 0 0 10081 active+clean 23h 8093'55481 8093:4742014 [4,25,34]p4 [4,25,34]p4 2023-07- 31T08:11:23.105601+0000 2023-07-31T08:11:23.105601+0000 309 periodic scrub scheduled @ 2023-08-01T15:11:19.240302+0000 5.18 26143 0 0 0 8846680 0 0 10014 active+clean 23h 8093'55015 8093:4927120 [22,11,36]p22 [22,11,36]p22 2023-07- 31T08:18:27.757381+0000 2023-07-27T19:04:02.036522+0000 5 periodic scrub scheduled @ 2023-08-01T16:25:12.741673+0000 5.19 26117 0 0 0 8864860 0 0 10073 active+clean 27h 8093'55173 8093:5001362 [1,28,27]p1 [1,28,27]p1 2023-07- 31T03:37:40.525594+0000 2023-07-28T10:50:18.232627+0000 5 periodic scrub scheduled @ 2023-08-01T15:06:48.300456+0000 5.1a 26186 0 0 0 8870466 0 0 10025 active+clean 31h 8093'54025 8093:4991279 [34,17,22]p34 [34,17,22]p34 2023-07- 31T00:20:28.853158+0000 2023-07-25T21:31:55.045662+0000 5 periodic scrub scheduled @ 2023-08-01T09:42:20.646380+0000 5.1b 26070 0 0 0 8854703 0 0 10087 active+clean 8h 8093'56487 8093:4886996 [22,7,37]p22 [22,7,37]p22 2023-07- 31T22:39:17.793412+0000 2023-07-30T16:07:40.211725+0000 6 periodic scrub scheduled @ 2023-08-01T23:23:58.360930+0000 5.1c 26302 0 0 0 8925675 0 0 10015 active+clean 16h 8093'54915 8093:4986627 [33,11,2]p33 [33,11,2]p33 2023-07- 31T15:07:27.474683+0000 2023-07-27T15:11:42.794360+0000 5 periodic scrub scheduled @ 2023-08-01T15:51:01.248015+0000 5.1d 26075 0 0 0 8839499 0 0 10075 active+clean 21h 8093'52575 8093:4857107 [33,16,17]p33 [33,16,17]p33 2023-07- 31T09:23:24.577919+0000 2023-07-24T16:52:37.965968+0000 5 periodic deep scrub scheduled @ 2023-08-01T11:33:35.067584+0000 5.1e 26182 0 0 0 8905099 0 0 8905 active+clean+scrubbing+deep 4m 8093'56905 8093:4936987 [35,20,15]p35 [35,20,15]p35 2023-07- 31T04:25:51.171690+0000 2023-07-24T14:28:19.531701+0000 5 deep scrubbing for 259s 5.1f 25978 0 0 0 8796478 0 0 10068 active+clean 13h 8092'56868 8093:4813791 [26,30,13]p26 [26,30,13]p26 2023-07- 31T17:50:40.349450+0000 2023-07-31T17:50:40.349450+0000 311 periodic scrub scheduled @ 2023-08-02T04:39:41.913504+0000 On Tue, 2023-08-01 at 06:14 +0000, Eugen Block wrote: > Yeah, regarding data distribution increasing the pg_num of the data > pool is recommended. But could you also share the output of: > > ceph pg ls-by-pool default.rgw.meta > > That's where the large omap was reported, maybe you'll need to > increase the pg_num for that pool as well. Personally, I always > disable the autoscaler. > > Zitat von Mark Johnson <markj@xxxxxxxxx>: > > > Thanks Bailey, > > > > With regards to the PG count, we've been relying on PG autoscale > > and > > it is currently enabled. I figure I'd need to disable autoscale > > and > > manually increase the PG count on the default.rgw.buckets.data > > pool, > > correct? We're coming from our existing clusters running Jewel to > > this new Quincy cluster and have no prior experience with > > autoscale > > so we were just assuming autoscale would manage PG counts better > > than us doing it manually. As you can probably guess, we don't > > have > > much experience with Ceph. > > > > Regards, > > Mark Johnson > > > > > > On Mon, 2023-07-31 at 21:54 -0300, Bailey Allison wrote: > > [You don't often get email from > > ballison@xxxxxxxxxxxx<mailto:ballison@xxxxxxxxxxxx>. Learn why > > this > > is important at https://aka.ms/LearnAboutSenderIdentification ;] > > > > Hi, > > > > It appears you have quite a low PG count on your cluster (approx. > > 20 > > PGs per each OSD). > > > > Usually is recommended to have about 100-150 per each OSD. With a > > lower PG count you can have issues with balancing data and cause > > errors such as large OMAP objects. > > > > Might not be the fix in this case but either way would still > > recommend increasing PGs on your pools. > > > > If you look at the OMAP value in your ceph osd df you can see that > > some OSDs have 2GB while some have 500MB. Even for data some > > drives > > are holding 900GB while others 2TB. > > > > You will have to issue a deep-scrub on the PGs as well to get > > updated OMAP data once the PGs are increased. > > > > Regards, > > > > Bailey > > > > -----Original Message----- > > From: Mark Johnson <markj@xxxxxxxxx<mailto:markj@xxxxxxxxx>> > > Sent: July 31, 2023 9:01 PM > > To: eblock@xxxxxx<mailto:eblock@xxxxxx>; > > ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> > > Subject: Re: 1 Large omap object found > > > > Sure thing. Thanks for the reply. > > > > ceph df > > > > --- RAW STORAGE --- > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 291 TiB 244 TiB 47 TiB 47 TiB 16.02 > > TOTAL 291 TiB 244 TiB 47 TiB 47 TiB 16.02 > > > > --- POOLS --- > > POOL ID PGS STORED OBJECTS USED > > %USED > > MAX AVAIL > > .mgr 1 1 459 MiB 116 1.3 GiB > > 0 > > 65 TiB > > .rgw.root 2 32 1.3 KiB 4 48 KiB > > 0 > > 65 TiB > > default.rgw.log 3 32 5.3 KiB 209 468 KiB > > 0 > > 65 TiB > > default.rgw.control 4 32 0 B 8 0 B > > 0 > > 65 TiB > > default.rgw.meta 5 32 452 MiB 828.75k 10 GiB > > 0 > > 65 TiB > > default.rgw.buckets.index 6 32 17 GiB 4.56M 51 GiB > > 0.03 > > 65 TiB > > default.rgw.buckets.data 7 128 15 TiB 54.51M 46 TiB > > 19.24 > > 65 TiB > > cephfs_metadata 8 16 258 MiB 98 775 MiB > > 0 > > 65 TiB > > cephfs_data 9 32 1.9 GiB 998 5.6 GiB > > 0 > > 65 TiB > > > > > > ceph osd df > > > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > > META > > AVAIL %USE VAR PGS STATUS > > 0 hdd 7.27739 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 550 MiB > > 12 > > GiB 5.7 TiB 21.70 1.35 21 up > > 1 hdd 7.27739 1.00000 7.3 TiB 995 GiB 986 GiB 1.1 GiB > > 7.6 > > GiB 6.3 TiB 13.35 0.83 28 up > > 2 hdd 7.27739 1.00000 7.3 TiB 996 GiB 986 GiB 2.1 GiB > > 7.9 > > GiB 6.3 TiB 13.37 0.83 22 up > > 3 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 513 MiB > > 10 > > GiB 5.9 TiB 18.35 1.15 28 up > > 4 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 527 MiB > > 8.3 > > GiB 6.2 TiB 15.02 0.94 22 up > > 5 hdd 7.27739 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.5 GiB > > 14 > > GiB 5.5 TiB 25.01 1.56 28 up > > 6 hdd 7.27739 1.00000 7.3 TiB 746 GiB 739 GiB 1.0 GiB > > 5.8 > > GiB 6.5 TiB 10.01 0.63 20 up > > 7 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 1.1 GiB > > 8.8 > > GiB 6.2 TiB 15.04 0.94 20 up > > 8 hdd 7.27739 1.00000 7.3 TiB 871 GiB 864 GiB 544 MiB > > 6.7 > > GiB 6.4 TiB 11.69 0.73 27 up > > 9 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 3.3 GiB > > 11 > > GiB 5.9 TiB 18.37 1.15 28 up > > 30 hdd 7.27739 1.00000 7.3 TiB 1.8 TiB 1.8 TiB 1.6 GiB > > 14 > > GiB 5.5 TiB 25.01 1.56 35 up > > 31 hdd 7.27739 1.00000 7.3 TiB 747 GiB 739 GiB 2.2 GiB > > 6.2 > > GiB 6.5 TiB 10.03 0.63 20 up > > 32 hdd 7.27739 1.00000 7.3 TiB 996 GiB 987 GiB 1.5 GiB > > 7.9 > > GiB 6.3 TiB 13.37 0.83 26 up > > 33 hdd 7.27739 1.00000 7.3 TiB 995 GiB 985 GiB 1.5 GiB > > 7.7 > > GiB 6.3 TiB 13.35 0.83 25 up > > 34 hdd 7.27739 1.00000 7.3 TiB 750 GiB 742 GiB 2.1 GiB > > 5.7 > > GiB 6.5 TiB 10.07 0.63 25 up > > 35 hdd 7.27739 1.00000 7.3 TiB 2.1 TiB 2.0 TiB 571 MiB > > 15 > > GiB 5.2 TiB 28.36 1.77 34 up > > 36 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 1.5 GiB > > 10 > > GiB 5.9 TiB 18.37 1.15 31 up > > 37 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 524 MiB > > 8.2 > > GiB 6.2 TiB 14.99 0.94 26 up > > 38 hdd 7.27739 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.6 GiB > > 12 > > GiB 5.7 TiB 21.70 1.35 28 up > > 39 hdd 7.27739 1.00000 7.3 TiB 1.5 TiB 1.4 TiB 2.4 GiB > > 11 > > GiB 5.8 TiB 20.04 1.25 30 up > > 10 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 1.6 GiB > > 10 > > GiB 5.9 TiB 18.34 1.14 26 up > > 12 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 1 KiB > > 9.9 > > GiB 5.9 TiB 18.37 1.15 25 up > > 14 hdd 7.27739 1.00000 7.3 TiB 1.5 TiB 1.4 TiB 593 MiB > > 10 > > GiB 5.8 TiB 19.98 1.25 22 up > > 16 hdd 7.27739 1.00000 7.3 TiB 997 GiB 987 GiB 2.2 GiB > > 7.5 > > GiB 6.3 TiB 13.38 0.84 19 up > > 18 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 1.1 GiB > > 8.7 > > GiB 6.2 TiB 15.02 0.94 26 up > > 20 hdd 7.27739 1.00000 7.3 TiB 1.6 TiB 1.6 TiB 1.1 GiB > > 12 > > GiB 5.7 TiB 21.68 1.35 26 up > > 22 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 559 MiB > > 10 > > GiB 5.9 TiB 18.34 1.14 22 up > > 24 hdd 7.27739 1.00000 7.3 TiB 872 GiB 864 GiB 1020 MiB > > 6.8 > > GiB 6.4 TiB 11.70 0.73 23 up > > 26 hdd 7.27739 1.00000 7.3 TiB 749 GiB 741 GiB 1.8 GiB > > 6.3 > > GiB 6.5 TiB 10.05 0.63 25 up > > 28 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 1.5 GiB > > 10 > > GiB 5.9 TiB 18.36 1.15 32 up > > 11 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 2.6 GiB > > 8.5 > > GiB 6.2 TiB 15.02 0.94 23 up > > 13 hdd 7.27739 1.00000 7.3 TiB 1.3 TiB 1.3 TiB 2.2 GiB > > 10 > > GiB 5.9 TiB 18.38 1.15 36 up > > 15 hdd 7.27739 1.00000 7.3 TiB 995 GiB 986 GiB 1.1 GiB > > 7.7 > > GiB 6.3 TiB 13.35 0.83 25 up > > 17 hdd 7.27739 1.00000 7.3 TiB 623 GiB 618 GiB 419 KiB > > 5.0 > > GiB 6.7 TiB 8.35 0.52 23 up > > 19 hdd 7.27739 1.00000 7.3 TiB 870 GiB 863 GiB 513 MiB > > 6.6 > > GiB 6.4 TiB 11.67 0.73 21 up > > 21 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 1.5 GiB > > 8.6 > > GiB 6.2 TiB 15.02 0.94 25 up > > 23 hdd 7.27739 1.00000 7.3 TiB 746 GiB 739 GiB 564 MiB > > 5.8 > > GiB 6.5 TiB 10.01 0.62 22 up > > 25 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 2.1 GiB > > 8.4 > > GiB 6.2 TiB 15.03 0.94 24 up > > 27 hdd 7.27739 1.00000 7.3 TiB 1.2 TiB 1.2 TiB 532 MiB > > 9.1 > > GiB 6.1 TiB 16.68 1.04 23 up > > 29 hdd 7.27739 1.00000 7.3 TiB 1.1 TiB 1.1 TiB 1.1 GiB > > 8.4 > > GiB 6.2 TiB 14.99 0.94 19 up > > TOTAL 291 TiB 47 TiB 46 TiB 51 GiB > > 359 > > GiB 244 TiB 16.02 > > MIN/MAX VAR: 0.52/1.77 STDDEV: 4.56 > > > > > > On Mon, 2023-07-31 at 09:22 +0000, Eugen Block wrote: > > Hi, > > > > can you share some more details like 'ceph df' and 'ceph osd df'? I > > don't have too much advice yet, but to see all entries in your meta > > pool you need add the --all flag because those objects are stored > > in > > namespaces: > > > > rados -p default.rgw.meta ls --all > > > > That pool contains user and bucket information (example): > > > > # rados -p default.rgw.meta ls --all > > users.uid admin.buckets > > users.keys c0fba3ea7d9c4321b5205752c85baa85 users.uid > > admin > > users.keys JBWPRAPP1AQG471AMGC4 users.uid > > e434b82737cf4138b899c0785b49112d.buckets > > users.uid e434b82737cf4138b899c0785b49112d > > > > > > > > Zitat von Mark Johnson <markj@xxxxxxxxx<mailto:markj@xxxxxxxxx>>: > > > > I've been going round and round in circles trying to work this one > > out but I'm getting nowhere. We're running a 4 node quincy cluster > > (17.2.6) which recently reported the following: > > > > ceph.log-20230729.gz:2023-07-28T08:31:42.390003+0000 osd.26 > > (osd.26) > > 13834 : cluster [WRN] Large omap object found. Object: > > 5:6c65dd84:users.uid::callrecordings$callrecordings_rw.buckets:head > > > > PG: 5.21bba636 (5.16) Key count: 378454 Size (bytes): 75565579 > > > > This happened a week or so ago (only the key count was only just > > over the 200000 threshold on that occasion) and after much > > searching > > around, I found an article that suggested a deep scrub on the pg > > would likely resolve the issue, so I forced a deep scrub and > > shortly > > after, the warning cleared. Came into the office today to discover > > the above. It's on the same PG as before which is in the > > default.rgw.meta pool. This time, after forcing a deep-scrub on > > that PG, nothing changed. I did it a second time just to be sure > > but got the same result. > > > > I keep finding a suse article that simply suggests increasing the > > threshold to the previous default of 2,000,000, but other articles > > I > > read say it was lowered for a reason and that by the time it hits > > that figure, it's too late so I don't want to just mask it. > > Problem > > is that I don't really understand it. I found a thread here from > > a > > bit over two years ago but their issue was in the > > default.rgw.buckets.index pool. A step in the solution was to list > > out the problematic object id and check the objects per shard > > however, if I issue the command "rados -p default.rgw.meta ls" it > > returns nothing. I get a big list from "rados -p > > default.rgw.buckets.index ls" just nothing from the first pool. I > > think it may be because the meta pool isn't indexed based on > > something I read, but I really don't know what I'm talking about > > tbh. > > > > I don't know if this is helpful, but if I list out all the PGs for > > that pool, there are 32 PGs and 5.16 shows 80186950 bytes and > > 401505 > > keys. PG 5.c has 75298 and 384 keys. The remaining 30 PGs show > > zero bytes and zero keys. I'm really not sure how to troubleshoot > > and resolve from here. For the record, dynamic resharding is > > enabled in that no options have been set in the config and that is > > the default setting. > > > > Based on the suse article I mentioned which also references the > > default.rgw.meta pool, I'm gathering our issue is because we have > > so > > many buckets that are all owned by the one user and the solution is > > either: > > > > * delete unused buckets > > * create multiple users and spread buckets evenly across all users > > (not something we can do) > > * increase the threshold to stop the warning > > > > Problem is that I'm having trouble verifying this is the issue. > > I've > > tried dumping out bucket stats to a file (radosgw-admin bucket > > stats > > bucket_stats.txt) but after three hours this is still running with > > no output. > > > > Thanks for your time, > > Mark > > _______________________________________________ > > ceph-users mailing list -- > > ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send > > an > > email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > > > > > > _______________________________________________ > > ceph-users mailing list -- > > ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send > > an > > email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > > > > _______________________________________________ > > ceph-users mailing list -- > > ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send > > an > > email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |