Hi Ramin, that's an interesting approach for part of the problem. Two comments: 1) We don't have RGW services, we observe large omap objects in a ceph fs meta-data pool. Hence, the script doesn't help here. 2) The command "radosgw-admin" has the very annoying property of creating pools without asking. It would be great if you could add a sanity check that confirms that RGW services are actually present *before* executing any radosgw-admin command and exiting if none are present. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Ramin Najjarbashi <ramin.najarbashi@xxxxxxxxx> Sent: Tuesday, October 31, 2023 11:22 AM To: Frank Schilder Cc: Eugen Block; ceph-users@xxxxxxx Subject: Re: Re: find PG with large omap object https://gist.github.com/RaminNietzsche/b8702014c333f3a44a995d5a6d4a56be On Mon, Oct 16, 2023 at 4:27 PM Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>> wrote: Hi Eugen, the warning threshold is per omap object, not per PG (which apparently has more than 1 omap object). Still, I misread the numbers by 1 order, which means that the difference between the last 2 entries is about 100000, which does point towards a large omap object in PG 12.193. I issued a deep-scrub on this PG and the warning is resolved. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@xxxxxx<mailto:eblock@xxxxxx>> Sent: Monday, October 16, 2023 2:41 PM To: Frank Schilder Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> Subject: Re: Re: find PG with large omap object Hi Frank, > # jq '[.pg_stats[] | {"id": .pgid, "nk": .stat_sum.num_omap_keys}] | > sort_by(.nk)' pgs.dump | tail > }, > { > "id": "12.17b", > "nk": 1493776 > }, > { > "id": "12.193", > "nk": 1583589 > } those numbers are > 1 million and the warning threshold is 200k. So a warning is expected. Zitat von Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>>: > Hi Eugen, > > thanks for the one-liner :) I'm afraid I'm in the same position as > before though. > > I dumped all PGs to a file and executed these 2 commands: > > # jq '[.pg_stats[] | {"id": .pgid, "nk": .stat_sum.num_omap_bytes}] > | sort_by(.nk)' pgs.dump | tail > }, > { > "id": "12.193", > "nk": 1002401056 > }, > { > "id": "21.0", > "nk": 1235777228 > } > ] > > # jq '[.pg_stats[] | {"id": .pgid, "nk": .stat_sum.num_omap_keys}] | > sort_by(.nk)' pgs.dump | tail > }, > { > "id": "12.17b", > "nk": 1493776 > }, > { > "id": "12.193", > "nk": 1583589 > } > ] > > Neither is beyond the warn limit and pool 12 is indeed the pool > where the warnings came from. OK, now back to the logs: > > # zgrep -i 'Large omap object found. Object:' /var/log/ceph/ceph.log-* > /var/log/ceph/ceph.log-20231008.gz:2023-10-05T01:25:14.581962+0200 > osd.592 (osd.592) 104 : cluster [WRN] Large omap object found. > Object: 12:c05de58b:::63b.00000000:head PG: 12.d1a7ba03 (12.3) Key > count: 200001 Size (bytes): 230080309 > /var/log/ceph/ceph.log-20231008.gz:2023-10-07T04:33:02.678879+0200 > osd.949 (osd.949) 6897 : cluster [WRN] Large omap object found. > Object: 12:c9a32586:::63a.00000000:head PG: 12.61a4c593 (12.193) Key > count: 200243 Size (bytes): 230307097 > /var/log/ceph/ceph.log-20231008.gz:2023-10-07T07:22:40.512228+0200 > osd.988 (osd.988) 4365 : cluster [WRN] Large omap object found. > Object: 12:eb96322f:::637.00000000:head PG: 12.f44c69d7 (12.1d7) Key > count: 200329 Size (bytes): 230310393 > /var/log/ceph/ceph.log-20231008.gz:2023-10-07T15:08:03.785186+0200 > osd.50 (osd.50) 4549 : cluster [WRN] Large omap object found. > Object: 12:08fb0eb7:::635.00000000:head PG: 12.ed70df10 (12.110) Key > count: 200183 Size (bytes): 230150641 > /var/log/ceph/ceph.log-20231008.gz:2023-10-07T16:37:12.901470+0200 > osd.18 (osd.18) 7011 : cluster [WRN] Large omap object found. > Object: 12:d6758956:::634.00000000:head PG: 12.6a91ae6b (12.6b) Key > count: 200247 Size (bytes): 230343371 > /var/log/ceph/ceph.log-20231008.gz:2023-10-08T01:25:16.125068+0200 > osd.980 (osd.980) 308 : cluster [WRN] Large omap object found. > Object: 12:63f985e7:::639.00000000:head PG: 12.e7a19fc6 (12.1c6) Key > count: 200160 Size (bytes): 230179282 > /var/log/ceph/ceph.log-20231015:2023-10-09T00:51:32.587849+0200 > osd.563 (osd.563) 3661 : cluster [WRN] Large omap object found. > Object: 12:44346421:::632.00000000:head PG: 12.84262c22 (12.22) Key > count: 200325 Size (bytes): 230481029 > /var/log/ceph/ceph.log-20231015:2023-10-09T15:35:28.803117+0200 > osd.949 (osd.949) 7088 : cluster [WRN] Large omap object found. > Object: 12:c9a32586:::63a.00000000:head PG: 12.61a4c593 (12.193) Key > count: 200327 Size (bytes): 230404872 > /var/log/ceph/ceph.log-20231015:2023-10-09T18:51:35.615096+0200 > osd.592 (osd.592) 461 : cluster [WRN] Large omap object found. > Object: 12:c05de58b:::63b.00000000:head PG: 12.d1a7ba03 (12.3) Key > count: 200228 Size (bytes): 230347361 > > The warnings report a key count > 200000, but none of the PGs in the > dump does. Apparently, all these PGs were (deep-) scrubbed already > and the omap key count was updated (or am I misunderstanding > something here). I still don't know and can neither conclude which > PG the warning originates from. As far as I can tell, the warning > should not be there. > > Do you have an idea how to continue diagnosis from here apart from > just trying a deep scrub on all PGs in the list from the log? > > Thanks and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Eugen Block <eblock@xxxxxx<mailto:eblock@xxxxxx>> > Sent: Monday, October 16, 2023 1:41 PM > To: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> > Subject: Re: find PG with large omap object > > Hi, > not sure if this is what you need, but if you know the pool id (you > probably should) you could try this, it's from an Octopus test cluster > (assuming the warning was for the number of keys, not bytes): > > $ ceph -f json pg dump pgs 2>/dev/null | jq -r '.pg_stats[] | select > (.pgid | startswith("17.")) | .pgid + " " + > "\(.stat_sum.num_omap_keys)"' > 17.6 191 > 17.7 759 > 17.4 358 > 17.5 0 > 17.2 177 > 17.3 1 > 17.0 375 > 17.1 176 > > If you don't know the pool you could sort the ouput by the second > column and see which PG has the largest number of omap_keys. > > Regards, > Eugen > > Zitat von Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>>: > >> Hi all, >> >> we had a bunch of large omap object warnings after a user deleted a >> lot of files on a ceph fs with snapshots. After the snapshots were >> rotated out, all but one of these warnings disappeared over time. >> However, one warning is stuck and I wonder if its something else. >> >> Is there a reasonable way (say, one-liner with no more than 120 >> characters) to get ceph to tell me which PG this is coming from? I >> just want to issue a deep scrub to check if it disappears and going >> through the logs and querying every single object for its key count >> seems a bit of a hassle for something that ought to be part of "ceph >> health detail". >> >> Best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |