Hello, Yeah, still: the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l 290005 and the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l 289378 And just make me happy more I have one more .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l 181588 This is my crush tree (I'm using host based crush rule) https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cca94d/raw/e5e4a918d327769bb874aaed279a8428fd7150d5/gistfile1.txt I'm thinking could that be the issue that host 2s13-15 has less nvme osd (however size wise same as in the other 12 host where have 8x nvme osd) than the others? But the pgs are located like this: pg26.427 osd.261 host8 osd.488 host13 osd.276 host4 pg26.606 osd.443 host12 osd.197 host8 osd.524 host14 pg26.78c osd.89 host7 osd.406 host11 osd.254 host6 If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution based on host is the issue, however this pg is not located on any of the 4x nvme osd nodes 😕 Ty ________________________________ From: Eugen Block <eblock@xxxxxx> Sent: Tuesday, July 9, 2024 6:02 PM To: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: Large omap in index pool even if properly sharded and not "OVER" Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Hi, the number of shards looks fine, maybe this was just a temporary burst? Did you check if the rados objects in the index pool still have more than 200k omap objects? I would try someting like rados -p <index_pool> listomapkeys .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>: > Hi, > > I have a pretty big bucket which sharded with 1999 shard so in > theory can hold close to 200m objects (199.900.000). > Currently it has 54m objects. > > Bucket limit check looks also good: > "bucket": ""xyz, > "tenant": "", > "num_objects": 53619489, > "num_shards": 1999, > "objects_per_shard": 26823, > "fill_status": "OK" > > This is the bucket id: > "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1" > > This is the log lines: > 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster > [WRN] Large omap object found. Object: > 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): > 89969920 > > 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] > Large omap object found. Object: > 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): > 95560458 > > Tried to deep scrub the affected pgs, tried to deep-scrub the > mentioned osds in the log but didn't help. > Why? What I'm missing? > > Thank you in advance for your help. > > ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by > copyright or other legal rules. If you have received it by mistake > please let us know by reply email and delete it from your system. It > is prohibited to copy this message or disclose its content to > anyone. Any confidentiality or privilege is not waived or lost by > any mistaken delivery or unauthorized disclosure of the message. All > messages sent to and from Agoda may be monitored to ensure > compliance with company policies, to protect the company's interests > and to remove potential malware. Electronic messages may be > intercepted, amended, lost or deleted, or contain viruses. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |