Are those three different buckets? Could you share the stats for each of them?
radosgw-admin bucket stats --bucket=<BUCKET>
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>:
Hello,
Yeah, still:
the .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
290005
and the
.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726 | wc -l
289378
And just make me happy more I have one more
.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.6 | wc -l
181588
This is my crush tree (I'm using host based crush rule)
https://gist.githubusercontent.com/Badb0yBadb0y/9bea911701184a51575619bc99cca94d/raw/e5e4a918d327769bb874aaed279a8428fd7150d5/gistfile1.txt
I'm thinking could that be the issue that host 2s13-15 has less nvme
osd (however size wise same as in the other 12 host where have 8x
nvme osd) than the others?
But the pgs are located like this:
pg26.427
osd.261 host8
osd.488 host13
osd.276 host4
pg26.606
osd.443 host12
osd.197 host8
osd.524 host14
pg26.78c
osd.89 host7
osd.406 host11
osd.254 host6
If pg26.78c wouldn't be here I'd say 100% the nvme osd distribution
based on host is the issue, however this pg is not located on any of
the 4x nvme osd nodes 😕
Ty
________________________________
From: Eugen Block <eblock@xxxxxx>
Sent: Tuesday, July 9, 2024 6:02 PM
To: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: Re: Large omap in index pool even if properly
sharded and not "OVER"
Email received from the internet. If in doubt, don't click any link
nor open any attachment !
________________________________
Hi,
the number of shards looks fine, maybe this was just a temporary
burst? Did you check if the rados objects in the index pool still have
more than 200k omap objects? I would try someting like
rados -p <index_pool> listomapkeys
.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151 | wc -l
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>:
Hi,
I have a pretty big bucket which sharded with 1999 shard so in
theory can hold close to 200m objects (199.900.000).
Currently it has 54m objects.
Bucket limit check looks also good:
"bucket": ""xyz,
"tenant": "",
"num_objects": 53619489,
"num_shards": 1999,
"objects_per_shard": 26823,
"fill_status": "OK"
This is the bucket id:
"id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1"
This is the log lines:
2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster
[WRN] Large omap object found. Object:
26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size
(bytes):
89969920
2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN]
Large omap object found. Object:
26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size
(bytes):
95560458
Tried to deep scrub the affected pgs, tried to deep-scrub the
mentioned osds in the log but didn't help.
Why? What I'm missing?
Thank you in advance for your help.
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to
anyone. Any confidentiality or privilege is not waived or lost by
any mistaken delivery or unauthorized disclosure of the message. All
messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests
and to remove potential malware. Electronic messages may be
intercepted, amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to
anyone. Any confidentiality or privilege is not waived or lost by
any mistaken delivery or unauthorized disclosure of the message. All
messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests
and to remove potential malware. Electronic messages may be
intercepted, amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
