Hi Danish, Can you access this KB article [1]? A free developer account should allow you to. It pretty much describes what you're facing and suggests to trim the sync error log of recovering shards. Actually, every log entry **except the last one**. 1/ Identify the shards with the most sync errors log entries: radosgw-admin sync error list --max-entries=1000000 | grep shard_id | sort -n | uniq -c | sort -h 2/ For each shard, list every sync error log entry along with their ids: radosgw-admin sync error list --shard-id=X 3/ Remove them **except the last one** with: radosgw-admin sync error trim --shard-id=X --marker=1_1682101321.201434_8669.1 the --marker above being the log entry id. Are the replication threads running on the same RGWs that S3 clients are using? If so, using dedicated RGWs for the sync job might help you avoid non-recovering shards in the future, as described in Matthew's post [2] Regards, Frédéric. [1] https://access.redhat.com/solutions/7023912 [2] https://www.spinics.net/lists/ceph-users/msg83988.html ----- Le 12 Mar 25, à 11:15, Danish Khan danish52.jmi@xxxxxxxxx a écrit : > Dear All, > > My ceph cluster is giving Large OMAP warning from approx 2-3 Months. I > tried a few things like : > *Deep scrub of PGs* > *Compact OSDs* > *Trim log* > But these didn't work out. > > I guess the main issue is that 4 shards in replication site are always > recovering from 2-3 months. > > Any suggestions are highly appreciated. > > Sync status: > root@drhost1:~# radosgw-admin sync status > realm e259e0a92 (object-storage) > zonegroup 7a8606d2 (staas) > zone c8022ad1 (repstaas) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 2072ee2d32 (masterstaas) > syncing > full sync: 0/128 shards > incremental sync: 128/128 shards > data is behind on 3 shards > behind shards: [7,90,100] > oldest incremental change not applied: > 2025-03-12T13:14:10.268469+0530 [7] > 4 shards are recovering > recovering shards: [31,41,55,80] > > > Master site: > 1. *root@master1:~# for obj in $(rados ls -p masterstaas.rgw.log); do echo > "$(rados listomapkeys -p masterstaas.rgw.log $obj | wc -l) $obj";done | > sort -nr | head -10* > 1225387 data_log.91 > 1225065 data_log.86 > 1224662 data_log.87 > 1224448 data_log.92 > 1224018 data_log.89 > 1222156 data_log.93 > 1201489 data_log.83 > 1174125 data_log.90 > 363498 data_log.84 > 258709 data_log.6 > > > 2. *root@master1:~# for obj in data_log.91 data_log.86 data_log.87 > data_log.92 data_log.89 data_log.93 data_log.83 data_log.90; do rados stat > -p masterstaas.rgw.log $obj; done* > masterstaas.rgw.log/data_log.91 mtime 2025-02-24T15:09:25.000000+0530, size > 0 > masterstaas.rgw.log/data_log.86 mtime 2025-02-24T15:01:25.000000+0530, size > 0 > masterstaas.rgw.log/data_log.87 mtime 2025-02-24T15:02:25.000000+0530, size > 0 > masterstaas.rgw.log/data_log.92 mtime 2025-02-24T15:11:01.000000+0530, size > 0 > masterstaas.rgw.log/data_log.89 mtime 2025-02-24T14:54:55.000000+0530, size > 0 > masterstaas.rgw.log/data_log.93 mtime 2025-02-24T14:53:25.000000+0530, size > 0 > masterstaas.rgw.log/data_log.83 mtime 2025-02-24T14:16:21.000000+0530, size > 0 > masterstaas.rgw.log/data_log.90 mtime 2025-02-24T15:05:25.000000+0530, size > 0 > > *3. ceph cluster log :* > 2025-02-22T04:18:27.324886+0530 osd.173 (osd.173) 19 : cluster [WRN] Large > omap object found. Object: 124:b2ddf551:::data_log.93:head PG: 124.8aafbb4d > (124.d) Key count: 1218170 Size (bytes): 297085860 > 2025-02-22T04:18:28.735886+0530 osd.65 (osd.65) 308 : cluster [WRN] Large > omap object found. Object: 124:f2081d70:::data_log.92:head PG: 124.eb8104f > (124.f) Key count: 1220420 Size (bytes): 295240028 > 2025-02-22T04:18:30.668884+0530 mon.master1 (mon.0) 7974038 : cluster [WRN] > Health check update: 3 large omap objects (LARGE_OMAP_OBJECTS) > 2025-02-22T04:18:31.127585+0530 osd.18 (osd.18) 224 : cluster [WRN] Large > omap object found. Object: 124:d1061236:::data_log.86:head PG: 124.6c48608b > (124.b) Key count: 1221047 Size (bytes): 295398274 > 2025-02-22T04:18:33.189561+0530 osd.37 (osd.37) 32665 : cluster [WRN] Large > omap object found. Object: 124:9a2e04b7:::data_log.87:head PG: 124.ed207459 > (124.19) Key count: 1220584 Size (bytes): 295290366 > 2025-02-22T04:18:35.007117+0530 osd.77 (osd.77) 135 : cluster [WRN] Large > omap object found. Object: 124:6b9e929a:::data_log.89:head PG: 124.594979d6 > (124.16) Key count: 1219913 Size (bytes): 295127816 > 2025-02-22T04:18:36.189141+0530 mon.master1 (mon.0) 7974039 : cluster [WRN] > Health check update: 5 large omap objects (LARGE_OMAP_OBJECTS) > 2025-02-22T04:18:36.340247+0530 osd.112 (osd.112) 259 : cluster [WRN] Large > omap object found. Object: 124:0958bece:::data_log.83:head PG: 124.737d1a90 > (124.10) Key count: 1200406 Size (bytes): 290280292 > 2025-02-22T04:18:38.523766+0530 osd.73 (osd.73) 1064 : cluster [WRN] Large > omap object found. Object: 124:fddd971f:::data_log.91:head PG: 124.f8e9bbbf > (124.3f) Key count: 1221183 Size (bytes): 295425320 > 2025-02-22T04:18:42.619926+0530 osd.92 (osd.92) 285 : cluster [WRN] Large > omap object found. Object: 124:7dc404fa:::data_log.90:head PG: 124.5f2023be > (124.3e) Key count: 1169895 Size (bytes): 283025576 > 2025-02-22T04:18:44.242655+0530 mon.master1 (mon.0) 7974043 : cluster [WRN] > Health check update: 8 large omap objects (LARGE_OMAP_OBJECTS) > > Replica site: > 1. *for obj in $(rados ls -p repstaas.rgw.log); do echo "$(rados > listomapkeys -p repstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head > -10* > > 432850 data_log.91 > 432384 data_log.87 > 432323 data_log.93 > 431783 data_log.86 > 431510 data_log.92 > 427959 data_log.89 > 414522 data_log.90 > 407571 data_log.83 > 151015 data_log.84 > 109790 data_log.4 > > > 2. *ceph cluster log:* > grep -ir "Large omap object found" /var/log/ceph/ > /var/log/ceph/ceph-mon.drhost1.log:2025-03-12T14:49:59.997+0530 > 7fc4ad544700 0 log_channel(cluster) log [WRN] : Search the cluster log > for 'Large omap object found' for more details. > /var/log/ceph/ceph.log:2025-03-12T14:49:02.078108+0530 osd.10 (osd.10) 21 : > cluster [WRN] Large omap object found. Object: > 6:b2ddf551:::data_log.93:head PG: 6.8aafbb4d (6.d) Key count: 432323 Size > (bytes): 105505884 > /var/log/ceph/ceph.log:2025-03-12T14:49:02.389288+0530 osd.48 (osd.48) 37 : > cluster [WRN] Large omap object found. Object: > 6:d1061236:::data_log.86:head PG: 6.6c48608b (6.b) Key count: 431782 Size > (bytes): 104564674 > /var/log/ceph/ceph.log:2025-03-12T14:49:07.166954+0530 osd.24 (osd.24) 24 : > cluster [WRN] Large omap object found. Object: > 6:0958bece:::data_log.83:head PG: 6.737d1a90 (6.10) Key count: 407571 Size > (bytes): 98635522 > /var/log/ceph/ceph.log:2025-03-12T14:49:09.100110+0530 osd.63 (osd.63) 5 : > cluster [WRN] Large omap object found. Object: > 6:9a2e04b7:::data_log.87:head PG: 6.ed207459 (6.19) Key count: 432384 Size > (bytes): 104712350 > /var/log/ceph/ceph.log:2025-03-12T14:49:08.703760+0530 osd.59 (osd.59) 11 : > cluster [WRN] Large omap object found. Object: > 6:6b9e929a:::data_log.89:head PG: 6.594979d6 (6.16) Key count: 427959 Size > (bytes): 103773777 > /var/log/ceph/ceph.log:2025-03-12T14:49:11.126132+0530 osd.40 (osd.40) 24 : > cluster [WRN] Large omap object found. Object: > 6:f2081d70:::data_log.92:head PG: 6.eb8104f (6.f) Key count: 431508 Size > (bytes): 104520406 > /var/log/ceph/ceph.log:2025-03-12T14:49:13.799473+0530 osd.43 (osd.43) 61 : > cluster [WRN] Large omap object found. Object: > 6:fddd971f:::data_log.91:head PG: 6.f8e9bbbf (6.1f) Key count: 432850 Size > (bytes): 104418869 > /var/log/ceph/ceph.log:2025-03-12T14:49:14.398480+0530 osd.3 (osd.3) 55 : > cluster [WRN] Large omap object found. Object: > 6:7dc404fa:::data_log.90:head PG: 6.5f2023be (6.1e) Key count: 414521 Size > (bytes): 100396561 > /var/log/ceph/ceph.log:2025-03-12T14:50:00.000484+0530 mon.drhost1 (mon.0) > 207423 : cluster [WRN] Search the cluster log for 'Large omap object > found' for more details. > > Regards, > Danish > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |