Still trying to solve this one. Here is the corresponding log entry when the large omap object was found: ceph-osd.1284.log.2.gz:2019-09-18 11:43:39.237 7fcd68f96700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 26:86e4c833:::usage.22:head Key count: 2009548 Size (bytes): 369641376 I have since trimmed the entire usage log and disabled it entirely. You can see from the output below that there's nothing in these usage log objects. for i in `rados -p .usage ls`; do echo $i; rados -p .usage listomapkeys $i | wc -l; done usage.29 0 usage.12 0 usage.1 0 usage.26 0 usage.20 0 usage.24 0 usage.16 0 usage.15 0 usage.3 0 usage.19 0 usage.23 0 usage.5 0 usage.11 0 usage.7 0 usage.30 0 usage.18 0 usage.21 0 usage.27 0 usage.13 0 usage.22 0 usage.25 0 . 4 usage.10 0 usage.8 0 usage.9 0 usage.28 0 usage.2 0 usage.4 0 usage.6 0 usage.31 0 usage.17 0 root@infra:~# rados -p .usage listomapkeys usage.22 root@infra:~# On Thu, Sep 19, 2019 at 12:54 PM Charles Alva <charlesalva@xxxxxxxxx> wrote: > > Could you please share how you trimmed the usage log? > > Kind regards, > > Charles Alva > Sent from Gmail Mobile > > > On Thu, Sep 19, 2019 at 11:46 PM shubjero <shubjero@xxxxxxxxx> wrote: >> >> Hey all, >> >> Yesterday our cluster went in to HEALTH_WARN due to 1 large omap >> object in the .usage pool (I've posted about this in the past). Last >> time we resolved the issue by trimming the usage log below the alert >> threshold but this time it seems like the alert wont clear even after >> trimming and (this time) disabling the usage log entirely. >> >> ceph health detail >> HEALTH_WARN 1 large omap objects >> LARGE_OMAP_OBJECTS 1 large omap objects >> 1 large objects found in pool '.usage' >> Search the cluster log for 'Large omap object found' for more details. >> >> I've bounced ceph-mon, ceph-mgr, radosgw and even issued osd scrub on >> the two osd's that hold pg's for the .usage pool but the alert wont >> clear. >> >> It's been over 24 hours since I trimmed the usage log. >> >> Any suggestions? >> >> Jared Baker >> Cloud Architect, OICR >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |