The deep scrub of the pg updated the cluster that the large omap was gone. HEALTH_OK !
On Fri., Sep. 20, 2019, 2:31 p.m. shubjero, <shubjero@xxxxxxxxx> wrote:
Still trying to solve this one.
Here is the corresponding log entry when the large omap object was found:
ceph-osd.1284.log.2.gz:2019-09-18 11:43:39.237 7fcd68f96700 0
log_channel(cluster) log [WRN] : Large omap object found. Object:
26:86e4c833:::usage.22:head Key count: 2009548 Size (bytes): 369641376
I have since trimmed the entire usage log and disabled it entirely.
You can see from the output below that there's nothing in these usage
log objects.
for i in `rados -p .usage ls`; do echo $i; rados -p .usage
listomapkeys $i | wc -l; done
usage.29
0
usage.12
0
usage.1
0
usage.26
0
usage.20
0
usage.24
0
usage.16
0
usage.15
0
usage.3
0
usage.19
0
usage.23
0
usage.5
0
usage.11
0
usage.7
0
usage.30
0
usage.18
0
usage.21
0
usage.27
0
usage.13
0
usage.22
0
usage.25
0
.
4
usage.10
0
usage.8
0
usage.9
0
usage.28
0
usage.2
0
usage.4
0
usage.6
0
usage.31
0
usage.17
0
root@infra:~# rados -p .usage listomapkeys usage.22
root@infra:~#
On Thu, Sep 19, 2019 at 12:54 PM Charles Alva <charlesalva@xxxxxxxxx> wrote:
>
> Could you please share how you trimmed the usage log?
>
> Kind regards,
>
> Charles Alva
> Sent from Gmail Mobile
>
>
> On Thu, Sep 19, 2019 at 11:46 PM shubjero <shubjero@xxxxxxxxx> wrote:
>>
>> Hey all,
>>
>> Yesterday our cluster went in to HEALTH_WARN due to 1 large omap
>> object in the .usage pool (I've posted about this in the past). Last
>> time we resolved the issue by trimming the usage log below the alert
>> threshold but this time it seems like the alert wont clear even after
>> trimming and (this time) disabling the usage log entirely.
>>
>> ceph health detail
>> HEALTH_WARN 1 large omap objects
>> LARGE_OMAP_OBJECTS 1 large omap objects
>> 1 large objects found in pool '.usage'
>> Search the cluster log for 'Large omap object found' for more details.
>>
>> I've bounced ceph-mon, ceph-mgr, radosgw and even issued osd scrub on
>> the two osd's that hold pg's for the .usage pool but the alert wont
>> clear.
>>
>> It's been over 24 hours since I trimmed the usage log.
>>
>> Any suggestions?
>>
>> Jared Baker
>> Cloud Architect, OICR
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
