Hi,
Unfortunately, the "large omap objects" message recurred last weekend. So I ran the script you showed to check the situation. `used_.*` is small, but `omap_.*` is large, which is strange. Do you have any idea what it is?
id used_mbytes used_objects omap_used_mbytes omap_used_keys
-- ----------- ------------ ---------------- --------------
6.0 0 0 0 0
6.1 0 0 0 0
6.2 0 0 86.14682674407959 298586
6.3 0 0 93.08089542388916 323902
6.4 0 1 0 0
6.5 0 1 2.124929428100586 7744
6.6 0 0 0 0
6.7 0 0 2.2477407455444336 8192
6.8 0 0 0 0
6.9 0 0 439.5090618133545 1524746
6.a 0 0 0 0
6.b 0 0 3.4069366455078125 12416
6.c 0 0 0 0
6.d 0 0 0 0
6.e 0 0 0 0
6.f 0 1 0 0
6.10 0 1 2.177792549133301 7936
6.11 0 0 3.9340572357177734 14336
6.12 0 0 7.727175712585449 28160
6.13 0 0 114.01904964447021 394996
6.14 0 0 0 0
6.15 0 0 88.56490707397461 307353
6.16 0 0 0 0
6.17 0 0 7.6217451095581055 27776
6.18 0 0 3.933901786804199 14336
6.19 0 1 0 0
6.1a 0 1 0 0
6.1b 0 0 0 0
6.1c 0 0 88.36568355560303 306677
6.1d 0 0 0 0
6.1e 0 1 0 0
6.1f 0 0 92.21501541137695 320707
6.20 0 1 2.1074790954589844 7680
6.21 0 0 0 0
6.22 0 0 0 0
6.23 0 0 8.605427742004395 31360
6.24 0 0 7.938144683837891 28928
6.25 0 0 0 0
6.26 0 0 0 0
6.27 0 1 2.10748291015625 7680
6.28 0 0 0 0
6.29 0 0 2.1601409912109375 7872
6.2a 0 1 0 0
6.2b 0 0 0 0
6.2c 0 0 5.479369163513184 19968
6.2d 0 0 0 0
6.2e 0 0 0 0
6.2f 0 0 0 0
6.30 0 0 117.55222415924072 407521
6.31 0 1 0 0
6.32 0 1 0 0
6.33 0 0 5.812973976135254 21184
6.34 0 0 0 0
6.35 0 0 0 0
6.36 0 0 5.865510940551758 21376
6.37 0 0 86.26362419128418 298993
6.38 0 0 93.97305393218994 327089
6.39 0 0 15.493829727172852 71787
6.3a 0 0 0 0
6.3b 0 0 4.056745529174805 14784
6.3c 0 0 4.039289474487305 14720
6.3d 0 0 0 0
6.3e 0 0 0 0
6.3f 0 0 0 0
6.40 0 0 2.1073970794677734 7680
6.41 0 1 4.004250526428223 14592
6.42 0 0 3.9866724014282227 14528
6.43 0 0 345.3690414428711 1197068
6.44 0 0 0 0
6.45 0 1 0 0
6.46 0 0 3.968973159790039 14464
6.47 0 0 0 0
6.48 0 0 0 0
6.49 0 0 263.9479990005493 914805
6.4a 0 0 94.751708984375 336275
6.4b 0 0 0 0
6.4c 0 0 0 0
6.4d 0 0 270.53627490997314 937581
6.4e 0 1 0 0
6.4f 0 0 0 0
6.50 0 0 1.8790569305419922 6848
6.51 0 1 0 0
6.52 0 0 4.05673885345459 14784
6.53 0 0 0 0
6.54 0 0 0 0
6.55 0 0 0 0
6.56 0 0 27.734545707702637 133186
6.57 0 0 0 0
6.58 0 0 0 0
6.59 0 0 1.4927978515625 5440
6.5a 0 0 3.775785446166992 13760
6.5b 0 0 3.722942352294922 13568
6.5c 0 0 27.70587921142578 132115
6.5d 0 1 0 0
6.5e 0 0 5.848148345947266 21312
6.5f 0 0 2.125082015991211 7744
6.60 0 0 0 0
6.61 0 0 5.865522384643555 21376
6.62 0 0 0 0
6.63 0 0 0 0
6.64 0 0 14.687395095825195 67716
6.65 0 0 4.021788597106934 14656
6.66 0 1 0 0
6.67 0 0 0 0
6.68 0 0 86.15152931213379 298605
6.69 0 0 13.16762924194336 65594
6.6a 0 0 7.709336280822754 28096
6.6b 0 0 1.8440046310424805 6720
6.6c 0 0 1.7913312911987305 6528
6.6d 0 0 0 0
6.6e 0 0 1.8790216445922852 6848
6.6f 0 0 0 0
6.70 0 3 0 0
6.71 0 0 6.041498184204102 22016
6.72 0 0 1.8790807723999023 6848
6.73 0 0 1.8790874481201172 6848
6.74 0 0 0 0
6.75 0 0 170.30711841583252 590287
6.76 0 0 5.848110198974609 21312
6.77 0 0 86.37962532043457 299393
6.78 0 0 2.1776771545410156 7936
6.79 0 0 2.2303342819213867 8128
6.7a 0 1 0 0
6.7b 0 0 0 0
6.7c 0 0 0 0
6.7d 0 0 0 0
6.7e 0 0 0 0
6.7f 0 0 12.507318496704102 62305
Thanks,
Yuji
From: Konstantin Shalygin <k0ste@xxxxxxxx>
Sent: Tuesday, October 4, 2022 16:39
To: Yuji Ito (伊藤 祐司) <yuji-ito@xxxxxxxxxxxx>
Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: Re: How to remove remaining bucket index shard objects
Hi,
On 4 Oct 2022, at 03:36, Yuji Ito (伊藤 祐司) <yuji-ito@xxxxxxxxxxxx> wrote:
After removing the index objects, I ran deep-scrub for all PGs of the index pool. However, the problem wasn't resolved.
Seems you just have large OMAPs, not because 'bogus shard' objects. Try to look PG stats with 'show_osd_pool_pg_usage.sh' script:
#!/bin/bash
OSD_POOL="$1"
if [ -z "${OSD_POOL}" ]
then
echo -e "Please provide osd pool name as first arg"
exit 1
fi
(header="id used_mbytes used_objects omap_used_mbytes omap_used_keys"
echo "${header}"
echo "${header}" | tr '[[:alpha:]_' '-'
ceph pg ls-by-pool "${OSD_POOL}" --format=json | jq -r '.pg_stats |
sort_by(.stat_sum.num_bytes) | .[] | (.pgid, .stat_sum.num_bytes/1024/1024,
.stat_sum.num_objects, .stat_sum.num_omap_bytes/1024/1024,
.stat_sum.num_omap_keys)' | paste - - - - -) | column -t
k
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
