Hi, Mysteriously, the large omap objects alert recurred recently. The values for omap_used_mbytes and omap_used_keys are slightly different from the previous investigation, but very close. Our team is going to keep this cluster to investigate and create another cluster to work. Therefore, my reply may be slow. Previous values: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/TNQM2W4EDG3J33W7CML2JLCDNFDA6Q3W/ ``` $ kubectl exec -n ceph-poc deploy/rook-ceph-tools -- ceph -s cluster: id: 49bd471e-84e6-412e-8ed0-75d7bc176657 health: HEALTH_WARN 25 large omap objects services: mon: 3 daemons, quorum b,d,f (age 36h) mgr: b(active, since 38h), standbys: a osd: 96 osds: 96 up (since 31h), 96 in (since 31h) rgw: 6 daemons active (6 hosts, 2 zones) data: pools: 16 pools, 4432 pgs objects: 10.74k objects, 34 GiB usage: 158 GiB used, 787 TiB / 787 TiB avail pgs: 4432 active+clean io: client: 2.2 KiB/s rd, 169 B/s wr, 2 op/s rd, 0 op/s wr ``` ``` $ (header="id used_mbytes used_objects omap_used_mbytes omap_used_keys" > echo "${header}" > echo "${header}" | tr '[[:alpha:]_' '-' > kubectl exec -n ceph-poc deploy/rook-ceph-tools -- ceph pg ls-by-pool "${OSD_POOL}" --format=json | jq -r '.pg_stats | > sort_by(.stat_sum.num_bytes) | .[] | (.pgid, .stat_sum.num_bytes/1024/1024, > .stat_sum.num_objects, .stat_sum.num_omap_bytes/1024/1024, > .stat_sum.num_omap_keys)' | paste - - - - -) | column -t id used_mbytes used_objects omap_used_mbytes omap_used_keys -- ----------- ------------ ---------------- -------------- 6.0 0 0 0 0 6.1 0 0 0 0 6.2 0 0 86.14682674407959 298586 6.3 0 0 93.08089542388916 323902 6.4 0 1 0 0 6.5 0 1 0 0 6.6 0 0 0 0 6.7 0 0 0 0 6.8 0 0 0 0 6.9 0 0 439.5090618133545 1524746 6.a 0 0 0 0 6.b 0 0 3.4069366455078125 12416 6.c 0 0 0 0 6.d 0 0 0 0 6.e 0 0 0 0 6.f 0 1 0 0 6.10 0 1 0 0 6.11 0 0 0 0 6.12 0 0 7.727175712585449 28160 6.13 0 0 114.01904964447021 394996 6.14 0 0 0 0 6.15 0 0 0 0 6.16 0 0 0 0 6.17 0 0 7.6217451095581055 27776 6.18 0 0 0 0 6.19 0 1 0 0 6.1a 0 1 0 0 6.1b 0 0 0 0 6.1c 0 0 88.36568355560303 306677 6.1d 0 0 0 0 6.1e 0 1 0 0 6.1f 0 0 0 0 6.20 0 1 0 0 6.21 0 0 0 0 6.22 0 0 5.883256912231445 21440 6.23 0 0 0 0 6.24 0 0 7.938144683837891 28928 6.25 0 0 0 0 6.26 0 0 4.267669677734375 15552 6.27 0 1 0 0 6.28 0 0 0 0 6.29 0 0 2.1601409912109375 7872 6.2a 0 1 0 0 6.2b 0 0 0 0 6.2c 0 0 5.479369163513184 19968 6.2d 0 0 0 0 6.2e 0 0 0 0 6.2f 0 0 0 0 6.30 0 0 0 0 6.31 0 1 0 0 6.32 0 1 0 0 6.33 0 0 5.812973976135254 21184 6.34 0 0 0 0 6.35 0 0 0 0 6.36 0 0 5.865510940551758 21376 6.37 0 0 0 0 6.38 0 0 93.97305393218994 327089 6.39 0 0 15.493829727172852 71787 6.3a 0 0 0 0 6.3b 0 0 4.056745529174805 14784 6.3c 0 0 4.039289474487305 14720 6.3d 0 0 0 0 6.3e 0 0 0 0 6.3f 0 0 0 0 6.40 0 0 0 0 6.41 0 1 4.004250526428223 14592 6.42 0 0 0 0 6.43 0 0 345.3690414428711 1197068 6.44 0 0 0 0 6.45 0 1 0 0 6.46 0 0 0 0 6.47 0 0 0 0 6.48 0 0 0 0 6.49 0 0 263.9479990005493 914805 6.4a 0 0 94.751708984375 336275 6.4b 0 0 7.604428291320801 27712 6.4c 0 0 0 0 6.4d 0 0 270.53627490997314 937581 6.4e 0 1 0 0 6.4f 0 0 11.229312896728516 55949 6.50 0 0 1.8790569305419922 6848 6.51 0 1 0 0 6.52 0 0 4.05673885345459 14784 6.53 0 0 0 0 6.54 0 0 42.36539554595947 146958 6.55 0 0 0 0 6.56 0 0 27.734545707702637 133186 6.57 0 0 0 0 6.58 0 0 0 0 6.59 0 0 1.4927978515625 5440 6.5a 0 0 3.775785446166992 13760 6.5b 0 0 3.722942352294922 13568 6.5c 0 0 0 0 6.5d 0 1 0 0 6.5e 0 0 0 0 6.5f 0 0 0 0 6.60 0 0 0 0 6.61 0 0 5.865522384643555 21376 6.62 0 0 0 0 6.63 0 0 0 0 6.64 0 0 14.687395095825195 67716 6.65 0 0 4.021788597106934 14656 6.66 0 1 0 0 6.67 0 0 0 0 6.68 0 0 0 0 6.69 0 0 13.16762924194336 65594 6.6a 0 0 7.709336280822754 28096 6.6b 0 0 0 0 6.6c 0 0 1.7913312911987305 6528 6.6d 0 0 0 0 6.6e 0 0 1.8790216445922852 6848 6.6f 0 0 0 0 6.70 0 3 0 0 6.71 0 0 6.041498184204102 22016 6.72 0 0 1.8790807723999023 6848 6.73 0 0 1.8790874481201172 6848 6.74 0 0 0 0 6.75 0 0 170.30711841583252 590287 6.76 0 0 5.848110198974609 21312 6.77 0 0 86.37962532043457 299393 6.78 0 0 2.1776771545410156 7936 6.79 0 0 2.2303342819213867 8128 6.7a 0 1 0 0 6.7b 0 0 0 0 6.7c 0 0 90.31485748291016 313783 6.7d 0 0 0 0 6.7e 0 0 0 0 6.7f 0 0 12.507318496704102 62305 ``` Thanks, Yuji ________________________________ From: Yuji Ito (伊藤 祐司) <yuji-ito@xxxxxxxxxxxx> Sent: Tuesday, October 25, 2022 10:33 To: Konstantin Shalygin <k0ste@xxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: How to remove remaining bucket index shard objects Hi, The large omap alert looks resolved last week, Although I don't know the underlying reasons. When I got your email and tried to get the data, I noticed that the alerts had stopped. OMAP was 0 Bytes as follows. To make sure, I ran a deep scrub and waited for a while, but the alert has not recurred until now. Before the alerts stopped, the other team restarted the node where OSD and other modules were running for maintenance, which may have had an impact. However, reboots are done every week and have been done three times after compaction. It is therefore uncertain as to the root reason. There is a possibility of recurrence, so I will take a wait-and-see approach. ``` $ kubectl exec -n ceph-poc deploy/rook-ceph-tools -- ceph -s cluster: id: 49bd471e-84e6-412e-8ed0-75d7bc176657 health: HEALTH_OK services: mon: 3 daemons, quorum b,d,f (age 4d) mgr: b(active, since 4d), standbys: a osd: 96 osds: 96 up (since 4d), 96 in (since 4d) rgw: 6 daemons active (6 hosts, 2 zones) data: pools: 16 pools, 4432 pgs objects: 10.19k objects, 34 GiB usage: 161 GiB used, 787 TiB / 787 TiB avail pgs: 4432 active+clean io: client: 3.1 KiB/s rd, 931 B/s wr, 3 op/s rd, 2 op/s wr $ OSD_POOL=ceph-poc-object-store-ssd-index.rgw.buckets.index $ (header="id used_mbytes used_objects omap_used_mbytes omap_used_keys" > echo "${header}" > echo "${header}" | tr '[[:alpha:]_' '-' > kubectl exec -n ceph-poc deploy/rook-ceph-tools -- ceph pg ls-by-pool "${OSD_POOL}" --format=json | jq -r '.pg_stats | > sort_by(.stat_sum.num_bytes) | .[] | (.pgid, .stat_sum.num_bytes/1024/1024, > .stat_sum.num_objects, .stat_sum.num_omap_bytes/1024/1024, > .stat_sum.num_omap_keys)' | paste - - - - -) | column -t id used_mbytes used_objects omap_used_mbytes omap_used_keys -- ----------- ------------ ---------------- -------------- 6.0 0 0 0 0 6.1 0 0 0 0 6.2 0 0 0 0 6.3 0 0 0 0 6.4 0 1 0 0 6.5 0 1 0 0 6.6 0 0 0 0 6.7 0 0 0 0 6.8 0 0 0 0 6.9 0 0 0 0 6.a 0 0 0 0 6.b 0 0 0 0 6.c 0 0 0 0 6.d 0 0 0 0 6.e 0 0 0 0 6.f 0 1 0 0 6.10 0 1 0 0 6.11 0 0 0 0 6.12 0 0 0 0 6.13 0 0 0 0 6.14 0 0 0 0 6.15 0 0 0 0 6.16 0 0 0 0 6.17 0 0 0 0 6.18 0 0 0 0 6.19 0 1 0 0 6.1a 0 1 0 0 6.1b 0 0 0 0 6.1c 0 0 0 0 6.1d 0 0 0 0 6.1e 0 1 0 0 6.1f 0 0 0 0 6.20 0 1 0 0 6.21 0 0 0 0 6.22 0 0 0 0 6.23 0 0 0 0 6.24 0 0 0 0 6.25 0 0 0 0 6.26 0 0 0 0 6.27 0 1 0 0 6.28 0 0 0 0 6.29 0 0 0 0 6.2a 0 1 0 0 6.2b 0 0 0 0 6.2c 0 0 0 0 6.2d 0 0 0 0 6.2e 0 0 0 0 6.2f 0 0 0 0 6.30 0 0 0 0 6.31 0 1 0 0 6.32 0 1 0 0 6.33 0 0 0 0 6.34 0 0 0 0 6.35 0 0 0 0 6.36 0 0 0 0 6.37 0 0 0 0 6.38 0 0 0 0 6.39 0 0 0 0 6.3a 0 0 0 0 6.3b 0 0 0 0 6.3c 0 0 0 0 6.3d 0 0 0 0 6.3e 0 0 0 0 6.3f 0 0 0 0 6.40 0 0 0 0 6.41 0 1 0 0 6.42 0 0 0 0 6.43 0 0 0 0 6.44 0 0 0 0 6.45 0 1 0 0 6.46 0 0 0 0 6.47 0 0 0 0 6.48 0 0 0 0 6.49 0 0 0 0 6.4a 0 0 0 0 6.4b 0 0 0 0 6.4c 0 0 0 0 6.4d 0 0 0 0 6.4e 0 1 0 0 6.4f 0 0 0 0 6.50 0 0 0 0 6.51 0 1 0 0 6.52 0 0 0 0 6.53 0 0 0 0 6.54 0 0 0 0 6.55 0 0 0 0 6.56 0 0 0 0 6.57 0 0 0 0 6.58 0 0 0 0 6.59 0 0 0 0 6.5a 0 0 0 0 6.5b 0 0 0 0 6.5c 0 0 0 0 6.5d 0 1 0 0 6.5e 0 0 0 0 6.5f 0 0 0 0 6.60 0 0 0 0 6.61 0 0 0 0 6.62 0 0 0 0 6.63 0 0 0 0 6.64 0 0 0 0 6.65 0 0 0 0 6.66 0 1 0 0 6.67 0 0 0 0 6.68 0 0 0 0 6.69 0 0 0 0 6.6a 0 0 0 0 6.6b 0 0 0 0 6.6c 0 0 0 0 6.6d 0 0 0 0 6.6e 0 0 0 0 6.6f 0 0 0 0 6.70 0 3 0 0 6.71 0 0 0 0 6.72 0 0 0 0 6.73 0 0 0 0 6.74 0 0 0 0 6.75 0 0 0 0 6.76 0 0 0 0 6.77 0 0 0 0 6.78 0 0 0 0 6.79 0 0 0 0 6.7a 0 1 0 0 6.7b 0 0 0 0 6.7c 0 0 0 0 6.7d 0 0 0 0 6.7e 0 0 0 0 6.7f 0 0 0 0 ``` Thanks, Yuji From: Konstantin Shalygin <k0ste@xxxxxxxx> Sent: Wednesday, October 19, 2022 16:42 To: Yuji Ito (伊藤 祐司) <yuji-ito@xxxxxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: How to remove remaining bucket index shard objects This strange stats, at least one object should be exists for this OMAP's. Try to deep-scrub this PG, try to list objects in this PG `rados ls --pgid 6.2` k Sent from my iPhone _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |