Do you see *keys* aka omap traffic? Especially if you have RGW set up? > On Apr 24, 2024, at 15:37, David Orman <ormandj@xxxxxxxxxxxx> wrote: > > Did you ever figure out what was happening here? > > David > > On Mon, May 29, 2023, at 07:16, Hector Martin wrote: >> On 29/05/2023 20.55, Anthony D'Atri wrote: >>> Check the uptime for the OSDs in question >> >> I restarted all my OSDs within the past 10 days or so. Maybe OSD >> restarts are somehow breaking these stats? >> >>> >>>> On May 29, 2023, at 6:44 AM, Hector Martin <marcan@xxxxxxxxx> wrote: >>>> >>>> Hi, >>>> >>>> I'm watching a cluster finish a bunch of backfilling, and I noticed that >>>> quite often PGs end up with zero misplaced objects, even though they are >>>> still backfilling. >>>> >>>> Right now the cluster is down to 6 backfilling PGs: >>>> >>>> data: >>>> volumes: 1/1 healthy >>>> pools: 6 pools, 268 pgs >>>> objects: 18.79M objects, 29 TiB >>>> usage: 49 TiB used, 25 TiB / 75 TiB avail >>>> pgs: 262 active+clean >>>> 6 active+remapped+backfilling >>>> >>>> But there are no misplaced objects, and the misplaced column in `ceph pg >>>> dump` is zero for all PGs. >>>> >>>> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >>>> increasing for these PGs... but the misplaced count is still 0. >>>> >>>> Is there something else that would cause recoveries/backfills other than >>>> misplaced objects? Or perhaps there is a bug somewhere causing the >>>> misplaced object count to be misreported as 0 sometimes? >>>> >>>> # ceph -v >>>> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >>>> (stable) >>>> >>>> - Hector >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >>> >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |