Hi Matt and Dan, I too suspect it’s the issue Matt linked to. That bug only affects versioned buckets, so I’m guessing your bucket is versioned, Dan. This bug is triggered when the final instance of an object in a versioned bucket is deleted, but for reasons we do not yet understand, the object was not fully deleted from the bucket index. And then a reshard moves part of the object index to shard 0. Upgrading to a version that included Casey’s fix would mean this situation is not re-created in the future. An automated clean-up is non-trivial but feasible. It would have to take into account that an object with the same name as the previously deleted one was re-created in the versioned bucket. Eric > On Oct 1, 2020, at 8:46 AM, Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: > > Hi Dan, > > Possibly you're reproducing https://tracker.ceph.com/issues/46456. > > That explains how the underlying issue worked, I don't remember how a > bucked exhibiting this is repaired. > > Eric? > > Matt > > > On Thu, Oct 1, 2020 at 8:41 AM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: >> >> Dear friends, >> >> Running 14.2.11, we have one particularly large bucket with a very >> strange distribution of objects among the shards. The bucket has 512 >> shards, and most shards have ~75k entries, but shard 0 has 1.75M >> entries: >> >> # rados -p default.rgw.buckets.index listomapkeys >> .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.0 | wc -l >> 1752085 >> >> # rados -p default.rgw.buckets.index listomapkeys >> .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.1 | wc -l >> 78388 >> >> # rados -p default.rgw.buckets.index listomapkeys >> .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.2 | wc -l >> 78764 >> >> We had resharded this bucket (manually) from 32 up to 512 shards just >> before upgrading from 12.2.12 to 14.2.11 a couple weeks ago. >> >> Any idea why shard .0 is getting such an imbalance of entries? >> Should we manually reshard this bucket again? >> >> Thanks! >> >> Dan >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |