Is your client using the DeleteObjects call to delete 1000 per request?: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html On Fri, Jun 3, 2022 at 9:35 AM J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx> wrote: > Read/writes are super fast. It's only deletes that are incredibly slow, > both through the s3 api and radosgw commands. It doesn't have a lot of > index shards with 11 and it basically has one bucket with huge objects. > We're trying to store backups in there, essentially. New backups write > themselves about 100 times faster than old backups can be deleted. > > On 6/2/22 19:15, Wesley Dillingham wrote: > > Is it just your deletes which are slow or writes and read as well? > > > > On Thu, Jun 2, 2022, 4:09 PM J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx> > > wrote: > > > > I'm following up on this as we upgraded to Pacific 16.2.9 and deletes > > are still incredibly slow. The pool rgw is using is a fairly small > > erasure coding pool set at 8 + 3. Is there anyone who's having the > > same > > issue? > > > > On 5/16/22 15:23, J-P Methot wrote: > > > Hi, > > > > > > First of all, a quick google search shows me that questions > > about the > > > s3 API slow object deletion speed have been asked before and are > > well > > > documented. My issue is slightly different, because I am getting > > > abysmal speeds of 11 objects/second on a full SSD ceph running > > Octopus > > > with about a hundred OSDs. This is much lower than the Redhat > > reported > > > limit of 1000 objects/second. > > > > > > I've seen elsewhere that it was a Rocksdb limitation and that it > > would > > > be fixed in Pacific, but the Pacific release logs do not show me > > > anything that suggest that. Furthermore, I have limited control > > over > > > the s3client deleting the files as it's a 3rd-party open source > > > automatic backup program. > > > > > > Could updating to Pacific fix this issue? Is there any > > configuration > > > change I could do to speed up object deletion? > > > > > -- > > Jean-Philippe Méthot > > Senior Openstack system administrator > > Administrateur système Openstack sénior > > PlanetHoster inc. > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > -- > Jean-Philippe Méthot > Senior Openstack system administrator > Administrateur système Openstack sénior > PlanetHoster inc. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |