For mailing list archive readers in the future:
On Tue, Jul 9, 2019 at 1:22 PM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
Try to add "--inconsistent-index" (caution: will obviously leave your bucket in a broken state during the deletion, so don't try to use the bucket)
this was bad advice as long as https://tracker.ceph.com/issues/40700 is not fixed, don't do that.
You can also speed up the deletion with "--max-concurrent-ios" (default 32). The documentation incorrectly claims that "--max-concurrent-ios" is only for other operations but that's wrong, it is used for most bucket operations including deleteion.
this, however, is a good idea to speed up deletion of large buckets.
Try to combine the deletion command with timeout or something to not run into OOM all the time affecting other services.
(or use cgroups to limit RAM)
Paul
Paul--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90On Tue, Jul 9, 2019 at 1:11 PM Harald Staub <harald.staub@xxxxxxxxx> wrote:Currently removing a bucket with a lot of objects:
radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects
This process was killed by the out-of-memory killer. Then looking at the
graphs, we see a continuous increase of memory usage for this process,
about +24 GB per day. Removal rate is about 3 M objects per day.
It is not the fastest hardware, and this index pool is still without
SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now
about 500 OSDs.
So with this bucket with 60 M objects, we would need about 480 GB of RAM
to come through. Or is there a workaround? Should I open a tracker issue?
The killed remove command can just be called again, but it will be
killed again before it finishes. Also, it has to run some time until it
continues to actually remove objects. This "wait time" is also
increasing. Last time, after about 16 M objects already removed, the
wait time was nearly 9 hours. Also during this time, there is a memory
ramp, but not so steep.
BTW it feels strange that the removal of objects is slower (about 3
times) than adding objects.
Harry
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
