Created https://tracker.ceph.com/issues/40700 (sry forgot to mention).
On 11.07.19 16:41, Matt Benjamin wrote:
I don't think one has been created yet. Eric Ivancich and Mark Kogan
of my team are investigating this behavior.
Matt
On Thu, Jul 11, 2019 at 10:40 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
Is there already a tracker issue?
I'm seeing the same problem here. Started deletion of a bucket with a few hundred million objects a week ago or so and I've now noticed that it's also leaking memory and probably going to crash.
Going to investigate this further...
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Tue, Jul 9, 2019 at 1:26 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote:
Hi Harald,
Please file a tracker issue, yes. (Deletes do tend to be slower,
presumably due to rocksdb compaction.)
Matt
On Tue, Jul 9, 2019 at 7:12 AM Harald Staub <harald.staub@xxxxxxxxx> wrote:
Currently removing a bucket with a lot of objects:
radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects
This process was killed by the out-of-memory killer. Then looking at the
graphs, we see a continuous increase of memory usage for this process,
about +24 GB per day. Removal rate is about 3 M objects per day.
It is not the fastest hardware, and this index pool is still without
SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now
about 500 OSDs.
So with this bucket with 60 M objects, we would need about 480 GB of RAM
to come through. Or is there a workaround? Should I open a tracker issue?
The killed remove command can just be called again, but it will be
killed again before it finishes. Also, it has to run some time until it
continues to actually remove objects. This "wait time" is also
increasing. Last time, after about 16 M objects already removed, the
wait time was nearly 9 hours. Also during this time, there is a memory
ramp, but not so steep.
BTW it feels strange that the removal of objects is slower (about 3
times) than adding objects.
Harry
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
