Is there already a tracker issue?
I'm seeing the same problem here. Started deletion of a bucket with a few hundred million objects a week ago or so and I've now noticed that it's also leaking memory and probably going to crash.
Going to investigate this further...
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Tue, Jul 9, 2019 at 1:26 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote:
Hi Harald,
Please file a tracker issue, yes. (Deletes do tend to be slower,
presumably due to rocksdb compaction.)
Matt
On Tue, Jul 9, 2019 at 7:12 AM Harald Staub <harald.staub@xxxxxxxxx> wrote:
>
> Currently removing a bucket with a lot of objects:
> radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects
>
> This process was killed by the out-of-memory killer. Then looking at the
> graphs, we see a continuous increase of memory usage for this process,
> about +24 GB per day. Removal rate is about 3 M objects per day.
>
> It is not the fastest hardware, and this index pool is still without
> SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now
> about 500 OSDs.
>
> So with this bucket with 60 M objects, we would need about 480 GB of RAM
> to come through. Or is there a workaround? Should I open a tracker issue?
>
> The killed remove command can just be called again, but it will be
> killed again before it finishes. Also, it has to run some time until it
> continues to actually remove objects. This "wait time" is also
> increasing. Last time, after about 16 M objects already removed, the
> wait time was nearly 9 hours. Also during this time, there is a memory
> ramp, but not so steep.
>
> BTW it feels strange that the removal of objects is slower (about 3
> times) than adding objects.
>
> Harry
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
--
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101
fax. 734-769-8938
cel. 734-216-5309
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
