I don't think one has been created yet. Eric Ivancich and Mark Kogan of my team are investigating this behavior. Matt On Thu, Jul 11, 2019 at 10:40 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote: > > Is there already a tracker issue? > > I'm seeing the same problem here. Started deletion of a bucket with a few hundred million objects a week ago or so and I've now noticed that it's also leaking memory and probably going to crash. > Going to investigate this further... > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > > On Tue, Jul 9, 2019 at 1:26 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: >> >> Hi Harald, >> >> Please file a tracker issue, yes. (Deletes do tend to be slower, >> presumably due to rocksdb compaction.) >> >> Matt >> >> On Tue, Jul 9, 2019 at 7:12 AM Harald Staub <harald.staub@xxxxxxxxx> wrote: >> > >> > Currently removing a bucket with a lot of objects: >> > radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects >> > >> > This process was killed by the out-of-memory killer. Then looking at the >> > graphs, we see a continuous increase of memory usage for this process, >> > about +24 GB per day. Removal rate is about 3 M objects per day. >> > >> > It is not the fastest hardware, and this index pool is still without >> > SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now >> > about 500 OSDs. >> > >> > So with this bucket with 60 M objects, we would need about 480 GB of RAM >> > to come through. Or is there a workaround? Should I open a tracker issue? >> > >> > The killed remove command can just be called again, but it will be >> > killed again before it finishes. Also, it has to run some time until it >> > continues to actually remove objects. This "wait time" is also >> > increasing. Last time, after about 16 M objects already removed, the >> > wait time was nearly 9 hours. Also during this time, there is a memory >> > ramp, but not so steep. >> > >> > BTW it feels strange that the removal of objects is slower (about 3 >> > times) than adding objects. >> > >> > Harry >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> > >> >> >> -- >> >> Matt Benjamin >> Red Hat, Inc. >> 315 West Huron Street, Suite 140A >> Ann Arbor, Michigan 48103 >> >> http://www.redhat.com/en/technologies/storage >> >> tel. 734-821-5101 >> fax. 734-769-8938 >> cel. 734-216-5309 >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |