Re: memory usage of: radosgw-admin bucket rm

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I don't think one has been created yet.  Eric Ivancich and Mark Kogan
of my team are investigating this behavior.

Matt

On Thu, Jul 11, 2019 at 10:40 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
>
> Is there already a tracker issue?
>
> I'm seeing the same problem here. Started deletion of a bucket with a few hundred million objects a week ago or so and I've now noticed that it's also leaking memory and probably going to crash.
> Going to investigate this further...
>
> Paul
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
>
> On Tue, Jul 9, 2019 at 1:26 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote:
>>
>> Hi Harald,
>>
>> Please file a tracker issue, yes.  (Deletes do tend to be slower,
>> presumably due to rocksdb compaction.)
>>
>> Matt
>>
>> On Tue, Jul 9, 2019 at 7:12 AM Harald Staub <harald.staub@xxxxxxxxx> wrote:
>> >
>> > Currently removing a bucket with a lot of objects:
>> > radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects
>> >
>> > This process was killed by the out-of-memory killer. Then looking at the
>> > graphs, we see a continuous increase of memory usage for this process,
>> > about +24 GB per day. Removal rate is about 3 M objects per day.
>> >
>> > It is not the fastest hardware, and this index pool is still without
>> > SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now
>> > about 500 OSDs.
>> >
>> > So with this bucket with 60 M objects, we would need about 480 GB of RAM
>> > to come through. Or is there a workaround? Should I open a tracker issue?
>> >
>> > The killed remove command can just be called again, but it will be
>> > killed again before it finishes. Also, it has to run some time until it
>> > continues to actually remove objects. This "wait time" is also
>> > increasing. Last time, after about 16 M objects already removed, the
>> > wait time was nearly 9 hours. Also during this time, there is a memory
>> > ramp, but not so steep.
>> >
>> > BTW it feels strange that the removal of objects is slower (about 3
>> > times) than adding objects.
>> >
>> >   Harry
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> >
>>
>>
>> --
>>
>> Matt Benjamin
>> Red Hat, Inc.
>> 315 West Huron Street, Suite 140A
>> Ann Arbor, Michigan 48103
>>
>> http://www.redhat.com/en/technologies/storage
>>
>> tel.  734-821-5101
>> fax.  734-769-8938
>> cel.  734-216-5309
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux