Also, if that doesn't help, look at the following configurables:
config_opts.h:OPTION(rgw_gc_processor_max_time, OPT_INT, 3600) //
total run time for a single gc processor work
config_opts.h:OPTION(rgw_gc_processor_period, OPT_INT, 3600) // gc
processor cycle time
You may want to reduce the gc cycle time (and match the total run time also).
Yehuda
On Sat, Nov 15, 2014 at 3:23 AM, Daniel Hoffman <daniel@xxxxxxxxxx> wrote:
> Thanks.
>
> We have that set
>
> rgw gc max objs = 997
>
> The problem we have is we have Commvault connected to the cluster. What
> commvault does is every sunday does a full backup of an entire network
> (around 40TB) then ages out the old data, which is basically deleting all
> the old objects. Then all weel commvault makes incrimental backups and then
> at the end of the week does the same full backup and then purge of old data.
> Simple and standard backup.
>
> The problem is, that over the space of a week, 80TB gets put into the cloud
> and should also be purged deleted. We are not seeing the delete complete in
> an entire week, so we are basically slowly running out of space as the data
> is never cleaned up in time.
>
> Any thoughts anyone?
>
> I might try a larger prime number in like the 7919 range...
>
>
>
> On 2014-11-15 20:42, Jean-Charles LOPEZ wrote:
>>
>> Hi,
>>
>> this is an old thing I remember of and it may not be exactly related
>> but just in case, let’s try it out.
>>
>> Just verify the value of the following parameter rgw_gc_max_objs in
>> the RGW configuration via the admin socket (cep daemon {row.id} config
>> get rgw_gc_max_objs
>> .
>> If the value is 32, update the ceph.conf file for your RGW and set it
>> to 37 or any higher prime number such as 997 and restart your RGW.
>> Make sure you do the restart before your next big batch of object
>> removal. It will take some time to clear the overloaded buckets, may
>> be, but it should at least distribute the new ones evenly onto all gc
>> buckets after this and make the removal of the newly deleted objects
>> way quicker.
>>
>> Keep us posted to tell us if it has improved anything.
>>
>> JC
>>
>>
>>
>>
>>> On Nov 14, 2014, at 01:20, Daniel Hoffman <daniel@xxxxxxxxxx> wrote:
>>>
>>> Hi All.
>>>
>>> Running a Ceph Cluster (firefly) ceph version 0.80.5
>>>
>>> We use ceph mainly for backups via the radosGW at the moment.
>>>
>>> There had to be an account deleted/bucket removed which had a very large
>>> number of objects and was about 60TB in space.
>>>
>>> We have been monitoring it for days now, and the data is purging but very
>>> very slowly. We are actually putting new backups in much faster than the old
>>> data is being removed.
>>>
>>> As we are doing a lot of work with backups and aging out data we need to
>>> find a way to improve the cleanup process.
>>>
>>> Is there a way to improve the purge/clean performance?
>>> Is the clean/purge performance impacted by disk thread ioprio class
>>> setting?
>>>
>>> Any advice or tunables to improve the removal of data is appreciated.
>>>
>>> Thanks
>>>
>>> Daniel
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
