None of this helps with trying to remove defunct shadow files which
number in the 10s of millions.
Is there a quick way to see which shadow files are safe to delete
easily?
Remembering that there are MILLIONS of objects.
We have a 320TB cluster which is 272TB full. Of this, we should only
actually be seeing 190TB. There is 80TB of shadow files that should no
longer exist.
On 2015-03-18 02:00, Yehuda Sadeh-Weinraub wrote:
----- Original Message -----
From: "Ben" <b@benjackson.email>
To: "Craig Lewis" <clewis@xxxxxxxxxxxxxxxxxx>
Cc: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx>, "ceph-users"
<ceph-users@xxxxxxxx>
Sent: Monday, March 16, 2015 3:38:42 PM
Subject: Re: Shadow files
Thats the thing. The peaks and troughs are in USERS BUCKETS only.
The actual cluster usage does not go up and down, it just goes up up
up.
I would expect to see peaks and troughs much the same as the user
buckets peaks and troughs on the overall cluster disk usage.
But this is not the case.
We upgraded the cluster and radosgws to GIANT (0.87.1) yesterday, and
now we are seeing a large number of misplaced(??) objects being moved
around.
Does this mean it has found all the shadow files that shouldn't exist
anymore, and is deleting them? If so I would expect to start seeing
overall cluster usage drop, but this hasn't happened yet.
No, I don't think so. Sounds like your cluster is recovering, and it
happens in a completely different layer.
Any ideas?
try running:
$ radosgw-admin gc list --include-all
This should be showing all the shadow objects that are pending for
delete. Note that if you have a non-default radosgw configuration,
make sure you run radosgw-admin using the same user and config that
radosgw is running (e.g., add -n client.<user> appropriately),
otherwise it might not look at the correct zone data.
You could create an object, identify the shadow objects for that
object, remove it, check to see that the gc list command shows these
shadow objects. Then, wait the configured time (2 hours?), and see if
it was removed.
Yehuda
On 2015-03-17 06:12, Craig Lewis wrote:
> Out of curiousity, what's the frequency of the peaks and troughs?
>
> RadosGW has configs on how long it should wait after deleting before
> garbage collecting, how long between GC runs, and how many objects it
> can GC in per run.
>
> The defaults are 2 hours, 1 hour, and 32 respectively. Search
> http://docs.ceph.com/docs/master/radosgw/config-ref/ [2] for "rgw gc".
>
> If your peaks and troughs have a frequency less than 1 hour, then GC
> is going to delay and alias the disk usage w.r.t. the object count.
>
> If you have millions of objects, you probably need to tweak those
> values. If RGW is only GCing 32 objects an hour, it's never going to
> catch up.
>
> Now that I think about it, I bet I'm having issues here too. I delete
> more than (32*24) objects per day...
>
> On Sun, Mar 15, 2015 at 4:41 PM, Ben <b@benjackson.email> wrote:
>
>> It is either a problem with CEPH, Civetweb or something else in our
>> configuration.
>> But deletes in user buckets is still leaving a high number of old
>> shadow files. Since we have millions and millions of objects, it is
>> hard to reconcile what should and shouldnt exist.
>>
>> Looking at our cluster usage, there are no troughs, it is just a
>> rising peak.
>> But when looking at users data usage, we can see peaks and troughs
>> as you would expect as data is deleted and added.
>>
>> Our ceph version 0.80.9
>>
>> Please ideas?
>>
>> On 2015-03-13 02:25, Yehuda Sadeh-Weinraub wrote:
>>
>> ----- Original Message -----
>> From: "Ben" <b@benjackson.email>
>> To: ceph-users@xxxxxxxx
>> Sent: Wednesday, March 11, 2015 8:46:25 PM
>> Subject: Re: Shadow files
>>
>> Anyone got any info on this?
>>
>> Is it safe to delete shadow files?
>>
>> It depends. Shadow files are badly named objects that represent
>> part
>> of the objects data. They are only safe to remove if you know that
>> the
>> corresponding objects no longer exist.
>>
>> Yehuda
>>
>> On 2015-03-11 10:03, Ben wrote:
>>> We have a large number of shadow files in our cluster that aren't
>>> being deleted automatically as data is deleted.
>>>
>>> Is it safe to delete these files?
>>> Is there something we need to be aware of when deleting them?
>>> Is there a script that we can run that will delete these safely?
>>>
>>> Is there something wrong with our cluster that it isn't deleting
>> these
>>> files when it should be?
>>>
>>> We are using civetweb with radosgw, with tengine ssl proxy
>> infront of
>>> it
>>>
>>> Any advice please
>>> Thanks
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [1]
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [1]
>
>
>
> Links:
> ------
> [1] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> [2] http://docs.ceph.com/docs/master/radosgw/config-ref/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
