On Mon, Mar 16, 2015 at 12:12 PM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx> wrote: > Out of curiousity, what's the frequency of the peaks and troughs? > > RadosGW has configs on how long it should wait after deleting before garbage > collecting, how long between GC runs, and how many objects it can GC in per > run. > > The defaults are 2 hours, 1 hour, and 32 respectively. Search > http://docs.ceph.com/docs/master/radosgw/config-ref/ for "rgw gc". > > If your peaks and troughs have a frequency less than 1 hour, then GC is > going to delay and alias the disk usage w.r.t. the object count. > > If you have millions of objects, you probably need to tweak those values. > If RGW is only GCing 32 objects an hour, it's never going to catch up. > > > Now that I think about it, I bet I'm having issues here too. I delete more > than (32*24) objects per day... Uh, that's not quite what rgw_gc_max_objs mean. That param configures how the garbage control data objects and internal classes are sharded, and each grouping will only delete one object at a time. So it controls the parallelism, but not the total number of objects! Also, Yehuda says that changing this can be a bit dangerous because it currently needs to be consistent across any program doing or generating GC work. -Greg > > > > On Sun, Mar 15, 2015 at 4:41 PM, Ben <b@benjackson.email> wrote: >> >> It is either a problem with CEPH, Civetweb or something else in our >> configuration. >> But deletes in user buckets is still leaving a high number of old shadow >> files. Since we have millions and millions of objects, it is hard to >> reconcile what should and shouldnt exist. >> >> Looking at our cluster usage, there are no troughs, it is just a rising >> peak. >> But when looking at users data usage, we can see peaks and troughs as you >> would expect as data is deleted and added. >> >> Our ceph version 0.80.9 >> >> Please ideas? >> >> On 2015-03-13 02:25, Yehuda Sadeh-Weinraub wrote: >>> >>> ----- Original Message ----- >>>> >>>> From: "Ben" <b@benjackson.email> >>>> To: ceph-users@xxxxxxxx >>>> Sent: Wednesday, March 11, 2015 8:46:25 PM >>>> Subject: Re: Shadow files >>>> >>>> Anyone got any info on this? >>>> >>>> Is it safe to delete shadow files? >>> >>> >>> It depends. Shadow files are badly named objects that represent part >>> of the objects data. They are only safe to remove if you know that the >>> corresponding objects no longer exist. >>> >>> Yehuda >>> >>>> >>>> On 2015-03-11 10:03, Ben wrote: >>>> > We have a large number of shadow files in our cluster that aren't >>>> > being deleted automatically as data is deleted. >>>> > >>>> > Is it safe to delete these files? >>>> > Is there something we need to be aware of when deleting them? >>>> > Is there a script that we can run that will delete these safely? >>>> > >>>> > Is there something wrong with our cluster that it isn't deleting these >>>> > files when it should be? >>>> > >>>> > We are using civetweb with radosgw, with tengine ssl proxy infront of >>>> > it >>>> > >>>> > Any advice please >>>> > Thanks >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |