Re: Shadow Files

[Ben <b@benjackson.email>]

Definitely need something to help clear out these old shadow files.

I'm sure our cluster has around 100TB of these shadow files.

I've written a script to go through known objects to get prefixes of objects that should exist to compare to ones that shouldn't, but the time it takes to do this over millions and millions of objects is just too long.

On 25/04/15 09:53, Ben Hines wrote:

When these are fixed it would be great to get good steps for listing / cleaning up any orphaned objects. I have suspicions this is affecting us.

thanks-

-Ben

On Fri, Apr 24, 2015 at 3:10 PM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:

These ones:

http://tracker.ceph.com/issues/10295
http://tracker.ceph.com/issues/11447

----- Original Message -----
> From: "Ben Jackson" <b@benjackson.email>
> To: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx>
> Cc: "ceph-users" <ceph-users@xxxxxxxx>
> Sent: Friday, April 24, 2015 3:06:02 PM
> Subject: Re: Shadow Files
>
> We were firefly, then we upgraded to giant, now we are on hammer.
>
> What issues?
>
> On 25 Apr 2015 2:12 am, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:
> >
> > What version are you running? There are two different issues that we were
> > fixing this week, and we should have that upstream pretty soon.
> >
> > Yehuda
> >
> > ----- Original Message -----
> > > From: "Ben" <b@benjackson.email>
> > > To: "ceph-users" <ceph-users@xxxxxxxx>
> > > Cc: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx>
> > > Sent: Thursday, April 23, 2015 7:42:06 PM
> > > Subject: Shadow Files
> > >
> > > We are still experiencing a problem with out gateway not properly
> > > clearing out shadow files.
> > >
> > > I have done numerous tests where I have:
> > > -Uploaded a file of 1.5GB in size using s3browser application
> > > -Done an object stat on the file to get its prefix
> > > -Done rados ls -p .rgw.buckets | grep <prefix> to count the number of
> > > shadow files associated (in this case it is around 290 shadow files)
> > > -Deleted said file with s3browser
> > > -Performed a gc list, which shows the ~290 files listed
> > > -Waited 24 hours to redo the rados ls -p .rgw.buckets | grep <prefix> to
> > > recount the shadow files only to be left with 290 files still there
> > >
> > >  From log output /var/log/ceph/radosgw.log, I can see the following when
> > > clicking DELETE (this appears 290 times)
> > > 2015-04-24 10:43:29.996523 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996557 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996564 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=13107200 stripe_ofs=13107200 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996570 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=17301504 stripe_ofs=17301504 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996576 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=21495808 stripe_ofs=21495808 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996581 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=25690112 stripe_ofs=25690112 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996586 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=29884416 stripe_ofs=29884416 part_ofs=0 rule->part_size=0
> > > 2015-04-24 10:43:29.996592 7f0b0afb5700  0 RGWObjManifest::operator++():
> > > result: ofs=34078720 stripe_ofs=34078720 part_ofs=0 rule->part_size=0
> > >
> > > In this same log, I also see the gc process saying it is removing said
> > > file (these records appear 290 times too)
> > > 2015-04-23 14:16:27.926952 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > > 2015-04-23 14:16:27.928572 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > > 2015-04-23 14:16:27.929636 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > > 2015-04-23 14:16:27.930448 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > > 2015-04-23 14:16:27.931226 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > > 2015-04-23 14:16:27.932103 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > > 2015-04-23 14:16:27.933470 7f15be0ee700  0 gc::process: removing
> > > .rgw.buckets:<objectname>
> > >
> > > So even though it appears that the GC is processing its removal, the
> > > shadow files remain!
> > >
> > > Please help!
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com