It will get to the ceph mainline eventually. We're still reviewing and testing the fix, and there's more work to be done on the cleanup tool. Yehuda ----- Original Message ----- > From: "Ben" <b@benjackson.email> > To: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx> > Cc: "ceph-users" <ceph-users@xxxxxxxx> > Sent: Sunday, April 26, 2015 11:02:23 PM > Subject: Re: Shadow Files > > Are these fixes going to make it into the repository versions of ceph, > or will we be required to compile and install manually? > > On 2015-04-26 02:29, Yehuda Sadeh-Weinraub wrote: > > Yeah, that's definitely something that we'd address soon. > > > > Yehuda > > > > ----- Original Message ----- > >> From: "Ben" <b@benjackson.email> > >> To: "Ben Hines" <bhines@xxxxxxxxx>, "Yehuda Sadeh-Weinraub" > >> <yehuda@xxxxxxxxxx> > >> Cc: "ceph-users" <ceph-users@xxxxxxxx> > >> Sent: Friday, April 24, 2015 5:14:11 PM > >> Subject: Re: Shadow Files > >> > >> Definitely need something to help clear out these old shadow files. > >> > >> I'm sure our cluster has around 100TB of these shadow files. > >> > >> I've written a script to go through known objects to get prefixes of > >> objects > >> that should exist to compare to ones that shouldn't, but the time it > >> takes > >> to do this over millions and millions of objects is just too long. > >> > >> On 25/04/15 09:53, Ben Hines wrote: > >> > >> > >> > >> When these are fixed it would be great to get good steps for listing / > >> cleaning up any orphaned objects. I have suspicions this is affecting > >> us. > >> > >> thanks- > >> > >> -Ben > >> > >> On Fri, Apr 24, 2015 at 3:10 PM, Yehuda Sadeh-Weinraub < > >> yehuda@xxxxxxxxxx > > >> wrote: > >> > >> > >> These ones: > >> > >> http://tracker.ceph.com/issues/10295 > >> http://tracker.ceph.com/issues/11447 > >> > >> ----- Original Message ----- > >> > From: "Ben Jackson" <b@benjackson.email> > >> > To: "Yehuda Sadeh-Weinraub" < yehuda@xxxxxxxxxx > > >> > Cc: "ceph-users" < ceph-users@xxxxxxxx > > >> > Sent: Friday, April 24, 2015 3:06:02 PM > >> > Subject: Re: Shadow Files > >> > > >> > We were firefly, then we upgraded to giant, now we are on hammer. > >> > > >> > What issues? > >> > > >> > On 25 Apr 2015 2:12 am, Yehuda Sadeh-Weinraub < yehuda@xxxxxxxxxx > > >> > wrote: > >> > > > >> > > What version are you running? There are two different issues that we > >> > > were > >> > > fixing this week, and we should have that upstream pretty soon. > >> > > > >> > > Yehuda > >> > > > >> > > ----- Original Message ----- > >> > > > From: "Ben" <b@benjackson.email> > >> > > > To: "ceph-users" < ceph-users@xxxxxxxx > > >> > > > Cc: "Yehuda Sadeh-Weinraub" < yehuda@xxxxxxxxxx > > >> > > > Sent: Thursday, April 23, 2015 7:42:06 PM > >> > > > Subject: Shadow Files > >> > > > > >> > > > We are still experiencing a problem with out gateway not properly > >> > > > clearing out shadow files. > >> > > > > >> > > > I have done numerous tests where I have: > >> > > > -Uploaded a file of 1.5GB in size using s3browser application > >> > > > -Done an object stat on the file to get its prefix > >> > > > -Done rados ls -p .rgw.buckets | grep <prefix> to count the number > >> > > > of > >> > > > shadow files associated (in this case it is around 290 shadow files) > >> > > > -Deleted said file with s3browser > >> > > > -Performed a gc list, which shows the ~290 files listed > >> > > > -Waited 24 hours to redo the rados ls -p .rgw.buckets | grep > >> > > > <prefix> > >> > > > to > >> > > > recount the shadow files only to be left with 290 files still there > >> > > > > >> > > > From log output /var/log/ceph/radosgw.log, I can see the following > >> > > > when > >> > > > clicking DELETE (this appears 290 times) > >> > > > 2015-04-24 10:43:29.996523 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996557 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996564 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=13107200 stripe_ofs=13107200 part_ofs=0 > >> > > > rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996570 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=17301504 stripe_ofs=17301504 part_ofs=0 > >> > > > rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996576 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=21495808 stripe_ofs=21495808 part_ofs=0 > >> > > > rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996581 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=25690112 stripe_ofs=25690112 part_ofs=0 > >> > > > rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996586 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=29884416 stripe_ofs=29884416 part_ofs=0 > >> > > > rule->part_size=0 > >> > > > 2015-04-24 10:43:29.996592 7f0b0afb5700 0 > >> > > > RGWObjManifest::operator++(): > >> > > > result: ofs=34078720 stripe_ofs=34078720 part_ofs=0 > >> > > > rule->part_size=0 > >> > > > > >> > > > In this same log, I also see the gc process saying it is removing > >> > > > said > >> > > > file (these records appear 290 times too) > >> > > > 2015-04-23 14:16:27.926952 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > 2015-04-23 14:16:27.928572 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > 2015-04-23 14:16:27.929636 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > 2015-04-23 14:16:27.930448 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > 2015-04-23 14:16:27.931226 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > 2015-04-23 14:16:27.932103 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > 2015-04-23 14:16:27.933470 7f15be0ee700 0 gc::process: removing > >> > > > .rgw.buckets:<objectname> > >> > > > > >> > > > So even though it appears that the GC is processing its removal, the > >> > > > shadow files remain! > >> > > > > >> > > > Please help! > >> > > > _______________________________________________ > >> > > > ceph-users mailing list > >> > > > ceph-users@xxxxxxxxxxxxxx > >> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > >> > > > >> > _______________________________________________ > >> > ceph-users mailing list > >> > ceph-users@xxxxxxxxxxxxxx > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |