How long are you thinking here?
We added more storage to our cluster to overcome these issues, and we
can't keep throwing storage at it until the issues are fixed.
On 28/04/15 01:49, Yehuda Sadeh-Weinraub wrote:
It will get to the ceph mainline eventually. We're still reviewing and testing the fix, and there's more work to be done on the cleanup tool.
Yehuda
----- Original Message -----
From: "Ben" <b@benjackson.email>
To: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxx>
Sent: Sunday, April 26, 2015 11:02:23 PM
Subject: Re: Shadow Files
Are these fixes going to make it into the repository versions of ceph,
or will we be required to compile and install manually?
On 2015-04-26 02:29, Yehuda Sadeh-Weinraub wrote:
Yeah, that's definitely something that we'd address soon.
Yehuda
----- Original Message -----
From: "Ben" <b@benjackson.email>
To: "Ben Hines" <bhines@xxxxxxxxx>, "Yehuda Sadeh-Weinraub"
<yehuda@xxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxx>
Sent: Friday, April 24, 2015 5:14:11 PM
Subject: Re: Shadow Files
Definitely need something to help clear out these old shadow files.
I'm sure our cluster has around 100TB of these shadow files.
I've written a script to go through known objects to get prefixes of
objects
that should exist to compare to ones that shouldn't, but the time it
takes
to do this over millions and millions of objects is just too long.
On 25/04/15 09:53, Ben Hines wrote:
When these are fixed it would be great to get good steps for listing /
cleaning up any orphaned objects. I have suspicions this is affecting
us.
thanks-
-Ben
On Fri, Apr 24, 2015 at 3:10 PM, Yehuda Sadeh-Weinraub <
yehuda@xxxxxxxxxx >
wrote:
These ones:
http://tracker.ceph.com/issues/10295
http://tracker.ceph.com/issues/11447
----- Original Message -----
From: "Ben Jackson" <b@benjackson.email>
To: "Yehuda Sadeh-Weinraub" < yehuda@xxxxxxxxxx >
Cc: "ceph-users" < ceph-users@xxxxxxxx >
Sent: Friday, April 24, 2015 3:06:02 PM
Subject: Re: Shadow Files
We were firefly, then we upgraded to giant, now we are on hammer.
What issues?
On 25 Apr 2015 2:12 am, Yehuda Sadeh-Weinraub < yehuda@xxxxxxxxxx >
wrote:
What version are you running? There are two different issues that we
were
fixing this week, and we should have that upstream pretty soon.
Yehuda
----- Original Message -----
From: "Ben" <b@benjackson.email>
To: "ceph-users" < ceph-users@xxxxxxxx >
Cc: "Yehuda Sadeh-Weinraub" < yehuda@xxxxxxxxxx >
Sent: Thursday, April 23, 2015 7:42:06 PM
Subject: Shadow Files
We are still experiencing a problem with out gateway not properly
clearing out shadow files.
I have done numerous tests where I have:
-Uploaded a file of 1.5GB in size using s3browser application
-Done an object stat on the file to get its prefix
-Done rados ls -p .rgw.buckets | grep <prefix> to count the number
of
shadow files associated (in this case it is around 290 shadow files)
-Deleted said file with s3browser
-Performed a gc list, which shows the ~290 files listed
-Waited 24 hours to redo the rados ls -p .rgw.buckets | grep
<prefix>
to
recount the shadow files only to be left with 290 files still there
From log output /var/log/ceph/radosgw.log, I can see the following
when
clicking DELETE (this appears 290 times)
2015-04-24 10:43:29.996523 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996557 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996564 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=13107200 stripe_ofs=13107200 part_ofs=0
rule->part_size=0
2015-04-24 10:43:29.996570 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=17301504 stripe_ofs=17301504 part_ofs=0
rule->part_size=0
2015-04-24 10:43:29.996576 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=21495808 stripe_ofs=21495808 part_ofs=0
rule->part_size=0
2015-04-24 10:43:29.996581 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=25690112 stripe_ofs=25690112 part_ofs=0
rule->part_size=0
2015-04-24 10:43:29.996586 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=29884416 stripe_ofs=29884416 part_ofs=0
rule->part_size=0
2015-04-24 10:43:29.996592 7f0b0afb5700 0
RGWObjManifest::operator++():
result: ofs=34078720 stripe_ofs=34078720 part_ofs=0
rule->part_size=0
In this same log, I also see the gc process saying it is removing
said
file (these records appear 290 times too)
2015-04-23 14:16:27.926952 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.928572 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.929636 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.930448 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.931226 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.932103 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.933470 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
So even though it appears that the GC is processing its removal, the
shadow files remain!
Please help!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
