We are still experiencing a problem with out gateway not properly
clearing out shadow files.
I have done numerous tests where I have:
-Uploaded a file of 1.5GB in size using s3browser application
-Done an object stat on the file to get its prefix
-Done rados ls -p .rgw.buckets | grep <prefix> to count the number of
shadow files associated (in this case it is around 290 shadow files)
-Deleted said file with s3browser
-Performed a gc list, which shows the ~290 files listed
-Waited 24 hours to redo the rados ls -p .rgw.buckets | grep <prefix> to
recount the shadow files only to be left with 290 files still there
From log output /var/log/ceph/radosgw.log, I can see the following when
clicking DELETE (this appears 290 times)
2015-04-24 10:43:29.996523 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996557 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996564 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=13107200 stripe_ofs=13107200 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996570 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=17301504 stripe_ofs=17301504 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996576 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=21495808 stripe_ofs=21495808 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996581 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=25690112 stripe_ofs=25690112 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996586 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=29884416 stripe_ofs=29884416 part_ofs=0 rule->part_size=0
2015-04-24 10:43:29.996592 7f0b0afb5700 0 RGWObjManifest::operator++():
result: ofs=34078720 stripe_ofs=34078720 part_ofs=0 rule->part_size=0
In this same log, I also see the gc process saying it is removing said
file (these records appear 290 times too)
2015-04-23 14:16:27.926952 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.928572 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.929636 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.930448 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.931226 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.932103 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
2015-04-23 14:16:27.933470 7f15be0ee700 0 gc::process: removing
.rgw.buckets:<objectname>
So even though it appears that the GC is processing its removal, the
shadow files remain!
Please help!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
