Yehuda Sadeh-Weinraub writes: > On Tue, Apr 19, 2016 at 9:10 AM, Abhishek Lekshmanan <abhishek@xxxxxxxx> wrote: >> Trying deleting objects & buckets from a secondary zone in a RGW >> multisite configuration leads to some wierdness: >> >> 1. On deleting an object and the bucket immediately will mostly lead to >> object and bucket getting deleted in the secondary zone, but since we >> forward the bucket deletion to master only after we delete in secondary >> it will fail with 409 (BucketNotEmpty) and gets reraised as a 500 to the >> client. This _seems_ simple enough to fix if we forward the bucket >> deletion request to master zone before attempting deletion locally, >> (issue: http://tracker.ceph.com/issues/15540, possible fix: https://github.com/ceph/ceph/pull/8655) >> > > Yeah, this looks good. We'll get it through testing. > >> 2. Deletion of objects themselves: deletion of objects themselves seems >> to be a bit racy, deleting an object on a secondary zone succeeds, >> listing the bucket seems to show an empty list, but gets populated with >> the object again sometimes (this time with a newer timestamp), this is >> not always guaranteed to be reproduce, but I've seen this often with >> multipart uploads, as an eg: >> >> $ s3 -u list test-mp >> Key Last Modified Size >> -------------------------------------------------- -------------------- ----- >> test.img 2016-04-19T13:00:17Z 40M >> $ s3 -u delete test-mp/test.img >> $ s3 -u list test-mp >> Key Last Modified Size >> -------------------------------------------------- -------------------- ----- >> test.img 2016-04-19T13:00:45Z 40M >> $ s3 -u delete test-mp/test.img # wait for a min >> $ s3 -us list test-mp >> -------------------------------------------------- -------------------- ----- >> test.img 2016-04-19T13:01:52Z 40M >> >> >> Mostly seeing log entries of this form in both the cases ie. where >> delete object seems to be successfully delete in both master and >> secondary zone and the case where it succeeds in master and fails in >> secondary : >> >> 20 parsed entry: id=00000000027.27.2 iter->object=foo iter->instance= name=foo instance= ns= >> 20 [inc sync] skipping object: dkr:d8e0ec3d-b3da-43f8-a99b-38a5b4941b6f.14113.2:-1/foo: non-complete operation >> 20 parsed entry: id=00000000028.28.2 iter->object=foo iter->instance= name=foo instance= ns= >> 20 [inc sync] skipping object: dkr:d8e0ec3d-b3da-43f8-a99b-38a5b4941b6f.14113.2:-1/foo: canceled operation >> >> Any ideas on this? >> > > Do you have more than 2 zones syncing? Is it an object delete that > came right after the object creation? Only 2 zones ie. one master and one secondary, req, on secondary. The delete came right after the create though > > Yehuda -- Abhishek -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
