Hi Matthew, That's interesting to hear - especially that you are not using bucket versioning and are seeing the same issue. I was hoping this might go away if I turned off versioning, but if that's not the case this gets a bit more worrying for us! Thanks, Alex -----Original Message----- From: Matthew Darwin <bugs@xxxxxxxxxx> Sent: Friday, December 23, 2022 3:13 PM To: ceph-users@xxxxxxx Subject: [EXTERNAL] Re: S3 Deletes in Multisite Sometimes Not Syncing Hi Alex, We also have a multi-site setup (17.2.5). I just deleted a bunch of files from one side and some files got deleted on the other side but not others. I waited 10 hours to see if the files would delete. I didn't do an exhaustive test like yours, but seems similar issues. In our case, like yours, the two ceph sites are geographically separated. We don't have versioning enabled. I would love to hear from anyone who has replication working perfectly. On 2022-12-22 07:17, Alex Hussein-Kershaw (HE/HIM) wrote: > Hi Folks, > > Have made a strange observation on one of our Storage Clusters. > > * Running Ceph 15.2.13. > * Set up as a multisite pair of siteA and siteB. The two sites are geographically separated. > * We are using S3 with a bucket in versioning suspended state (we previously had versioning on but decided it’s not required). > * We’re using pubsub in conjunction with our S3 usage, don’t think this is relevant but figured I should mention just in case. > > We wrote 2413 small objects (no more than a few MB each) into the cluster via S3 on siteA. Then we deleted those objects via the S3 interface on siteA. Once the deleting was complete, we had 11 objects of the 2413 in a strange state on siteB but not siteA. > > On both sites the objects were set to zero size, I think this is expected. On siteA, where the deletes were sent, the objects were marked with “delete-marker”. On siteB, the objects were not marked with “delete-marker”. “DELETE_MARKER_CREATE” pubsub events on siteA were generated for these objects, but not on siteB (expecting the problem is not at the pubsub level). > > I followed a specific object through in logs and saw the following: > > * Object created: 00:11:16 > * Object deleted: 01:04:02 > * Pubsub on SiteB generated “OBJECT_CREATE” events at 00:11:31, 00:11:34, 01:04:18. > > > My observations from this are: > > * There is plenty time between the create and the delete for this not to be some niche timing issue. > * The final “OBJECT_CREATE” event is after the delete so I expect is a result of the multisite sync informing siteB of the change. > * I expect this final event to be a “DELETE_MARKER_CREATE” event, not an “OBJECT_CREATE”. > > We can manually delete the objects from siteB to clean-up, but this is painful and makes us look a bit silly when we get support calls from customers for this sort of thing – so I’m keen find a better solution. > > I’ve failed to find a reason why this would occur due to us doing something wrong in our setup, it seems this is not the intended behaviour given that it’s only affecting a small number of the objects (most are marked as deleted on both sites as expected). > > * Has anyone else experienced this sort of thing? > * I wonder if it’s related to our versioning suspended state. > * How well tested is this scenario i.e., multisite + bucket versioning together? > * Is there something we can do it mitigate it? As I understand, we can’t return to a versioning disabled state for this bucket. > > Thanks, and Season’s Greetings 😊 > > Alex Kershaw |alexhus@xxxxxxxxxxxxx<mailto:alexhus@xxxxxxxxxxxxx> > Software Engineer | Azure for Operators > > _______________________________________________ > ceph-users mailing list --ceph-users@xxxxxxx To unsubscribe send an > email toceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |