Hi Folks, Have made a strange observation on one of our Storage Clusters. * Running Ceph 15.2.13. * Set up as a multisite pair of siteA and siteB. The two sites are geographically separated. * We are using S3 with a bucket in versioning suspended state (we previously had versioning on but decided it’s not required). * We’re using pubsub in conjunction with our S3 usage, don’t think this is relevant but figured I should mention just in case. We wrote 2413 small objects (no more than a few MB each) into the cluster via S3 on siteA. Then we deleted those objects via the S3 interface on siteA. Once the deleting was complete, we had 11 objects of the 2413 in a strange state on siteB but not siteA. On both sites the objects were set to zero size, I think this is expected. On siteA, where the deletes were sent, the objects were marked with “delete-marker”. On siteB, the objects were not marked with “delete-marker”. “DELETE_MARKER_CREATE” pubsub events on siteA were generated for these objects, but not on siteB (expecting the problem is not at the pubsub level). I followed a specific object through in logs and saw the following: * Object created: 00:11:16 * Object deleted: 01:04:02 * Pubsub on SiteB generated “OBJECT_CREATE” events at 00:11:31, 00:11:34, 01:04:18. My observations from this are: * There is plenty time between the create and the delete for this not to be some niche timing issue. * The final “OBJECT_CREATE” event is after the delete so I expect is a result of the multisite sync informing siteB of the change. * I expect this final event to be a “DELETE_MARKER_CREATE” event, not an “OBJECT_CREATE”. We can manually delete the objects from siteB to clean-up, but this is painful and makes us look a bit silly when we get support calls from customers for this sort of thing – so I’m keen find a better solution. I’ve failed to find a reason why this would occur due to us doing something wrong in our setup, it seems this is not the intended behaviour given that it’s only affecting a small number of the objects (most are marked as deleted on both sites as expected). * Has anyone else experienced this sort of thing? * I wonder if it’s related to our versioning suspended state. * How well tested is this scenario i.e., multisite + bucket versioning together? * Is there something we can do it mitigate it? As I understand, we can’t return to a versioning disabled state for this bucket. Thanks, and Season’s Greetings 😊 Alex Kershaw | alexhus@xxxxxxxxxxxxx<mailto:alexhus@xxxxxxxxxxxxx> Software Engineer | Azure for Operators _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |