Hi Yuval, Thanks for the response. I did managed to disable the feature, however I hope you can understand my hesitancy to design our move away from pubsub onto a deprecated replacement (i.e. "notifications v1"). The difference between this and the other operations that require forwarding to the master site, for me, is that I don't rely on any of those operations for day-to-day function. For example, S3 user creation I do at deployment time and never again; so if I lose the ability to do that in the event of a site failure that's OK. Perhaps I just need to implement a process to ensure the remaining zone is quickly promoted on a site failure. A further concern of the new feature... My intention is to use the bucket notification feature to send events generated by the multisite sync to my application server. If the topic is now multisite, will all kicks be duplicated? For example consider: SiteA receives an S3 write for object SiteA writes to the event queue and asynchronously delivers the event SiteB syncs object from siteA SiteB writes to the event queue and asynchronously delivers the event Really I only want the second event in this example, which is what I get with the pubsub module, and I think is what I will get with "notifications_v1". Best Wishes, Alex ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx> Sent: Tuesday, September 3, 2024 12:15 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx> Cc: ceph-users <ceph-users@xxxxxxx> Subject: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy Hi Alex, It should be possible to disable the v2 feature through an admin command. radosgw-admin onegroup modify --disable-feature=notification_v2 Also note that when creating a new squid cluster, v2 is enabled by default. But, when upgrading an existing cluster, you need to call: radosgw-admin onegroup modify --enable-feature=notification_v2 to enable v2. However, we do encourage moving to v2, as there are many benefits to the new data model (besides site syncing). Regarding the problem you encountered when the "master" site is down, how is this different from other operations that are forwarded to "master"? Yuval On Tue, Sep 3, 2024 at 11:31 AM Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx<mailto:alexhus@xxxxxxxxxxxxx>> wrote: Hi Folks, I see in the pending release notes for Squid a description of the "notification_v2" feature. ceph/PendingReleaseNotes at main · ceph/ceph (github.com<http://github.com/>)<https://github.com/ceph/ceph/blob/main/PendingReleaseNotes#L43> I have some concerns about the multisite nature of this feature, I am using Ceph multisite to provide geographic redundancy for my application data. That seems to be a mainline use case. My application is designed to cope if an entire zone fails. My application currently relies on pubsub (and soon to be replaced by bucket notifications) to provide service. More specifically I rely on the ability to semi-regularly automatically reconfigure topics to adjust the HTTP endpoint. I believe with the "notification_v2" feature all adjustments of the topic are routed through the master for metadata site (much like creating an S3 user). Fundamentally I think this breaks the combination of Ceph multisite, notifications_v2 and reconfiguring a topic; you cannot reconfigure a topic unless both sites are up. I tested this, by setting up a deployment, shutting down a site and trying to reconfigure a topic. I get a 500 HTTP response to my createTopic, and can see this in the RGW logs: debug 2024-09-03T08:24:39.997+0000 7f53a3372640 20 ERROR: curl error: Couldn't connect to server req_data->error_buf=Failed to connect to 10.225.20.19 port 7480: No route to host debug 2024-09-03T08:24:39.997+0000 7f5329a7f640 4 req 2968544598970267460 0.686004043s sns:pubsub_topic_create CreateTopic forward_request_to_master returned ret = -2200 Have the implications of this been considered? For now I may be able to disable the notifications_v2 feature, however in the release note the predecessor is deprecated in Squid. Hopefully I'm missing something obvious. Kind regards, Alex _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |