I rebuilt the setup from scratch and left off --master from the zones in the other zonegroups and it had no effect on the outcome. Ulrich.Klein@xxxxxxxxxxxxxxx has tried this as well and sees the same failure. I hope there is someone who can see what we are doing wrong. This topology is shown here https://events.static.linuxfound.org/sites/events/files/slides/Linuxcon_multisitev2.pdf which is an official redhat doc. -- Mark Selby Sr Linux Administrator, The Voleon Group mselby@xxxxxxxxxx This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. On 4/19/22, 1:59 AM, "Eugen Block" <eblock@xxxxxx> wrote: Hi, unless there are copy/paste mistakes involved I believe you shouldn't specify '--master' for the secondary zone because you did that already for the first zone which is supposed to be the master zone. You specified '--rgw-zone=us-west-1' as the master zone within your realm, but then you run this command on the second cluster: radosgw-admin zone create --rgw-zone=eu-west-1 \ --rgw-zonegroup=eu \ --default \ --master \ --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph2dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AOAAqwVsdgCZZT3uRiLcIgpR5swszqo3JrCbT4dAowA%3D&reserved=0 That's the reason for this error when trying to commit on the second cluster: 2022-04-16T09:16:20.345-0700 7faf98ab6380 1 Cannot find zone id=9f8a06eb-5a1c-4052-b04d-359f21c95371 (name=eu-west-1), switching to local zonegroup configuration Because your master zone is this one: "master_zone": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd" I'd recommend to purge the secondary zone and start over. If this is not the root cause, please update your post. Regards, Eugen Zitat von Mark Selby <mselby@xxxxxxxxxx>: > I have been trying to build a multisite ceph rgw installation with a > single realm, multiple zonegroups, and a single zone per zonegroup. This > is a model around 3 locations spread over long distances. I have > sucessfully create an installation with a single realm, a single > zonegroup and multiple zones on that one zonegroup. > > I have had no luck getting my multiple zonegroup installation even > setup. I have read the docs over and over but I still think that I am > doing something incorrect (or possibly a bug?) > > I am running Pacific 16.2.7 in a containerized environment > > I have created a github gist of all of the commands and output show > below as that may be easier to read for some. > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Ftokenrain%2F4edf85b0060ce5004f2003aa8a66e67d&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wt%2BJsyiA0GjlwHgmqHaM7eV1Y4D9Y6Ci8uJm%2BxBb6nI%3D&reserved=0 > > Cluster 1 and Cluster 2 are separate ceph clusters. Cluster 1 commands > were run on a node in cluster1 and Cluster 2 commands were run on a node > in cluster2 > > All and any help is greatly appreciated. > > ############ > # TOPOLOGY # > ############ > > realm = acme.com > zonegroup = us > zone = us-west-1 > zonegroup = eu > zone = eu-west-1 > zonegroup = as > zone = as-west-1 > > ###################### > # CLUSTER 1 COMMANDS # > ###################### > > radosgw-admin realm create --rgw-realm=acme --default > > radosgw-admin zonegroup create --rgw-zonegroup=us --rgw-realm=acme > --master --default --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0 > > radosgw-admin zone create --rgw-zonegroup=us \ > --rgw-zone=us-west-1 \ > --master \ > --default \ > --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0 > > radosgw-admin user create --uid="sync-user" > --display-name="Synchronization User" --system > > radosgw-admin zone modify --rgw-zonegroup=us --rgw-zone=us-west-1 > --access-key=<ACCESS_KEY> --secret=<SECRET> > > radosgw-admin period update --commit > > ceph orch apply -i rgw-us-west-1.yml > > ######################### > # Cluster 1 -- RGW Spec # > ######################### > --- > service_type: rgw > service_id: us-west-1 > placement: > hosts: > - ceph1dev01.acme.com > - ceph1dev02.acme.com > - ceph1dev03.acme.com > spec: > ssl: true > rgw_realm: acme > rgw_zone: us-west-1 > rgw_frontend_port: 443 > rgw_frontend_type: beast > rgw_frontend_ssl_certificate: | > > ###################### > # CLUSTER 2 COMMANDS # > ###################### > > radosgw-admin realm pull --rgw-realm=acme > --url=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0 --access-key=<ACCESS_KEY> > --secret=<SECRET> --default > > radosgw-admin zonegroup create --rgw-realm=acme --rgw-zonegroup=eu > --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph2dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AOAAqwVsdgCZZT3uRiLcIgpR5swszqo3JrCbT4dAowA%3D&reserved=0 > > radosgw-admin zone create --rgw-zone=eu-west-1 \ > --rgw-zonegroup=eu \ > --default \ > --master \ > --endpoints=https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph2dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AOAAqwVsdgCZZT3uRiLcIgpR5swszqo3JrCbT4dAowA%3D&reserved=0 > > radosgw-admin zone modify --rgw-zone=eu-west-1 --rgw-zonegroup=eu > --access-key=<ACCESS_KEY> --secret=<SECRET > > radosgw-admin period update > > radosgw-admin period update --commit > > ceph orch apply -i rgw-eu-west1-2.yml > > ############################################## > # CLUSTER 1 OUTPUT OF period update --commit # > ############################################## > > { > "id": "b153187a-1d91-4bbf-a674-d3cad9fd23da", > "epoch": 1, > "predecessor_uuid": "740d6999-ce83-47ff-81f5-615a3a441a96", > "sync_status": [], > "period_map": { > "id": "b153187a-1d91-4bbf-a674-d3cad9fd23da", > "zonegroups": [ > { > "id": "e39e0b42-43a8-47eb-b6cd-6c2524ff51d2", > "name": "us", > "api_name": "us", > "is_master": "true", > "endpoints": [ > "https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0"; > ], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "zones": [ > { > "id": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "name": "us-west-1", > "endpoints": [ > "https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0"; > ], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 11, > "read_only": "false", > "tier_type": "", > "sync_from_all": "true", > "sync_from": [], > "redirect_zone": "" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [], > "storage_classes": [ > "STANDARD" > ] > } > ], > "default_placement": "default-placement", > "realm_id": "1504a901-d969-463e-b43c-ce362782c9e2", > "sync_policy": { > "groups": [] > } > } > ], > "short_zone_ids": [ > { > "key": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "val": 1041973059 > } > ] > }, > "master_zonegroup": "e39e0b42-43a8-47eb-b6cd-6c2524ff51d2", > "master_zone": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "period_config": { > "bucket_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "user_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > } > }, > "realm_id": "1504a901-d969-463e-b43c-ce362782c9e2", > "realm_name": "acme", > "realm_epoch": 2 > } > > > ##################################### > # CLUSTER 2 OUTPUT OF period update # > ##################################### > > { > "id": "1504a901-d969-463e-b43c-ce362782c9e2:staging", > "epoch": 1, > "predecessor_uuid": "b153187a-1d91-4bbf-a674-d3cad9fd23da", > "sync_status": [], > "period_map": { > "id": "b153187a-1d91-4bbf-a674-d3cad9fd23da", > "zonegroups": [ > { > "id": "7e0bc1f3-a15f-449a-aeb5-ef0cb0fd6d8d", > "name": "eu", > "api_name": "eu", > "is_master": "false", > "endpoints": [ > "https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph2dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AOAAqwVsdgCZZT3uRiLcIgpR5swszqo3JrCbT4dAowA%3D&reserved=0"; > ], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "9f8a06eb-5a1c-4052-b04d-359f21c95371", > "zones": [ > { > "id": "9f8a06eb-5a1c-4052-b04d-359f21c95371", > "name": "eu-west-1", > "endpoints": [ > "https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph2dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AOAAqwVsdgCZZT3uRiLcIgpR5swszqo3JrCbT4dAowA%3D&reserved=0"; > ], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 11, > "read_only": "false", > "tier_type": "", > "sync_from_all": "true", > "sync_from": [], > "redirect_zone": "" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [], > "storage_classes": [ > "STANDARD" > ] > } > ], > "default_placement": "default-placement", > "realm_id": "1504a901-d969-463e-b43c-ce362782c9e2", > "sync_policy": { > "groups": [] > } > }, > { > "id": "e39e0b42-43a8-47eb-b6cd-6c2524ff51d2", > "name": "us", > "api_name": "us", > "is_master": "true", > "endpoints": [ > "https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0"; > ], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "zones": [ > { > "id": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "name": "us-west-1", > "endpoints": [ > "https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fceph1dev01.acme.com%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ojLgv0XS7mQ49UVuZtYZ8SZMSBjbi75xCkeR1i3m3EE%3D&reserved=0"; > ], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 11, > "read_only": "false", > "tier_type": "", > "sync_from_all": "true", > "sync_from": [], > "redirect_zone": "" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [], > "storage_classes": [ > "STANDARD" > ] > } > ], > "default_placement": "default-placement", > "realm_id": "1504a901-d969-463e-b43c-ce362782c9e2", > "sync_policy": { > "groups": [] > } > } > ], > "short_zone_ids": [ > { > "key": "9f8a06eb-5a1c-4052-b04d-359f21c95371", > "val": 2999707275 > }, > { > "key": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "val": 1041973059 > } > ] > }, > "master_zonegroup": "e39e0b42-43a8-47eb-b6cd-6c2524ff51d2", > "master_zone": "d7ceaa4f-06c0-4c21-bcec-efe90f55ecfd", > "period_config": { > "bucket_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "user_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > } > }, > "realm_id": "1504a901-d969-463e-b43c-ce362782c9e2", > "realm_name": "acme", > "realm_epoch": 3 > } > > ############################################## > # CLUSTER 2 OUTPUT OF period update --commit # > ############################################## > > 2022-04-16T09:16:20.345-0700 7faf98ab6380 1 Cannot find zone > id=9f8a06eb-5a1c-4052-b04d-359f21c95371 (name=eu-west-1), switching > to local zonegroup configuration > 2022-04-16T09:16:20.345-0700 7faf98ab6380 -1 Cannot find zone > id=9f8a06eb-5a1c-4052-b04d-359f21c95371 (name=eu-west-1) > 2022-04-16T09:16:20.345-0700 7faf98ab6380 0 ERROR: failed to start > notify service ((22) Invalid argument > 2022-04-16T09:16:20.345-0700 7faf98ab6380 0 ERROR: failed to init > services (ret=(22) Invalid argument) > couldn't init storage provider > > -- > > > Mark Selby > Sr Linux Administrator, The Voleon Group > mselby@xxxxxxxxxx > > This email is subject to important conditions and disclosures that > are listed on this web page: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvoleon.com%2Fdisclaimer%2F&data=04%7C01%7Cmselby%40voleon.com%7C5a448eef012a44477c0908da21e28ac3%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637859555823860934%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qL%2FmgwTHQZftRqS6639mnFzoPji39VmKazxNa4tSn%2Bg%3D&reserved=0. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
