Hi Huy, The sync result you posted earlier appears to be from master zone. Have you checked the secondary zone with 'radosgw-admin sync status --rgw-zone=hn2'? Can you check that: - sync user exists in the realm with 'radosgw-admin user list --rgw-realm=multi-region' - sync user's access_key and secret_key are set on both zones with 'radosgw-admin zone get --rgw-zone=hn1' and 'radosgw-admin zone get --rgw-zone=hn2' - you pulled the realm on both RGWs after deployment - you restarted both gateways. Thing is you need to respect a certain order to set up the synchronization ; the sync user has to be part of the realm and its credentials need to be set on both zones, but you'll only be able to create this sync user after the realm is created which implies creating the master and default zonegroup and creating (or setting) a master and defaut zone to this zonegroup. If the master and default zone already exists with buckets and data in it, you could try these commands below: - Set vars: realm='multi-region' zonegroup='hn' master_zone='hn1' secondary_zone='hn2' syncuser='synchronization-user' master_endpoint='http://10.3.55.66:8080' secondary_endpoint='http://10.3.53.107:8080' - create realm: radosgw-admin realm create --rgw-realm=${realm} --default - create master and default zonegroup: radosgw-admin zonegroup create --rgw-zonegroup=${zonegroup} --endpoints=${master_endpoint} --rgw-realm=${realm} --master --default - create (or modify if exist) master and default zone **without** access and secret keys (since the sync user does not exist at this time): radosgw-admin zone [create|modify] --rgw-zonegroup=${zonegroup} --rgw-zone=${master_zone} --endpoints=${master_endpoint} --master --default - commit changes: radosgw-admin period update --commit - restart master zone's RGW: ceph orch daemon restart $(ceph orch ps | grep rgw.${realm}-${zonegroup}-${master_zone} | awk '{print $1}') - create sync user in the realm radosgw-admin user create --uid="${syncuser}" --display-name="Synchronization User" --system - set access_key and secret_key (needs jq) access_key=$(radosgw-admin user info --uid=${syncuser} --format json | jq -r '.keys[].access_key') secret_key=$(radosgw-admin user info --uid=${syncuser} --format json | jq -r '.keys[].secret_key') - create secondary zone **with** access_key and secret_key (now that the sync user exists) radosgw-admin zone create --rgw-zonegroup=${zonegroup} --rgw-zone=${secondary_zone} --endpoints=${secondary_endpoint} --access-key=${access_key} --secret-key=${secret_key} - modify master zone to add access_key and secret_key radosgw-admin zone modify --rgw-zonegroup=${zonegroup} --rgw-zone=${master_zone} --endpoints=${master_endpoint} --access-key=${access_key} --secret-key=${secret_key} --master --default - commit changes: radosgw-admin period update --commit - pull the realm on both gateways: on master zone's RGW: radosgw-admin realm pull --url ${secondary_endpoint} --access-key ${access_key} --secret-key ${secret_key} --rgw-zone=${master_zone} on secondary zone's RGW: radosgw-admin realm pull --url ${master_endpoint} --access-key ${access_key} --secret-key ${secret_key} --rgw-zone=${secondary_zone} Both commands should output the same result. - restart both gateways: ceph orch daemon restart $(ceph orch ps | grep rgw.${realm}-${zonegroup}-${master_zone} | awk '{print $1}') ceph orch daemon restart $(ceph orch ps | grep rgw.${realm}-${zonegroup}-${secondary_zone} | awk '{print $1}') - watch for sync status: watch radosgw-admin sync status --rgw-zone=${master_zone} watch radosgw-admin sync status --rgw-zone=${secondary_zone} - watch for RGW logs (journalctl or /var/log/ceph/$(ceph fsid)/ceph-client.rgw.*) - waych for any sync errors: radosgw-admin sync error list Uploading new data should not be necessary to trigger the synchronization between the 2 zones. Synchronization should happen right away. Also, you shouldn't have to create any rados pools. RGWs take care of each pool and hn[1-2].rgw.bucket.data pools should get created as soon as you upload new data or data synchronization starts. Been playing a bit on 19.1.0 RC lately and had success with these commands. I hope there's no typo. Otherwise, let us know. Regards, Frédéric. ----- Le 24 Juil 24, à 10:00, Huy Nguyen viplanghe6@xxxxxxxxx a écrit : > Hi, > Thanks for you reply. I'm not sure what do you means by full sync. I tried > data/metadata sync init/run but nothing happened. I tried with a fresh cluster > but still can't make it work. Here is the sync status: > > realm 26c6fd00-56f3-473f-8d9b-e583efb58a2b (multi-region) > zonegroup bc664368-6a1c-43fb-9e86-99b62e579236 (hn) > zone 92b1769f-8266-495d-a6df-3b89797f21d5 (hn1) > current time 2024-07-24T07:56:34Z > zonegroup features enabled: resharding > disabled: compress-encrypted > metadata sync no sync (zone is master) > data sync source: 4fe607da-8eb7-44ae-a34d-e6b5e47b6065 (hn2) > syncing > full sync: 0/128 shards > incremental sync: 128/128 shards > data is caught up with source > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |