Nevermind. Found the issue was that zone still existed in my period from some old testing.
Running from the original cluster 'radosgw-admin zonegroup remove --rgw-zonegroup=us --rgw-zone=chicago-data', updating the period, then re-doing the steps to add a secondary zone on the 2nd cluster worked.
Thanks for the help!
________________________________
From: Matt Dunavant
Sent: Monday, August 22, 2022 1:56:21 PM
To: Casey Bodley
Cc: ceph-users@xxxxxxx
Subject: Re: Problem adding secondary realm to rados-gw
Thanks, that got me past that issue. However I'm running into another in which it's not finding the zone after creating and then updating the period:
#radosgw-admin period update --commit --realm-id=26e78bc5-714c-4993-a4bd-b07918bd223a
2022-08-22T13:45:56.195-0400 7f7183538c80 1 Cannot find zone id=f3f76b66-d05d-4ac2-ae0a-0ea2af007ebc (name=chicago-data), switching to local zonegroup configuration
2022-08-22T13:45:56.195-0400 7f7183538c80 -1 Cannot find zone id=f3f76b66-d05d-4ac2-ae0a-0ea2af007ebc (name=chicago-data)
2022-08-22T13:45:56.195-0400 7f7183538c80 0 ERROR: failed to start notify service ((22) Invalid argument
2022-08-22T13:45:56.195-0400 7f7183538c80 0 ERROR: failed to init services (ret=(22) Invalid argument)
couldn't init storage provider
# radosgw-admin zone list
{
"default_info": "f3f76b66-d05d-4ac2-ae0a-0ea2af007ebc",
"zones": [
"chicago-dev",
"chicago-data"
]
}
The zone exists by the zone list command. Not sure what else I could be missing.
# radosgw-admin period update --commit --realm-id=26e78bc5-714c-4993-a4bd-b07918bd223a --debug-rgw=20
2022-08-22T13:54:38.464-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.468-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=46
2022-08-22T13:54:38.468-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.468-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=107
2022-08-22T13:54:38.468-0400 7ff280858c80 20 realm realm1 26e78bc5-714c-4993-a4bd-b07918bd223a
2022-08-22T13:54:38.468-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.468-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=107
2022-08-22T13:54:38.472-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.484-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=10
2022-08-22T13:54:38.484-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.508-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=1156
2022-08-22T13:54:38.508-0400 7ff280858c80 20 current period e9110c93-80e2-4da1-89b7-2634ebf64556
2022-08-22T13:54:38.508-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados_obj.operate() r=-2 bl.length=0
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=46
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=107
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.512-0400 7ff280858c80 20 rados_obj.operate() r=-2 bl.length=0
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.9b2ca341-b3b6-43e1-8211-51c1fbee6399.latest_epoch
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got realms.23113e5f-3b33-4cd8-99bb-f7a5f3fa247a
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.e9110c93-80e2-4da1-89b7-2634ebf64556.3
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.af9bb2f1-564a-4409-91f5-1a84afdc5247.latest_epoch
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got default.zone.26e78bc5-714c-4993-a4bd-b07918bd223a
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got zonegroups_names.us
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.e9110c93-80e2-4da1-89b7-2634ebf64556.1
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got realms_names.chicago-dev
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.9b2ca341-b3b6-43e1-8211-51c1fbee6399.1
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.26e78bc5-714c-4993-a4bd-b07918bd223a:staging
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got default.realm
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got zonegroup_info.b496a179-17d1-433c-a70e-5f9af6de4782
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got realms.26e78bc5-714c-4993-a4bd-b07918bd223a.control
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got realms_names.realm1
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.4be8111e-c67c-471f-91f5-d6e7f8512491:staging
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got period_config.4be8111e-c67c-471f-91f5-d6e7f8512491
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got zone_info.905ae9f2-fd56-473f-8b63-396c3f3964ec
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.e9110c93-80e2-4da1-89b7-2634ebf64556.2
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.6f18d5da-25fd-4dc3-8174-9d712d16b157.1
2022-08-22T13:54:38.572-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.af9bb2f1-564a-4409-91f5-1a84afdc5247.1
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got period_config.26e78bc5-714c-4993-a4bd-b07918bd223a
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.68096bc1-ea61-4d5c-afe2-29e980a91733.1
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got default.zonegroup.26e78bc5-714c-4993-a4bd-b07918bd223a
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got zone_names.chicago-dev
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got default.zonegroup.23113e5f-3b33-4cd8-99bb-f7a5f3fa247a
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.af9bb2f1-564a-4409-91f5-1a84afdc5247.2
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.6f18d5da-25fd-4dc3-8174-9d712d16b157.latest_epoch
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got default.zone.23113e5f-3b33-4cd8-99bb-f7a5f3fa247a
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.68096bc1-ea61-4d5c-afe2-29e980a91733.latest_epoch
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got zonegroups_names.chicago-dev
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got zonegroup_info.99569fad-bfbf-4b76-b676-837ec62769e7
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got realms.26e78bc5-714c-4993-a4bd-b07918bd223a
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got periods.e9110c93-80e2-4da1-89b7-2634ebf64556.latest_epoch
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got period_config.23113e5f-3b33-4cd8-99bb-f7a5f3fa247a
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got zone_info.f3f76b66-d05d-4ac2-ae0a-0ea2af007ebc
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got realms.23113e5f-3b33-4cd8-99bb-f7a5f3fa247a.control
2022-08-22T13:54:38.576-0400 7ff280858c80 20 RGWRados::pool_iterate: got zone_names.chicago-data
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados_obj.operate() r=-2 bl.length=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados_obj.operate() r=-2 bl.length=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados_obj.operate() r=-2 bl.length=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=46
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=107
2022-08-22T13:54:38.576-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=46
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=671
2022-08-22T13:54:38.592-0400 7ff280858c80 20 period zonegroup init ret 0
2022-08-22T13:54:38.592-0400 7ff280858c80 20 period zonegroup name us
2022-08-22T13:54:38.592-0400 7ff280858c80 20 using current period zonegroup us
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=46
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=107
2022-08-22T13:54:38.592-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.612-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=46
2022-08-22T13:54:38.612-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.624-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=1074
2022-08-22T13:54:38.624-0400 7ff280858c80 10 Cannot find current period zone using local zone
2022-08-22T13:54:38.624-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.624-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=1074
2022-08-22T13:54:38.624-0400 7ff280858c80 1 Cannot find zone id=f3f76b66-d05d-4ac2-ae0a-0ea2af007ebc (name=chicago-data), switching to local zonegroup configuration
2022-08-22T13:54:38.624-0400 7ff280858c80 20 rados->read ofs=0 len=0
2022-08-22T13:54:38.628-0400 7ff280858c80 20 rados_obj.operate() r=0 bl.length=671
2022-08-22T13:54:38.628-0400 7ff280858c80 20 zonegroup us
2022-08-22T13:54:38.628-0400 7ff280858c80 -1 Cannot find zone id=f3f76b66-d05d-4ac2-ae0a-0ea2af007ebc (name=chicago-data)
2022-08-22T13:54:38.628-0400 7ff280858c80 0 ERROR: failed to start notify service ((22) Invalid argument
2022-08-22T13:54:38.628-0400 7ff280858c80 0 ERROR: failed to init services (ret=(22) Invalid argument)
couldn't init storage provider
________________________________
From: Casey Bodley <cbodley@xxxxxxxxxx>
Sent: Monday, August 22, 2022 12:57:13 PM
To: Matt Dunavant
Cc: ceph-users@xxxxxxx
Subject: Re: Problem adding secondary realm to rados-gw
On Mon, Aug 22, 2022 at 12:37 PM Matt Dunavant
<MDunavant@xxxxxxxxxxxxxxxxxx> wrote:
>
> Hello,
>
>
> I'm trying to add a secondary realm to my ceph cluster but I'm getting the following error after running a 'radosgw-admin realm pull --rgw-realm=$REALM --url=http://URL:80 --access-key=$KEY --secret=$SECRET':
>
>
> request failed: (5) Input/output error
>
>
> Nothing on google seems to help with the error happening as the realm pull is attempted. Realm is up and running in our primary site just fime. Here's the logs when adding --debug-rgw=20:
>
>
> 2022-08-22T12:21:36.049-0400 7f46b94a9c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1]
> 2022-08-22T12:21:36.049-0400 7f46b94a9c80 20 check_secure_mon_conn(): mode 1 is insecure
> 2022-08-22T12:21:36.057-0400 7f46b94a9c80 20 > HTTP_DATE -> Mon Aug 22 16:21:36 2022
> 2022-08-22T12:21:36.057-0400 7f46b94a9c80 10 get_canon_resource(): dest=/admin/realm
> 2022-08-22T12:21:36.057-0400 7f46b94a9c80 10 generated canonical header: GET
>
>
> Mon Aug 22 16:21:36 2022
> /admin/realm
> 2022-08-22T12:21:36.057-0400 7f46b94a9c80 15 generated auth header: AWS RA91T371DX79FLGEXX3F:ZaEtUSD7tG/foKjqlaX4FFV/Z60=
> 2022-08-22T12:21:36.057-0400 7f46b94a9c80 20 sending request to http://URL:80/admin/realm?id=26e78bc5-714c-4993-a4bd-b07918bd223a
> 2022-08-22T12:21:36.057-0400 7f46b94a9c80 20 register_request mgr=0x55e4e1b7f8d0 req_data->id=0, curl_handle=0x55e4e1b7f8b0
> 2022-08-22T12:21:36.057-0400 7f468dfdb700 20 reqs_thread_entry: start
> 2022-08-22T12:21:36.057-0400 7f468dfdb700 20 link_request req_data=0x55e4e1a06330 req_data->id=0, curl_handle=0x55e4e1b7f8b0
> request failed: (5) Input/output error
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 receive_http_header
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 received header:HTTP/1.1 302 Moved Temporarily
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 receive_http_header
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 received header:Date: Mon, 22 Aug 2022 16:21:36 GMT
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 receive_http_header
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 received header:Proxy-Connection: close
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 receive_http_header
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 received header:Via: 1.1 proxy201.convokesystems.com
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 receive_http_header
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 received header:Location: http://10.2.22.20:15871/cgi-bin/blockpage.cgi?ws-session=4060937779
http://URL:80 appears to be going through a proxy that's trying to
redirect this request
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 receive_http_header
> 2022-08-22T12:21:36.113-0400 7f468dfdb700 10 received header:Content-Length: 0
>
> Ceph version is 16.2.10. Any ideas?
>
> Thanks,
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
