Problem solved - 2 of the pools (zone-2.rgw.meta and zone-2.rgw.log) did not have the "rgw" application enabled. Once that was fixed, it started working. ________________________________ From: Wyll Ingersoll <wyllys.ingersoll@xxxxxxxxxxxxxx> Sent: Tuesday, May 31, 2022 3:51 PM To: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Adding 2nd RGW zone using cephadm - fail. I'm having trouble adding a secondary zone RGW using cephadm, running with ceph 16.2.9. The master realm, zonegroup, and zone are already configured and working on another cluster. This is a new cluster configured with cephadm, everything is up and running but when I try to add an RGW and create a 2nd zone, the RGW always fails (errors below). Prior to creating the container, I created the RGW pools for the new zone (zone-2.rgw.) and pulled over the realm using "radosgw-admin realm pull ..." and that all works. I can see the zonegroup configuration. Then I create the new zone specifying the correct realm, zonegroup, and the new zone name (zone-2) and get no errors, and I can view the new zonegroup configuration with radosgw-admin. Then I update the period for the realm, successfully. I then populate a spec file for creating the RGW that looks like: --- service_type: rgw service_id: zone-2 placement: hosts: - k4 spec: rgw_frontend_type: beast rgw_frontend_port: 4443 rgw_realm: testrealm rgw_zone: zone-2 ssl: true The SSL certificate and key files are setup in the ceph config database and are ok. When the orchestrator starts the new container with the RGW, it results in an "error" state. Viewing the log I see the following errors: May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.316+0000 7ff87ddbc5c0 0 rgw main: ERROR: failed reading data (obj=zone-2.rgw.log:bucket.sync-source-hints.), r=-1 May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.316+0000 7ff87ddbc5c0 0 rgw main: ERROR: failed to update sources index for bucket=:[]) r=-1 May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.316+0000 7ff87ddbc5c0 0 rgw main: ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-1 May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.316+0000 7ff87ddbc5c0 -1 rgw main: ERROR: could not initialize zone policy handler for zone=zone-1 May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.316+0000 7ff87ddbc5c0 0 rgw main: ERROR: failed to start notify service ((1) Operation not permitted May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.316+0000 7ff87ddbc5c0 0 rgw main: ERROR: failed to init services (ret=(1) Operation not permitted) May 31 15:36:23 k4 bash[3525798]: debug 2022-05-31T19:36:23.320+0000 7ff87ddbc5c0 -1 Couldn't init storage provider (RADOS) May 31 15:36:24 k4 systemd[1]: ceph-a4cfb4a0-cbd6-11ec-b27f-c3fe3a7bc84c@rgw.keeper-2.k4.jugmpm.service: Main process exited, code=exited, status=5/NOTINSTALLED May 31 15:36:24 k4 systemd[1]: ceph-a4cfb4a0-cbd6-11ec-b27f-c3fe3a7bc84c@xxxxxxxx-2.k4.jugmpm.service: Failed with result 'exit-code'. I don't know what step I am missing. I've setup secondary zones manually before with no problems, so I wonder if there's a bug in the orchestrator when it configures rgw or if I'm just missing a parameter somewhere. thanks, Wyllys Ingersoll _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |