rgw service fails to start with zone not found

["Adiga, Anantha" <anantha.adiga@xxxxxxxxx>]

Hi,

An existing multisite configuration was removed.  But the radosgw services still see the old zone name and fail to start.

journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@rgw.default.default.fl31ca104ja0201.ninovs
...........
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740  0 deferred set uid:gid to 167:167 (ceph:ceph)
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740  0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989>
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740  0 framework: beast
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740  0 framework conf key: port, val: 8080
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740  1 radosgw_Main not setting numa affinity
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740  1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740  1 D3N datacache enabled: 0
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740  0 rgw main: ERROR: could not find zone (fl2site2)
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740  0 rgw main: ERROR: failed to start notify service>
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740  0 rgw main: ERROR: failed to init services (ret=(>
May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.921+0000 7f2f47634740 -1 Couldn't init storage provider (RADOS)
May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@rgw.default.default.fl31ca104ja0201.ninovs.service: Main pr>
May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@rgw.default.default.fl31ca104ja0201.ninovs.service: Failed

Here is the current configuration

root@fl31ca104ja0201:/# radosgw-admin period get
{
    "id": "729f7cef-6340-4750-b3ae-9164177c0df3",
    "epoch": 1,
    "predecessor_uuid": "1d124ad5-57b0-41de-8def-823bd40f72aa",
    "sync_status": [],
    "period_map": {
        "id": "729f7cef-6340-4750-b3ae-9164177c0df3",
        "zonegroups": [
            {
                "id": "21b8306c-be43-4567-a0c5-74ab69937535",
                "name": "default",
                "api_name": "default",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3",
                "zones": [
                    {
                        "id": "ff468492-8a14-414b-a8cf-7c20ab699af3",
                        "name": "default",
                        "endpoints": [],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 11,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": [],
                        "storage_classes": [
                            "STANDARD"
                        ]
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312",
                "sync_policy": {
                    "groups": []
                }
            }
        ],
        "short_zone_ids": [
            {
                "key": "ff468492-8a14-414b-a8cf-7c20ab699af3",
                "val": 3883136521
            }
        ]
    },
    "master_zonegroup": "21b8306c-be43-4567-a0c5-74ab69937535",
    "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_ratelimit": {
            "max_read_ops": 0,
            "max_write_ops": 0,
            "max_read_bytes": 0,
            "max_write_bytes": 0,
            "enabled": false
        },
        "bucket_ratelimit": {
            "max_read_ops": 0,
            "max_write_ops": 0,
            "max_read_bytes": 0,
            "max_write_bytes": 0,
            "enabled": false
        },
        "anonymous_ratelimit": {
            "max_read_ops": 0,
            "max_write_ops": 0,
            "max_read_bytes": 0,
            "max_write_bytes": 0,
            "enabled": false
        }
    },
    "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312",
    "realm_name": "default",
    "realm_epoch": 2
}

root@fl31ca104ja0201:/# radosgw-admin realm list
{
    "default_info": "8999e04c-17a4-4150-8845-cecd6672d312",
    "realms": [
        "default"
    ]
}
root@fl31ca104ja0201:/# radosgw-admin zonegroup list
{
    "default_info": "21b8306c-be43-4567-a0c5-74ab69937535",
    "zonegroups": [
        "default"
    ]
}
root@fl31ca104ja0201:/# radosgw-admin zone list
{
    "default_info": "",
    "zones": [
        "default"
    ]
}


Where is the radosgw service getting the old zone info from? How to fix this configuration?

Thank you,
Anantha

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx