Re: RGW unable to start gateway for 2nd realm

Brett Chancellor <bchancellor@xxxxxxxxxxxxxx> · Thu, 31 May 2018 13:51:43 -0400

On a whim, I looked at the period and it seems to be pointing to a zonegroup and zone that don't exist. Could that be the issue? And if so, how to fix? I'm paranoid about period updates because I don't want to interrupt operations to the working realm.
## List realms, all is as expected
$ sudo radosgw-admin realm list
{
    "default_info": "273f5fb3-00a1-43a5-b742-2793d6d7dcaf",
    "realms": [
        "realm01",
        "realmtest"
    ]
}

## Show the realm
$ sudo radosgw-admin realm get --rgw-realm realmtest
{
    "id": "ef91fd09-32a9-4da6-9707-85cb9922a034",
    "name": "realmtest",
    "current_period": "d01f2684-7bb9-4d1c-895f-dc4f3288aea6",
    "epoch": 2
}

## Get the period of the 'realmtest' realm, here it shows a 'default' zonegroup and 'default' zone listed, that don't seem to exist
$ sudo radosgw-admin period get --period d01f2684-7bb9-4d1c-895f-dc4f3288aea6
{
    "id": "d01f2684-7bb9-4d1c-895f-dc4f3288aea6",
    "epoch": 1,
    "predecessor_uuid": "9fa7cf43-f47d-46a1-997f-94996535e90f",
    "sync_status": [],
    "period_map": {
        "id": "d01f2684-7bb9-4d1c-895f-dc4f3288aea6",
        "zonegroups": [
            {
                "id": "73bf78f0-d5a3-4981-8d22-211795e86dc2",
                "name": "default",
                "api_name": "",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "66a7bbfb-2ecb-40ab-8252-8fbe3a6a0893",
                "zones": [
                    {
                        "id": "66a7bbfb-2ecb-40ab-8252-8fbe3a6a0893",
                        "name": "default",
                        "endpoints": [],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "ef91fd09-32a9-4da6-9707-85cb9922a034"
            },
            {
                "id": "c733d405-1e83-414d-93b7-07a5f2cf94fa",
                "name": "maintest",
                "api_name": "maintest",
                "is_master": "false",
                "endpoints": [
                    "http://10.252.174.9:80"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "8501d6d3-4a70-452b-b546-a9b9e95c7cdf",
                "zones": [
                    {
                        "id": "8501d6d3-4a70-452b-b546-a9b9e95c7cdf",
                        "name": "lumtest",
                        "endpoints": [
                            "http://10.252.174.9:80"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "ef91fd09-32a9-4da6-9707-85cb9922a034"
            }
        ],
        "short_zone_ids": [
            {
                "key": "66a7bbfb-2ecb-40ab-8252-8fbe3a6a0893",
                "val": 4096339366
            },
            {
                "key": "8501d6d3-4a70-452b-b546-a9b9e95c7cdf",
                "val": 2637200738
            }
        ]
    },
    "master_zonegroup": "73bf78f0-d5a3-4981-8d22-211795e86dc2",
    "master_zone": "66a7bbfb-2ecb-40ab-8252-8fbe3a6a0893",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "ef91fd09-32a9-4da6-9707-85cb9922a034",
    "realm_name": "realmtest",
    "realm_epoch": 2
}

## List zonegroups, looking for 'default', don't see it
$ sudo radosgw-admin zonegroup list
{
    "default_info": "f0417b11-ebe9-4eb9-bbbe-9396c0061190",
    "zonegroups": [
        "us",
        "maintest"
    ]
}

On Thu, May 31, 2018 at 11:08 AM, David Turner <drakonstein@xxxxxxxxx> wrote:
This is the documentation that has always worked for me to set up multiple realms.  http://docs.ceph.com/docs/luminous/radosgw/multisite/  It's for the multisite configuration, but if you aren't using multi-site, just stop after setting up the first site.  I didn't read through all of your steps, but I've never had a problem following these steps.  Make sure to replace 'luminous' with your ceph version if it isn't luminous.  One thing I noticed in Luminous is that you only need like 5 metadata pools for RGW now due to namespaces in the pools.

On Wed, May 30, 2018 at 11:38 AM Brett Chancellor <bchancellor@xxxxxxxxxxxxxx> wrote:
Hi All,  I'm having issues trying to get a 2nd Rados GW realm/zone up and running.  The configuration seemed to go well, but I'm unable to start the gateway.

2018-05-29 21:21:27.119192 7fd26cfdd9c0  0 ERROR: failed to decode obj from .rgw.root:zone_info.fe2e0680-d7e8-415f-bf91-501dda96d075
2018-05-29 21:21:27.119198 7fd26cfdd9c0  0 replace_region_with_zonegroup: error initializing default zone params: (5) Input/output error
2018-05-29 21:21:27.119207 7fd26cfdd9c0 -1 failed converting region to zonegroup : ret -5 (5) Input/output error
2018-05-29 21:21:27.120479 7fd26cfdd9c0  1 -- 10.252.174.9:0/3447328109 mark_down 0x55dddc157a30 -- 0x55dddc153630
2018-05-29 21:21:27.122221 7fd26cfdd9c0  1 -- 10.252.174.9:0/3447328109 mark_down_all
2018-05-29 21:21:27.122393 7fd26cfdd9c0  1 -- 10.252.174.9:0/3447328109 shutdown complete.
2018-05-29 21:21:27.122800 7fd26cfdd9c0 -1 Couldn't init storage provider (RADOS)

Existing RadosGW .. this one works fine
Realm : realm01 (default)
zonegroup: us (master, default)
zone: us-prd-1 (master, default)

The problem comes when I'm attempting to add a new realm.
New Realm: realmtest
new ZG: maintest (master)
new zone: lumtest (master)

Steps taken:
=========
* Created new rgw pools lumtest.rgw.* (14 pools) on dedicated root
*  radosgw-admin realm create --rgw-realm=realmtest 
*  radosgw-admin zonegroup create --rgw-zonegroup=maintest --rgw-realm=realmtest --master
* radosgw-admin zone create --rgw-realm=realmtest --rgw-zonegroup=maintest --rgw-zone=lumtest --master
* radosgw-admin user create --rgw-realm realmtest --rgw-zonegroup maintest --rgw-zone lumtest --uid="REMOVED" --display-name="System User" --system
* radosgw-admin zone modify -rgw-realm realmtest --rgw-zonegroup maintest --rgw-zone lumtest  [added the access key and secret of system user]
*  radosgw-admin user create --rgw-realm realmtest --rgw-zonegroup maintest --rgw-zone lumtest --uid="test" --display-name="test User"
* radosgw-admin period update --rgw-realm realmtest

ceph.conf
========
[client.radosgw.rgw-test]
host = rgw-test
keyring = /etc/ceph/ceph.client.radosgw.rgw-test
log file = /var/log/ceph/radosgw.rgw-test
rgw frontends = civetweb port=80
rgw realm=realmtest
rgw zonegroup=maintest
rgw zone=lumtest

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com