Re: Luminous RGW Metadata Search

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

So I did the exact same thing using Kraken and the same set of VMs, no issue. What is the magic to make it work in Luminous? Anyone lucky enough to have this RGW ElasticSearch working using Luminous?

On Mon, Jan 8, 2018 at 10:26 AM, Youzhong Yang <youzhong@xxxxxxxxx> wrote:
Hi Yehuda,

Thanks for replying. 

>radosgw failed to connect to your ceph cluster. Does the rados command
>with the same connection params work?

I am not quite sure what to do by running rados command to test.

So I tried again, could you please take a look and check what could have gone wrong?

Here are what I did:

**** On ceph admin node, I removed installation on ceph-rgw1 and ceph-rgw2, reinstalled rgw on ceph-rgw1, stoped rgw service, removed all rgw pools. Elasticsearch is running on ceph-rgw2 node on port 9200.

ceph-deploy purge ceph-rgw1
ceph-deploy purge ceph-rgw2
ceph-deploy purgedata ceph-rgw2
ceph-deploy purgedata ceph-rgw1
ceph-deploy install --release luminous ceph-rgw1
ceph-deploy admin ceph-rgw1
ceph-deploy rgw create ceph-rgw1
ssh ceph-rgw1 sudo systemctl stop ceph-radosgw@xxxxxxxx-rgw1
rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it
rados rmpool default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it
rados rmpool .rgw.root .rgw.root --yes-i-really-really-mean-it

**** On ceph-rgw1 node:

export RGWHOST="ceph-rgw1"
export ELASTICHOST="ceph-rgw2"
export REALM="demo"
export ZONEGRP="zone1"
export ZONE1="zone1-a"
export ZONE2="zone1-b"
export SYNC_AKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1 )"
export SYNC_SKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1 )"

radosgw-admin realm create --rgw-realm=${REALM} --default
radosgw-admin zonegroup create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP} --endpoints=http://${RGWHOST}:8000 --master --default
radosgw-admin zone create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP} --rgw-zone=${ZONE1} --endpoints=http://${RGWHOST}:8000 --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --master --default
radosgw-admin user create --uid=sync --display-name="zone sync" --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --system
radosgw-admin period update --commit
sudo systemctl start ceph-radosgw@rgw.${RGWHOST}

radosgw-admin zone create --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP} --rgw-zone=${ZONE2} --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --endpoints=http://${RGWHOST}:8002
radosgw-admin zone modify --rgw-realm=${REALM} --rgw-zonegroup=${ZONEGRP} --rgw-zone=${ZONE2} --tier-type=elasticsearch --tier-config=endpoint=http://${ELASTICHOST}:9200,num_replicas=1,num_shards=10
radosgw-admin period update --commit

sudo systemctl restart ceph-radosgw@rgw.${RGWHOST}
sudo radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f --rgw-zone=${ZONE2} --rgw-frontends="civetweb port=8002"
2018-01-08 00:21:54.389432 7f0fe9cd2e80 -1 Couldn't init storage provider (RADOS)

**** As you can see, starting rgw on port 8002 failed, but rgw on port 8000 was started successfully.
**** Here are some more info which may be useful for diagnosis:

$ cat /etc/ceph/ceph.conf
[global]
fsid = 3e5a32d4-e45e-48dd-a3c5-f6f28fef8edf
mon_initial_members = ceph-mon1, ceph-osd1, ceph-osd2, ceph-osd3
mon_host = 172.30.212.226,172.30.212.227,172.30.212.228,172.30.212.250
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_pool_default_size = 2
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 100
osd_pool_default_pgp_num = 100
bluestore_compression_algorithm = zlib
bluestore_compression_mode = force
rgw_max_put_size = 21474836480
[osd]
osd_max_object_size = 1073741824
[mon]
mon_allow_pool_delete = true
[client.rgw.ceph-rgw1]
host = ceph-rgw1
rgw frontends = civetweb port=8000

$ wget -O - -q http://ceph-rgw2:9200/
{
  "name" : "Hippolyta",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.1",
    "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp" : "2016-04-04T12:25:05Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

$ ceph df
GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    719G      705G       14473M          1.96
POOLS:
    NAME                    ID     USED     %USED     MAX AVAIL     OBJECTS
    .rgw.root               17     6035         0          333G          19
    zone1-a.rgw.control     18        0         0          333G           8
    zone1-a.rgw.meta        19      350         0          333G           2
    zone1-a.rgw.log         20       50         0          333G         176
    zone1-b.rgw.control     21        0         0          333G           8
    zone1-b.rgw.meta        22        0         0          333G           0

$ rados df
POOL_NAME           USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD    WR_OPS WR
.rgw.root           6035      19      0     38                  0       0        0    817  553k     55 37888
zone1-a.rgw.control    0       8      0     16                  0       0        0      0     0      0     0
zone1-a.rgw.log       50     176      0    352                  0       0        0   3703 3522k   2451  1024
zone1-a.rgw.meta     350       2      0      4                  0       0        0      9  7168      3  2048
zone1-b.rgw.control    0       8      0     16                  0       0        0      0     0      0     0
zone1-b.rgw.meta       0       0      0      0                  0       0        0      0     0      0     0

total_objects    213
total_used       14473M
total_avail      705G
total_space      719G

$ radosgw-admin zonegroup get
{
    "id": "80a70f06-3b8f-458a-b2fc-6ea648871891",
    "name": "zone1",
    "api_name": "zone1",
    "is_master": "true",
    "endpoints": [
        "http://ceph-rgw1:8000"
    ],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "0f265e95-d12b-43af-a60c-3685ee31d267",
    "zones": [
        {
            "id": "0f265e95-d12b-43af-a60c-3685ee31d267",
            "name": "zone1-a",
            "endpoints": [
                "http://ceph-rgw1:8000"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": []
        },
        {
            "id": "37c249e7-5159-4838-b287-a3641df6f1fb",
            "name": "zone1-b",
            "endpoints": [
                "http://ceph-rgw1:8002"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "elasticsearch",
            "sync_from_all": "true",
            "sync_from": []
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "b9e5c955-a8d9-4a2a-bd0a-9b786269e909"
}

$ radosgw-admin zone get --rgw-zone=zone1-a
{
    "id": "0f265e95-d12b-43af-a60c-3685ee31d267",
    "name": "zone1-a",
    "domain_root": "zone1-a.rgw.meta:root",
    "control_pool": "zone1-a.rgw.control",
    "gc_pool": "zone1-a.rgw.log:gc",
    "lc_pool": "zone1-a.rgw.log:lc",
    "log_pool": "zone1-a.rgw.log",
    "intent_log_pool": "zone1-a.rgw.log:intent",
    "usage_log_pool": "zone1-a.rgw.log:usage",
    "reshard_pool": "zone1-a.rgw.log:reshard",
    "user_keys_pool": "zone1-a.rgw.meta:users.keys",
    "user_email_pool": "zone1-a.rgw.meta:users.email",
    "user_swift_pool": "zone1-a.rgw.meta:users.swift",
    "user_uid_pool": "zone1-a.rgw.meta:users.uid",
    "system_key": {
        "access_key": "BoBwc6kUhr5L5GXZSKu0",
        "secret_key": "TPOQcoZf9ZDtb8pRB7UsbhkxPdLRGE0ruSBSBEuJ"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "zone1-a.rgw.buckets.index",
                "data_pool": "zone1-a.rgw.buckets.data",
                "data_extra_pool": "zone1-a.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": ""
}

$ radosgw-admin zone get --rgw-zone=zone1-b
{
    "id": "37c249e7-5159-4838-b287-a3641df6f1fb",
    "name": "zone1-b",
    "domain_root": "zone1-b.rgw.meta:root",
    "control_pool": "zone1-b.rgw.control",
    "gc_pool": "zone1-b.rgw.log:gc",
    "lc_pool": "zone1-b.rgw.log:lc",
    "log_pool": "zone1-b.rgw.log",
    "intent_log_pool": "zone1-b.rgw.log:intent",
    "usage_log_pool": "zone1-b.rgw.log:usage",
    "reshard_pool": "zone1-b.rgw.log:reshard",
    "user_keys_pool": "zone1-b.rgw.meta:users.keys",
    "user_email_pool": "zone1-b.rgw.meta:users.email",
    "user_swift_pool": "zone1-b.rgw.meta:users.swift",
    "user_uid_pool": "zone1-b.rgw.meta:users.uid",
    "system_key": {
        "access_key": "BoBwc6kUhr5L5GXZSKu0",
        "secret_key": "TPOQcoZf9ZDtb8pRB7UsbhkxPdLRGE0ruSBSBEuJ"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "zone1-b.rgw.buckets.index",
                "data_pool": "zone1-b.rgw.buckets.data",
                "data_extra_pool": "zone1-b.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [
        {
            "key": "endpoint",
            "val": "http://ceph-rgw2:9200"
        },
        {
            "key": "num_replicas",
            "val": "1"
        },
        {
            "key": "num_shards",
            "val": "10"
        }
    ],
    "realm_id": "b9e5c955-a8d9-4a2a-bd0a-9b786269e909"
}

# radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f --rgw-zone=zone1-b --rgw-frontends="civetweb port=8002" --debug-rgw=255
2018-01-08 09:54:53.118526 7f681a211e80 -1 Couldn't init storage provider (RADOS)

# tail -12 /var/log/ceph/ceph-client.admin.log
2018-01-08 09:54:52.914513 7f681a211e80 20 zone zone1-b
2018-01-08 09:54:52.919424 7f681a211e80 20 add_watcher() i=0
2018-01-08 09:54:52.924267 7f681a211e80 20 add_watcher() i=1
2018-01-08 09:54:52.928440 7f681a211e80 20 add_watcher() i=2
2018-01-08 09:54:52.933062 7f681a211e80 20 add_watcher() i=3
2018-01-08 09:54:52.937396 7f681a211e80 20 add_watcher() i=4
2018-01-08 09:54:52.942317 7f681a211e80 20 add_watcher() i=5
2018-01-08 09:54:52.946186 7f681a211e80 20 add_watcher() i=6
2018-01-08 09:54:52.950883 7f681a211e80 20 add_watcher() i=7
2018-01-08 09:54:52.950904 7f681a211e80  2 all 8 watchers are set, enabling cache
2018-01-08 09:54:52.950930 7f681a211e80 20 generating connection object for zone zone1-a id 0f265e95-d12b-43af-a60c-3685ee31d267
2018-01-08 09:54:53.118526 7f681a211e80 -1 Couldn't init storage provider (RADOS)

Thanks very much,

--Youzhong


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux