Yes, you're definitely right, docs can be improved. We'd be happy to get a pull request with any improvements if someone wants to pick it up. Thanks, Uejida On Tue, Jan 16, 2018 at 1:30 PM, Youzhong Yang <youzhong@xxxxxxxxx> wrote: > My bad ... Once I sent config request to us-east-1 (the master zone), it > works, and 'obo mdsearch' against "us-east-es" zone works like a charm. > > May I suggest that the following page be modified to reflect this > requirement so that someone else won't run into the same issue? I understand > it may sound obvious to experienced users ... > > http://ceph.com/rgw/new-luminous-rgw-metadata-search/ > > Thanks a lot. > > > On Tue, Jan 16, 2018 at 3:59 PM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> > wrote: >> >> On Tue, Jan 16, 2018 at 12:20 PM, Youzhong Yang <youzhong@xxxxxxxxx> >> wrote: >> > Hi Yehuda, >> > >> > I can use your tool obo to create a bucket, and upload a file to the >> > object >> > store, but when I tried to run the following command, it failed: >> > >> > # obo mdsearch buck --config='x-amz-meta-foo; string, x-amz-meta-bar; >> > integer' >> > ERROR: {"status": 405, "resource": null, "message": "", "error_code": >> > "MethodNotAllowed", "reason": "Method Not Allowed"} >> > >> > How to make the method 'Allowed'? >> >> >> Which rgw are you sending this request to? >> >> > >> > Thanks in advance. >> > >> > On Fri, Jan 12, 2018 at 7:25 PM, Yehuda Sadeh-Weinraub >> > <yehuda@xxxxxxxxxx> >> > wrote: >> >> >> >> The errors you're seeing there don't look like related to >> >> elasticsearch. It's a generic radosgw related error that says that it >> >> failed to reach the rados (ceph) backend. You can try bumping up the >> >> messenger log (debug ms =1) and see if there's any hint in there. >> >> >> >> Yehuda >> >> >> >> On Fri, Jan 12, 2018 at 12:54 PM, Youzhong Yang <youzhong@xxxxxxxxx> >> >> wrote: >> >> > So I did the exact same thing using Kraken and the same set of VMs, >> >> > no >> >> > issue. What is the magic to make it work in Luminous? Anyone lucky >> >> > enough to >> >> > have this RGW ElasticSearch working using Luminous? >> >> > >> >> > On Mon, Jan 8, 2018 at 10:26 AM, Youzhong Yang <youzhong@xxxxxxxxx> >> >> > wrote: >> >> >> >> >> >> Hi Yehuda, >> >> >> >> >> >> Thanks for replying. >> >> >> >> >> >> >radosgw failed to connect to your ceph cluster. Does the rados >> >> >> > command >> >> >> >with the same connection params work? >> >> >> >> >> >> I am not quite sure what to do by running rados command to test. >> >> >> >> >> >> So I tried again, could you please take a look and check what could >> >> >> have >> >> >> gone wrong? >> >> >> >> >> >> Here are what I did: >> >> >> >> >> >> **** On ceph admin node, I removed installation on ceph-rgw1 and >> >> >> ceph-rgw2, reinstalled rgw on ceph-rgw1, stoped rgw service, removed >> >> >> all rgw >> >> >> pools. Elasticsearch is running on ceph-rgw2 node on port 9200. >> >> >> >> >> >> ceph-deploy purge ceph-rgw1 >> >> >> ceph-deploy purge ceph-rgw2 >> >> >> ceph-deploy purgedata ceph-rgw2 >> >> >> ceph-deploy purgedata ceph-rgw1 >> >> >> ceph-deploy install --release luminous ceph-rgw1 >> >> >> ceph-deploy admin ceph-rgw1 >> >> >> ceph-deploy rgw create ceph-rgw1 >> >> >> ssh ceph-rgw1 sudo systemctl stop ceph-radosgw@xxxxxxxx-rgw1 >> >> >> rados rmpool default.rgw.log default.rgw.log >> >> >> --yes-i-really-really-mean-it >> >> >> rados rmpool default.rgw.meta default.rgw.meta >> >> >> --yes-i-really-really-mean-it >> >> >> rados rmpool default.rgw.control default.rgw.control >> >> >> --yes-i-really-really-mean-it >> >> >> rados rmpool .rgw.root .rgw.root --yes-i-really-really-mean-it >> >> >> >> >> >> **** On ceph-rgw1 node: >> >> >> >> >> >> export RGWHOST="ceph-rgw1" >> >> >> export ELASTICHOST="ceph-rgw2" >> >> >> export REALM="demo" >> >> >> export ZONEGRP="zone1" >> >> >> export ZONE1="zone1-a" >> >> >> export ZONE2="zone1-b" >> >> >> export SYNC_AKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w >> >> >> 20 >> >> >> | >> >> >> head -n 1 )" >> >> >> export SYNC_SKEY="$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w >> >> >> 40 >> >> >> | >> >> >> head -n 1 )" >> >> >> >> >> >> radosgw-admin realm create --rgw-realm=${REALM} --default >> >> >> radosgw-admin zonegroup create --rgw-realm=${REALM} >> >> >> --rgw-zonegroup=${ZONEGRP} --endpoints=http://${RGWHOST}:8000 >> >> >> --master >> >> >> --default >> >> >> radosgw-admin zone create --rgw-realm=${REALM} >> >> >> --rgw-zonegroup=${ZONEGRP} >> >> >> --rgw-zone=${ZONE1} --endpoints=http://${RGWHOST}:8000 >> >> >> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --master --default >> >> >> radosgw-admin user create --uid=sync --display-name="zone sync" >> >> >> --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} --system >> >> >> radosgw-admin period update --commit >> >> >> sudo systemctl start ceph-radosgw@rgw.${RGWHOST} >> >> >> >> >> >> radosgw-admin zone create --rgw-realm=${REALM} >> >> >> --rgw-zonegroup=${ZONEGRP} >> >> >> --rgw-zone=${ZONE2} --access-key=${SYNC_AKEY} --secret=${SYNC_SKEY} >> >> >> --endpoints=http://${RGWHOST}:8002 >> >> >> radosgw-admin zone modify --rgw-realm=${REALM} >> >> >> --rgw-zonegroup=${ZONEGRP} >> >> >> --rgw-zone=${ZONE2} --tier-type=elasticsearch >> >> >> >> >> >> >> >> >> --tier-config=endpoint=http://${ELASTICHOST}:9200,num_replicas=1,num_shards=10 >> >> >> radosgw-admin period update --commit >> >> >> >> >> >> sudo systemctl restart ceph-radosgw@rgw.${RGWHOST} >> >> >> sudo radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f >> >> >> --rgw-zone=${ZONE2} --rgw-frontends="civetweb port=8002" >> >> >> 2018-01-08 00:21:54.389432 7f0fe9cd2e80 -1 Couldn't init storage >> >> >> provider >> >> >> (RADOS) >> >> >> >> >> >> **** As you can see, starting rgw on port 8002 failed, but rgw on >> >> >> port >> >> >> 8000 was started successfully. >> >> >> **** Here are some more info which may be useful for diagnosis: >> >> >> >> >> >> $ cat /etc/ceph/ceph.conf >> >> >> [global] >> >> >> fsid = 3e5a32d4-e45e-48dd-a3c5-f6f28fef8edf >> >> >> mon_initial_members = ceph-mon1, ceph-osd1, ceph-osd2, ceph-osd3 >> >> >> mon_host = >> >> >> 172.30.212.226,172.30.212.227,172.30.212.228,172.30.212.250 >> >> >> auth_cluster_required = cephx >> >> >> auth_service_required = cephx >> >> >> auth_client_required = cephx >> >> >> osd_pool_default_size = 2 >> >> >> osd_pool_default_min_size = 2 >> >> >> osd_pool_default_pg_num = 100 >> >> >> osd_pool_default_pgp_num = 100 >> >> >> bluestore_compression_algorithm = zlib >> >> >> bluestore_compression_mode = force >> >> >> rgw_max_put_size = 21474836480 >> >> >> [osd] >> >> >> osd_max_object_size = 1073741824 >> >> >> [mon] >> >> >> mon_allow_pool_delete = true >> >> >> [client.rgw.ceph-rgw1] >> >> >> host = ceph-rgw1 >> >> >> rgw frontends = civetweb port=8000 >> >> >> >> >> >> $ wget -O - -q http://ceph-rgw2:9200/ >> >> >> { >> >> >> "name" : "Hippolyta", >> >> >> "cluster_name" : "elasticsearch", >> >> >> "version" : { >> >> >> "number" : "2.3.1", >> >> >> "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", >> >> >> "build_timestamp" : "2016-04-04T12:25:05Z", >> >> >> "build_snapshot" : false, >> >> >> "lucene_version" : "5.5.0" >> >> >> }, >> >> >> "tagline" : "You Know, for Search" >> >> >> } >> >> >> >> >> >> $ ceph df >> >> >> GLOBAL: >> >> >> SIZE AVAIL RAW USED %RAW USED >> >> >> 719G 705G 14473M 1.96 >> >> >> POOLS: >> >> >> NAME ID USED %USED MAX AVAIL >> >> >> OBJECTS >> >> >> .rgw.root 17 6035 0 333G >> >> >> 19 >> >> >> zone1-a.rgw.control 18 0 0 333G >> >> >> 8 >> >> >> zone1-a.rgw.meta 19 350 0 333G >> >> >> 2 >> >> >> zone1-a.rgw.log 20 50 0 333G >> >> >> 176 >> >> >> zone1-b.rgw.control 21 0 0 333G >> >> >> 8 >> >> >> zone1-b.rgw.meta 22 0 0 333G >> >> >> 0 >> >> >> >> >> >> $ rados df >> >> >> POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY >> >> >> UNFOUND >> >> >> DEGRADED RD_OPS RD WR_OPS WR >> >> >> .rgw.root 6035 19 0 38 0 >> >> >> 0 >> >> >> 0 817 553k 55 37888 >> >> >> zone1-a.rgw.control 0 8 0 16 0 >> >> >> 0 >> >> >> 0 0 0 0 0 >> >> >> zone1-a.rgw.log 50 176 0 352 0 >> >> >> 0 >> >> >> 0 3703 3522k 2451 1024 >> >> >> zone1-a.rgw.meta 350 2 0 4 0 >> >> >> 0 >> >> >> 0 9 7168 3 2048 >> >> >> zone1-b.rgw.control 0 8 0 16 0 >> >> >> 0 >> >> >> 0 0 0 0 0 >> >> >> zone1-b.rgw.meta 0 0 0 0 0 >> >> >> 0 >> >> >> 0 0 0 0 0 >> >> >> >> >> >> total_objects 213 >> >> >> total_used 14473M >> >> >> total_avail 705G >> >> >> total_space 719G >> >> >> >> >> >> $ radosgw-admin zonegroup get >> >> >> { >> >> >> "id": "80a70f06-3b8f-458a-b2fc-6ea648871891", >> >> >> "name": "zone1", >> >> >> "api_name": "zone1", >> >> >> "is_master": "true", >> >> >> "endpoints": [ >> >> >> "http://ceph-rgw1:8000"; >> >> >> ], >> >> >> "hostnames": [], >> >> >> "hostnames_s3website": [], >> >> >> "master_zone": "0f265e95-d12b-43af-a60c-3685ee31d267", >> >> >> "zones": [ >> >> >> { >> >> >> "id": "0f265e95-d12b-43af-a60c-3685ee31d267", >> >> >> "name": "zone1-a", >> >> >> "endpoints": [ >> >> >> "http://ceph-rgw1:8000"; >> >> >> ], >> >> >> "log_meta": "false", >> >> >> "log_data": "true", >> >> >> "bucket_index_max_shards": 0, >> >> >> "read_only": "false", >> >> >> "tier_type": "", >> >> >> "sync_from_all": "true", >> >> >> "sync_from": [] >> >> >> }, >> >> >> { >> >> >> "id": "37c249e7-5159-4838-b287-a3641df6f1fb", >> >> >> "name": "zone1-b", >> >> >> "endpoints": [ >> >> >> "http://ceph-rgw1:8002"; >> >> >> ], >> >> >> "log_meta": "false", >> >> >> "log_data": "true", >> >> >> "bucket_index_max_shards": 0, >> >> >> "read_only": "false", >> >> >> "tier_type": "elasticsearch", >> >> >> "sync_from_all": "true", >> >> >> "sync_from": [] >> >> >> } >> >> >> ], >> >> >> "placement_targets": [ >> >> >> { >> >> >> "name": "default-placement", >> >> >> "tags": [] >> >> >> } >> >> >> ], >> >> >> "default_placement": "default-placement", >> >> >> "realm_id": "b9e5c955-a8d9-4a2a-bd0a-9b786269e909" >> >> >> } >> >> >> >> >> >> $ radosgw-admin zone get --rgw-zone=zone1-a >> >> >> { >> >> >> "id": "0f265e95-d12b-43af-a60c-3685ee31d267", >> >> >> "name": "zone1-a", >> >> >> "domain_root": "zone1-a.rgw.meta:root", >> >> >> "control_pool": "zone1-a.rgw.control", >> >> >> "gc_pool": "zone1-a.rgw.log:gc", >> >> >> "lc_pool": "zone1-a.rgw.log:lc", >> >> >> "log_pool": "zone1-a.rgw.log", >> >> >> "intent_log_pool": "zone1-a.rgw.log:intent", >> >> >> "usage_log_pool": "zone1-a.rgw.log:usage", >> >> >> "reshard_pool": "zone1-a.rgw.log:reshard", >> >> >> "user_keys_pool": "zone1-a.rgw.meta:users.keys", >> >> >> "user_email_pool": "zone1-a.rgw.meta:users.email", >> >> >> "user_swift_pool": "zone1-a.rgw.meta:users.swift", >> >> >> "user_uid_pool": "zone1-a.rgw.meta:users.uid", >> >> >> "system_key": { >> >> >> "access_key": "BoBwc6kUhr5L5GXZSKu0", >> >> >> "secret_key": "TPOQcoZf9ZDtb8pRB7UsbhkxPdLRGE0ruSBSBEuJ" >> >> >> }, >> >> >> "placement_pools": [ >> >> >> { >> >> >> "key": "default-placement", >> >> >> "val": { >> >> >> "index_pool": "zone1-a.rgw.buckets.index", >> >> >> "data_pool": "zone1-a.rgw.buckets.data", >> >> >> "data_extra_pool": "zone1-a.rgw.buckets.non-ec", >> >> >> "index_type": 0, >> >> >> "compression": "" >> >> >> } >> >> >> } >> >> >> ], >> >> >> "metadata_heap": "", >> >> >> "tier_config": [], >> >> >> "realm_id": "" >> >> >> } >> >> >> >> >> >> $ radosgw-admin zone get --rgw-zone=zone1-b >> >> >> { >> >> >> "id": "37c249e7-5159-4838-b287-a3641df6f1fb", >> >> >> "name": "zone1-b", >> >> >> "domain_root": "zone1-b.rgw.meta:root", >> >> >> "control_pool": "zone1-b.rgw.control", >> >> >> "gc_pool": "zone1-b.rgw.log:gc", >> >> >> "lc_pool": "zone1-b.rgw.log:lc", >> >> >> "log_pool": "zone1-b.rgw.log", >> >> >> "intent_log_pool": "zone1-b.rgw.log:intent", >> >> >> "usage_log_pool": "zone1-b.rgw.log:usage", >> >> >> "reshard_pool": "zone1-b.rgw.log:reshard", >> >> >> "user_keys_pool": "zone1-b.rgw.meta:users.keys", >> >> >> "user_email_pool": "zone1-b.rgw.meta:users.email", >> >> >> "user_swift_pool": "zone1-b.rgw.meta:users.swift", >> >> >> "user_uid_pool": "zone1-b.rgw.meta:users.uid", >> >> >> "system_key": { >> >> >> "access_key": "BoBwc6kUhr5L5GXZSKu0", >> >> >> "secret_key": "TPOQcoZf9ZDtb8pRB7UsbhkxPdLRGE0ruSBSBEuJ" >> >> >> }, >> >> >> "placement_pools": [ >> >> >> { >> >> >> "key": "default-placement", >> >> >> "val": { >> >> >> "index_pool": "zone1-b.rgw.buckets.index", >> >> >> "data_pool": "zone1-b.rgw.buckets.data", >> >> >> "data_extra_pool": "zone1-b.rgw.buckets.non-ec", >> >> >> "index_type": 0, >> >> >> "compression": "" >> >> >> } >> >> >> } >> >> >> ], >> >> >> "metadata_heap": "", >> >> >> "tier_config": [ >> >> >> { >> >> >> "key": "endpoint", >> >> >> "val": "http://ceph-rgw2:9200"; >> >> >> }, >> >> >> { >> >> >> "key": "num_replicas", >> >> >> "val": "1" >> >> >> }, >> >> >> { >> >> >> "key": "num_shards", >> >> >> "val": "10" >> >> >> } >> >> >> ], >> >> >> "realm_id": "b9e5c955-a8d9-4a2a-bd0a-9b786269e909" >> >> >> } >> >> >> >> >> >> # radosgw --keyring /etc/ceph/ceph.client.admin.keyring -f >> >> >> --rgw-zone=zone1-b --rgw-frontends="civetweb port=8002" >> >> >> --debug-rgw=255 >> >> >> 2018-01-08 09:54:53.118526 7f681a211e80 -1 Couldn't init storage >> >> >> provider >> >> >> (RADOS) >> >> >> >> >> >> # tail -12 /var/log/ceph/ceph-client.admin.log >> >> >> 2018-01-08 09:54:52.914513 7f681a211e80 20 zone zone1-b >> >> >> 2018-01-08 09:54:52.919424 7f681a211e80 20 add_watcher() i=0 >> >> >> 2018-01-08 09:54:52.924267 7f681a211e80 20 add_watcher() i=1 >> >> >> 2018-01-08 09:54:52.928440 7f681a211e80 20 add_watcher() i=2 >> >> >> 2018-01-08 09:54:52.933062 7f681a211e80 20 add_watcher() i=3 >> >> >> 2018-01-08 09:54:52.937396 7f681a211e80 20 add_watcher() i=4 >> >> >> 2018-01-08 09:54:52.942317 7f681a211e80 20 add_watcher() i=5 >> >> >> 2018-01-08 09:54:52.946186 7f681a211e80 20 add_watcher() i=6 >> >> >> 2018-01-08 09:54:52.950883 7f681a211e80 20 add_watcher() i=7 >> >> >> 2018-01-08 09:54:52.950904 7f681a211e80 2 all 8 watchers are set, >> >> >> enabling cache >> >> >> 2018-01-08 09:54:52.950930 7f681a211e80 20 generating connection >> >> >> object >> >> >> for zone zone1-a id 0f265e95-d12b-43af-a60c-3685ee31d267 >> >> >> 2018-01-08 09:54:53.118526 7f681a211e80 -1 Couldn't init storage >> >> >> provider >> >> >> (RADOS) >> >> >> >> >> >> Thanks very much, >> >> >> >> >> >> --Youzhong >> >> >> >> >> > >> > >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |