On Thu, Nov 10, 2016 at 2:55 PM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote: > Orit, > > Here is what i've done just now: > > root@arh-ibstorage1-ib:~# service ceph-radosgw@radosgw.gateway stop > > (the above command was ran on both radosgw servers). Checked with ps and no radosgw services were running. After that I've done: > > > > root@arh-ibstorage1-ib:~# ./ceph-zones-fix.sh > + RADOSGW_ADMIN=radosgw-admin > + echo Exercise initialization code > Exercise initialization code > + radosgw-admin user info --uid=foo > could not fetch user info: no user info saved > + echo Get default zonegroup > Get default zonegroup > + radosgw-admin zonegroup get --rgw-zonegroup=default > + sed s/"id":.*/"id": "default",/g > + sed s/"master_zone.*/"master_zone": "default",/g > + echo Get default zone > Get default zone > + radosgw-admin zone get --zone-id=default > + echo Creating realm > Creating realm > + radosgw-admin realm create --rgw-realm=london-ldex > ERROR: couldn't create realm london-ldex: (17) File exists > 2016-11-10 13:44:48.872839 7f87a13d9a00 0 ERROR creating new realm object london-ldex: (17) File exists > + echo Creating default zonegroup > Creating default zonegroup > + radosgw-admin zonegroup set --rgw-zonegroup=default > { > "id": "default", > "name": "default", > "api_name": "", > "is_master": "true", > "endpoints": [], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "default", > "zones": [ > { > "id": "default", > "name": "default", > "endpoints": [], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 0, > "read_only": "false" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [] > } > ], > "default_placement": "default-placement", > "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" > } > + echo Creating default zone > Creating default zone > + radosgw-admin zone set --rgw-zone=default > zone id default{ > "id": "default", > "name": "default", > "domain_root": ".rgw", > "control_pool": ".rgw.control", > "gc_pool": ".rgw.gc", > "log_pool": ".log", > "intent_log_pool": ".intent-log", > "usage_log_pool": ".usage", > "user_keys_pool": ".users", > "user_email_pool": ".users.email", > "user_swift_pool": ".users.swift", > "user_uid_pool": ".users.uid", > "system_key": { > "access_key": "", > "secret_key": "" > }, > "placement_pools": [ > { > "key": "default-placement", > "val": { > "index_pool": ".rgw.buckets.index", > "data_pool": ".rgw.buckets", > "data_extra_pool": "default.rgw.buckets.non-ec", > "index_type": 0 > } > } > ], > "metadata_heap": ".rgw.meta", > "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" > } > + echo Setting default zonegroup to 'default' > Setting default zonegroup to 'default' > + radosgw-admin zonegroup default --rgw-zonegroup=default > + echo Setting default zone to 'default' > Setting default zone to 'default' > + radosgw-admin zone default --rgw-zone=default > > > > > > > After that i've ran the following to make sure the details have been updated: > > > > > root@arh-ibstorage1-ib:~# radosgw-admin zone get --rgw-zone=default > { > "id": "default", > "name": "default", > "domain_root": ".rgw", > "control_pool": ".rgw.control", > "gc_pool": ".rgw.gc", > "log_pool": ".log", > "intent_log_pool": ".intent-log", > "usage_log_pool": ".usage", > "user_keys_pool": ".users", > "user_email_pool": ".users.email", > "user_swift_pool": ".users.swift", > "user_uid_pool": ".users.uid", > "system_key": { > "access_key": "", > "secret_key": "" > }, > "placement_pools": [ > { > "key": "default-placement", > "val": { > "index_pool": ".rgw.buckets.index", > "data_pool": ".rgw.buckets", > "data_extra_pool": "default.rgw.buckets.non-ec", > "index_type": 0 > } > } > ], > "metadata_heap": ".rgw.meta", > "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" > } > > > > root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default > { > "id": "default", > "name": "default", > "api_name": "", > "is_master": "true", > "endpoints": [], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "default", > "zones": [ > { > "id": "default", > "name": "default", > "endpoints": [], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 0, > "read_only": "false" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [] > } > ], > "default_placement": "default-placement", > "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" > } > > > > > As far as I can see, the master_zone is now set to default. > > Now I start the radosgw service: > > > root@arh-ibstorage1-ib:~# service ceph-radosgw@radosgw.gateway start > root@arh-ibstorage1-ib:~# > root@arh-ibstorage1-ib:~# > root@arh-ibstorage1-ib:~# > root@arh-ibstorage1-ib:~# > root@arh-ibstorage1-ib:~# > root@arh-ibstorage1-ib:~# radosgw-admin zone get --rgw-zone=default > { > "id": "default", > "name": "default", > "domain_root": ".rgw", > "control_pool": ".rgw.control", > "gc_pool": ".rgw.gc", > "log_pool": ".log", > "intent_log_pool": ".intent-log", > "usage_log_pool": ".usage", > "user_keys_pool": ".users", > "user_email_pool": ".users.email", > "user_swift_pool": ".users.swift", > "user_uid_pool": ".users.uid", > "system_key": { > "access_key": "", > "secret_key": "" > }, > "placement_pools": [ > { > "key": "default-placement", > "val": { > "index_pool": ".rgw.buckets.index", > "data_pool": ".rgw.buckets", > "data_extra_pool": "default.rgw.buckets.non-ec", > "index_type": 0 > } > } > ], > "metadata_heap": ".rgw.meta", > "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" > > > root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default > { > "id": "default", > "name": "default", > "api_name": "", > "is_master": "true", > "endpoints": [], > "hostnames": [], > "hostnames_s3website": [], > "master_zone": "", > "zones": [ > { > "id": "default", > "name": "default", > "endpoints": [], > "log_meta": "false", > "log_data": "false", > "bucket_index_max_shards": 0, > "read_only": "false" > } > ], > "placement_targets": [ > { > "name": "default-placement", > "tags": [] > } > ], > "default_placement": "default-placement", > "realm_id": "" > } > > > > > > > As you can see from above, the realm_id and the master_zone have reverted back to being blank! Crazy!!! > Sad not crazy :( Are both radosgw and all the radosgw-admin are from jewel? I suspect you are hitting http://tracker.ceph.com/issues/17371 (it is also true for infernalis) It was fixed in 10.2.4 ... > > Andrei > > > > > > > > > > > > > > > ----- Original Message ----- >> From: "Orit Wasserman" <owasserm@xxxxxxxxxx> >> To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx> >> Cc: "Yoann Moulin" <yoann.moulin@xxxxxxx>, "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >> Sent: Thursday, 10 November, 2016 13:36:01 >> Subject: Re: radosgw - http status 400 while creating a bucket > >> On Thu, Nov 10, 2016 at 2:24 PM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote: >>> >>> Hi Orit, >>> >>> Thanks for the links. >>> >>> I've had a look at the link that you've sent >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011157.html and >>> followed the instructions. Created the script as depicted in the email. Changed >>> the realm name to something relevant. The script ran without any errors. I've >>> restarted radosgw services on both servers, but I am still unable to create >>> buckets. I am getting exactly the same error from the client: >>> >> >> So you have two radosgw running on different servers, on the same zone >> (default) and on the same rados cluster? >> Are both gateways from the same version? >> Did you shutdown both gateway when you run the script? >> >>> >>> S3ResponseError: 400 Bad Request >>> <?xml version="1.0" >>> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>my-new-bucket-31337</BucketName><RequestId>tx000000000000000000003-00582472a4-995ee8c-default</RequestId><HostId>995ee8c-default-default</HostId></Error> >>> >>> >>> I can delete a bucket, but I can't create a new one. >>> >>> >>> What I did notice is that after running the script and getting the zonegroup >>> info, I do see both the master_zone and the realm_id fields are set: >>> >>> >>> >>> "master_zone": "default", >>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" >>> >>> >>> However, after I restart radosgw service, they go back to being blank: >>> >>> >>> "master_zone": "", >>> "realm_id": "" >>> >>> >>> In any case, creation of buckets doesn't work either way. >>> >>> Cheers >>> >>> >>> Andrei >>> >>> >>> ----- Original Message ----- >>>> From: "Orit Wasserman" <owasserm@xxxxxxxxxx> >>>> To: "Yoann Moulin" <yoann.moulin@xxxxxxx> >>>> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >>>> Sent: Thursday, 10 November, 2016 10:04:46 >>>> Subject: Re: radosgw - http status 400 while creating a bucket >>> >>>> On Wed, Nov 9, 2016 at 10:20 PM, Yoann Moulin <yoann.moulin@xxxxxxx> wrote: >>>>> Hello, >>>>> >>>>>> many thanks for your help. I've tried setting the zone to master, followed by >>>>>> the period update --commit command. This is what i've had: >>>>> >>>>> maybe it's related to this issue : >>>>> >>>>> http://tracker.ceph.com/issues/16839 (fixe in Jewel 10.2.3) >>>>> >>>>> or this one : >>>>> >>>>> http://tracker.ceph.com/issues/17239 >>>>> >>>>> the "id" of the zonegroup shouldn't be "default" but an uuid afaik >>>>> >>>>> Best regards >>>>> >>>>> Yoann Moulin >>>>> >>>>>> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default >>>>>> { >>>>>> "id": "default", >>>>>> "name": "default", >>>>>> "api_name": "", >>>>>> "is_master": "true", >>>>>> "endpoints": [], >>>>>> "hostnames": [], >>>>>> "hostnames_s3website": [], >>>>>> "master_zone": "default", >>>>>> "zones": [ >>>>>> { >>>>>> "id": "default", >>>>>> "name": "default", >>>>>> "endpoints": [], >>>>>> "log_meta": "false", >>>>>> "log_data": "false", >>>>>> "bucket_index_max_shards": 0, >>>>>> "read_only": "false" >>>>>> } >>>>>> ], >>>>>> "placement_targets": [ >>>>>> { >>>>>> "name": "default-placement", >>>>>> "tags": [] >>>>>> } >>>>>> ], >>>>>> "default_placement": "default-placement", >>>>>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" >>>>>> } >>>>>> >>>>>> >>>>>> root@arh-ibstorage1-ib:~# radosgw-admin period update --commit >>>>>> cannot commit period: period does not have a master zone of a master zonegroup >>>>>> failed to commit period: (22) Invalid argument >>>>>> >>>>>> >>>> >>>> This is issue http://tracker.ceph.com/issues/17364 (I am working on it >>>> at he moment). >>>> >>>> Do the same procedure as before without the period update --commit command, >>>> it should fix the master zone problem. >>>> see also >>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011157.html >>>> >>>> This looks like an upgraded system (the id equals the name after an upgrade). >>>> >>>> Orit >>>> >>>>>> root@arh-ibstorage1-ib:~# radosgw-admin zonegroup get --rgw-zonegroup=default >>>>>> { >>>>>> "id": "default", >>>>>> "name": "default", >>>>>> "api_name": "", >>>>>> "is_master": "true", >>>>>> "endpoints": [], >>>>>> "hostnames": [], >>>>>> "hostnames_s3website": [], >>>>>> "master_zone": "", >>>>>> "zones": [ >>>>>> { >>>>>> "id": "default", >>>>>> "name": "default", >>>>>> "endpoints": [], >>>>>> "log_meta": "false", >>>>>> "log_data": "false", >>>>>> "bucket_index_max_shards": 0, >>>>>> "read_only": "false" >>>>>> } >>>>>> ], >>>>>> "placement_targets": [ >>>>>> { >>>>>> "name": "default-placement", >>>>>> "tags": [] >>>>>> } >>>>>> ], >>>>>> "default_placement": "default-placement", >>>>>> "realm_id": "" >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> The strange thing as you can see, following the "radosgw-admin period update >>>>>> --commit" command, the master_zone and the realm_id values reset to blank. What >>>>>> could be causing this? >>>>>> >>>>>> Here is my ceph infrastructure setup, perhaps it will help with finding the >>>>>> issue?: >>>>>> >>>>>> ceph osd and mon servers: >>>>>> arh-ibstorage1-ib (also radosgw server) >>>>>> arh-ibstorage2-ib (also radosgw server) >>>>>> arh-ibstorage3-ib >>>>>> >>>>>> ceph mon server: >>>>>> arh-cloud13-ib >>>>>> >>>>>> >>>>>> >>>>>> Thus, overall, i have 4 mon servers, 3 osd servers and 2 radosgw servers >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx> >>>>>>> To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx> >>>>>>> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >>>>>>> Sent: Wednesday, 9 November, 2016 17:12:30 >>>>>>> Subject: Re: radosgw - http status 400 while creating a bucket >>>>>> >>>>>>> On Wed, Nov 9, 2016 at 1:30 AM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote: >>>>>>>> Hi Yehuda, >>>>>>>> >>>>>>>> just tried to run the command to set the master_zone to default followed by the >>>>>>>> bucket create without doing the restart and I still have the same error on the >>>>>>>> client: >>>>>>>> >>>>>>>> <?xml version="1.0" >>>>>>>> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>my-new-bucket-31337</BucketName><RequestId>tx000000000000000000010-005822ebbd-9951ad8-default</RequestId><HostId>9951ad8-default-default</HostId></Error> >>>>>>>> >>>>>>> >>>>>>> After setting the master zone, try running: >>>>>>> >>>>>>> $ radosgw-admin period update --commit >>>>>>> >>>>>>> Yehuda >>>>>>> >>>>>>>> >>>>>>>> Andrei >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx> >>>>>>>>> To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx> >>>>>>>>> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >>>>>>>>> Sent: Wednesday, 9 November, 2016 01:13:48 >>>>>>>>> Subject: Re: radosgw - http status 400 while creating a bucket >>>>>>>> >>>>>>>>> On Tue, Nov 8, 2016 at 5:05 PM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote: >>>>>>>>>> Hi Yehuda, >>>>>>>>>> >>>>>>>>>> I don't have a multizone setup. The radosgw service was configured about two >>>>>>>>>> years ago according to the documentation on ceph.com and haven't changed with >>>>>>>>>> numerous version updates. All was working okay until i've upgraded to version >>>>>>>>>> 10.2.x. >>>>>>>>>> >>>>>>>>>> Could you please point me in the right direction what exactly needs to be done? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> # radosgw-admin zonegroup get --rgw-zonegroup=default >>>>>>>>>> { >>>>>>>>>> "id": "default", >>>>>>>>>> "name": "default", >>>>>>>>>> "api_name": "", >>>>>>>>>> "is_master": "true", >>>>>>>>>> "endpoints": [], >>>>>>>>>> "hostnames": [], >>>>>>>>>> "hostnames_s3website": [], >>>>>>>>>> "master_zone": "", >>>>>>>>>> "zones": [ >>>>>>>>>> { >>>>>>>>>> "id": "default", >>>>>>>>>> "name": "default", >>>>>>>>>> "endpoints": [], >>>>>>>>>> "log_meta": "false", >>>>>>>>>> "log_data": "false", >>>>>>>>>> "bucket_index_max_shards": 0, >>>>>>>>>> "read_only": "false" >>>>>>>>>> } >>>>>>>>>> ], >>>>>>>>>> "placement_targets": [ >>>>>>>>>> { >>>>>>>>>> "name": "default-placement", >>>>>>>>>> "tags": [] >>>>>>>>>> } >>>>>>>>>> ], >>>>>>>>>> "default_placement": "default-placement", >>>>>>>>>> "realm_id": "" >>>>>>>>>> } >>>>>>>>> >>>>>>>>> Try: >>>>>>>>> >>>>>>>>> $ radosgw-admin zonegroup get --rgw-zonegroup=default > zonegroup.json >>>>>>>>> >>>>>>>>> ... modify the master_zone to be "default" >>>>>>>>> >>>>>>>>> $ radosgw-admin zonegroup set --rgw-zonegroup=default < zonegroup.json >>>>>>>>> >>>>>>>>> (restart radosgw) >>>>>>>>> >>>>>>>>> Yehuda >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> # radosgw-admin zone get --rgw-zone=default >>>>>>>>>> { >>>>>>>>>> "id": "default", >>>>>>>>>> "name": "default", >>>>>>>>>> "domain_root": ".rgw", >>>>>>>>>> "control_pool": ".rgw.control", >>>>>>>>>> "gc_pool": ".rgw.gc", >>>>>>>>>> "log_pool": ".log", >>>>>>>>>> "intent_log_pool": ".intent-log", >>>>>>>>>> "usage_log_pool": ".usage", >>>>>>>>>> "user_keys_pool": ".users", >>>>>>>>>> "user_email_pool": ".users.email", >>>>>>>>>> "user_swift_pool": ".users.swift", >>>>>>>>>> "user_uid_pool": ".users.uid", >>>>>>>>>> "system_key": { >>>>>>>>>> "access_key": "", >>>>>>>>>> "secret_key": "" >>>>>>>>>> }, >>>>>>>>>> "placement_pools": [ >>>>>>>>>> { >>>>>>>>>> "key": "default-placement", >>>>>>>>>> "val": { >>>>>>>>>> "index_pool": ".rgw.buckets.index", >>>>>>>>>> "data_pool": ".rgw.buckets", >>>>>>>>>> "data_extra_pool": "default.rgw.buckets.non-ec", >>>>>>>>>> "index_type": 0 >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> ], >>>>>>>>>> "metadata_heap": ".rgw.meta", >>>>>>>>>> "realm_id": "5b41b1b2-0f92-463d-b582-07552f83e66c" >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx> >>>>>>>>>>> To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx> >>>>>>>>>>> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >>>>>>>>>>> Sent: Wednesday, 9 November, 2016 00:48:50 >>>>>>>>>>> Subject: Re: radosgw - http status 400 while creating a bucket >>>>>>>>>> >>>>>>>>>>> On Tue, Nov 8, 2016 at 3:36 PM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote: >>>>>>>>>>>> Hello >>>>>>>>>>>> >>>>>>>>>>>> I am having issues with creating buckets in radosgw. It started with an >>>>>>>>>>>> upgrade to version 10.2.x >>>>>>>>>>>> >>>>>>>>>>>> When I am creating a bucket I get the following error on the client side: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request >>>>>>>>>>>> <?xml version="1.0" >>>>>>>>>>>> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>my-new-bucket-31337</BucketName><RequestId>tx000000000000000000002-0058225bae-994d148-default</RequestId><HostId>994d148-default-default</HostId></Error> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The radosgw logs are (redacted): >>>>>>>>>>>> >>>>>>>>>>>> ############################### >>>>>>>>>>>> >>>>>>>>>>>> 2016-11-08 23:11:42.876862 7f026d953700 20 enqueued request >>>>>>>>>>>> req=0x7f02ba07b0e0 >>>>>>>>>>>> 2016-11-08 23:11:42.876892 7f026d953700 20 RGWWQ: >>>>>>>>>>>> 2016-11-08 23:11:42.876897 7f026d953700 20 req: 0x7f02ba07b0e0 >>>>>>>>>>>> 2016-11-08 23:11:42.876912 7f026d953700 10 allocated request >>>>>>>>>>>> req=0x7f02ba07b140 >>>>>>>>>>>> 2016-11-08 23:11:42.876975 7f026b94f700 20 dequeued request >>>>>>>>>>>> req=0x7f02ba07b0e0 >>>>>>>>>>>> 2016-11-08 23:11:42.876987 7f026b94f700 20 RGWWQ: empty >>>>>>>>>>>> 2016-11-08 23:11:42.877050 7f026b94f700 20 CONTENT_LENGTH=0 >>>>>>>>>>>> 2016-11-08 23:11:42.877060 7f026b94f700 20 CONTEXT_DOCUMENT_ROOT=/var/www >>>>>>>>>>>> 2016-11-08 23:11:42.877062 7f026b94f700 20 CONTEXT_PREFIX= >>>>>>>>>>>> 2016-11-08 23:11:42.877063 7f026b94f700 20 DOCUMENT_ROOT=/var/www >>>>>>>>>>>> 2016-11-08 23:11:42.877081 7f026b94f700 20 FCGI_ROLE=RESPONDER >>>>>>>>>>>> 2016-11-08 23:11:42.877083 7f026b94f700 20 GATEWAY_INTERFACE=CGI/1.1 >>>>>>>>>>>> 2016-11-08 23:11:42.877084 7f026b94f700 20 HTTP_ACCEPT_ENCODING=identity >>>>>>>>>>>> 2016-11-08 23:11:42.877086 7f026b94f700 20 HTTP_AUTHORIZATION=AWS >>>>>>>>>>>> XXXXXXXEDITEDXXXXXX:EDITEDXXXXXXXeWyiacaN26GcME >>>>>>>>>>>> 2016-11-08 23:11:42.877087 7f026b94f700 20 HTTP_DATE=Tue, 08 Nov 2016 >>>>>>>>>>>> 23:11:37 GMT >>>>>>>>>>>> 2016-11-08 23:11:42.877088 7f026b94f700 20 >>>>>>>>>>>> HTTP_HOST=s3service.editedname.com >>>>>>>>>>>> 2016-11-08 23:11:42.877089 7f026b94f700 20 HTTP_USER_AGENT=Boto/2.38.0 >>>>>>>>>>>> Python/2.7.12 Linux/4.8.4-040804-generic >>>>>>>>>>>> 2016-11-08 23:11:42.877090 7f026b94f700 20 HTTPS=on >>>>>>>>>>>> 2016-11-08 23:11:42.877092 7f026b94f700 20 >>>>>>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>>>>>>>> 2016-11-08 23:11:42.877093 7f026b94f700 20 proxy-nokeepalive=1 >>>>>>>>>>>> 2016-11-08 23:11:42.877094 7f026b94f700 20 QUERY_STRING= >>>>>>>>>>>> 2016-11-08 23:11:42.877095 7f026b94f700 20 REMOTE_ADDR=192.168.169.91 >>>>>>>>>>>> 2016-11-08 23:11:42.877096 7f026b94f700 20 REMOTE_PORT=45404 >>>>>>>>>>>> 2016-11-08 23:11:42.877097 7f026b94f700 20 REQUEST_METHOD=PUT >>>>>>>>>>>> 2016-11-08 23:11:42.877098 7f026b94f700 20 REQUEST_SCHEME=https >>>>>>>>>>>> 2016-11-08 23:11:42.877099 7f026b94f700 20 REQUEST_URI=/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.877104 7f026b94f700 20 >>>>>>>>>>>> SCRIPT_FILENAME=proxy:fcgi://localhost:9000/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.877105 7f026b94f700 20 SCRIPT_NAME=/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.877107 7f026b94f700 20 >>>>>>>>>>>> SCRIPT_URI=https://s3service.editedname.com/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.877108 7f026b94f700 20 SCRIPT_URL=/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.877109 7f026b94f700 20 SERVER_ADDR=192.168.169.201 >>>>>>>>>>>> 2016-11-08 23:11:42.877110 7f026b94f700 20 >>>>>>>>>>>> SERVER_ADMIN=andrei@xxxxxxxxxxxxxx >>>>>>>>>>>> 2016-11-08 23:11:42.877111 7f026b94f700 20 >>>>>>>>>>>> SERVER_NAME=s3service.editedname.com >>>>>>>>>>>> 2016-11-08 23:11:42.877112 7f026b94f700 20 SERVER_PORT=443 >>>>>>>>>>>> 2016-11-08 23:11:42.877113 7f026b94f700 20 SERVER_PROTOCOL=HTTP/1.1 >>>>>>>>>>>> 2016-11-08 23:11:42.877114 7f026b94f700 20 SERVER_SIGNATURE= >>>>>>>>>>>> 2016-11-08 23:11:42.877115 7f026b94f700 20 SERVER_SOFTWARE=Apache/2.4.18 >>>>>>>>>>>> (Ubuntu) >>>>>>>>>>>> 2016-11-08 23:11:42.877116 7f026b94f700 20 >>>>>>>>>>>> SSL_TLS_SNI=s3service.editedname.com >>>>>>>>>>>> 2016-11-08 23:11:42.877119 7f026b94f700 1 ====== starting new request >>>>>>>>>>>> req=0x7f02ba07b0e0 ===== >>>>>>>>>>>> 2016-11-08 23:11:42.877155 7f026b94f700 2 req 2:0.000035::PUT >>>>>>>>>>>> /my-new-bucket-31337/::initializing for trans_id = >>>>>>>>>>>> tx000000000000000000002-0058225bae-994d148-default >>>>>>>>>>>> 2016-11-08 23:11:42.877175 7f026b94f700 10 rgw api priority: s3=5 >>>>>>>>>>>> s3website=4 >>>>>>>>>>>> 2016-11-08 23:11:42.877179 7f026b94f700 10 host=s3service.editedname.com >>>>>>>>>>>> 2016-11-08 23:11:42.877199 7f026b94f700 20 subdomain= >>>>>>>>>>>> domain=s3service.editedname.com in_hosted_domain=1 >>>>>>>>>>>> in_hosted_domain_s3website=0 >>>>>>>>>>>> 2016-11-08 23:11:42.877203 7f026b94f700 20 final domain/bucket subdomain= >>>>>>>>>>>> domain=s3service.editedname.com in_hosted_domain=1 >>>>>>>>>>>> in_hosted_domain_s3website=0 s->info.domain=s3service.editedname.com >>>>>>>>>>>> s->info.request_uri=/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.877277 7f026b94f700 20 get_handler >>>>>>>>>>>> handler=25RGWHandler_REST_Bucket_S3 >>>>>>>>>>>> 2016-11-08 23:11:42.877286 7f026b94f700 10 >>>>>>>>>>>> handler=25RGWHandler_REST_Bucket_S3 >>>>>>>>>>>> 2016-11-08 23:11:42.877291 7f026b94f700 2 req 2:0.000172:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/::getting op 1 >>>>>>>>>>>> 2016-11-08 23:11:42.877326 7f026b94f700 10 op=27RGWCreateBucket_ObjStore_S3 >>>>>>>>>>>> 2016-11-08 23:11:42.877334 7f026b94f700 2 req 2:0.000215:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:authorizing >>>>>>>>>>>> 2016-11-08 23:11:42.877386 7f026b94f700 20 get_system_obj_state: >>>>>>>>>>>> rctx=0x7f026b94b7c0 obj=.users:XXXXXXXEDITEDXXXXXX state=0x7f02b70cdfe8 >>>>>>>>>>>> s->prefetch_data=0 >>>>>>>>>>>> 2016-11-08 23:11:42.877403 7f026b94f700 10 cache get: >>>>>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX : miss >>>>>>>>>>>> 2016-11-08 23:11:42.877516 7f026b94f700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> --> 192.168.168.201:6829/893 -- osd_op(client.160747848.0:1019 16.e7d6c03f >>>>>>>>>>>> XXXXXXXEDITEDXXXXXX [getxattrs,stat] snapc 0=[] ack+read+known_if_red >>>>>>>>>>>> irected e97148) v7 -- ?+0 0x7f02b70ab600 con 0x7f02c581a800 >>>>>>>>>>>> 2016-11-08 23:11:42.879265 7f02e9405700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> <== osd.3 192.168.168.201:6829/893 52 ==== osd_op_reply(1019 >>>>>>>>>>>> XXXXXXXEDITEDXXXXXX [getxattrs,stat] v0'0 uv7 ondisk = 0) v7 ==== 182+0+20 >>>>>>>>>>>> (2521936 >>>>>>>>>>>> 738 0 3070622072) 0x7f02c9897280 con 0x7f02c581a800 >>>>>>>>>>>> 2016-11-08 23:11:42.879391 7f026b94f700 10 cache put: >>>>>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX info.flags=6 >>>>>>>>>>>> 2016-11-08 23:11:42.879421 7f026b94f700 10 adding .users+XXXXXXXEDITEDXXXXXX >>>>>>>>>>>> to cache LRU end >>>>>>>>>>>> 2016-11-08 23:11:42.879442 7f026b94f700 20 get_system_obj_state: s->obj_tag >>>>>>>>>>>> was set empty >>>>>>>>>>>> 2016-11-08 23:11:42.879452 7f026b94f700 10 cache get: >>>>>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX : type miss (requested=1, cached=6) >>>>>>>>>>>> 2016-11-08 23:11:42.879461 7f026b94f700 20 get_system_obj_state: >>>>>>>>>>>> rctx=0x7f026b94b7c0 obj=.users:XXXXXXXEDITEDXXXXXX state=0x7f02b70cdfe8 >>>>>>>>>>>> s->prefetch_data=0 >>>>>>>>>>>> 2016-11-08 23:11:42.879480 7f026b94f700 20 rados->read ofs=0 len=524288 >>>>>>>>>>>> 2016-11-08 23:11:42.879533 7f026b94f700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> --> 192.168.168.201:6829/893 -- osd_op(client.160747848.0:1020 16.e7d6c03f >>>>>>>>>>>> XXXXXXXEDITEDXXXXXX [read 0~524288] snapc 0=[] ack+read+known_if_redi >>>>>>>>>>>> rected e97148) v7 -- ?+0 0x7f02b70ab980 con 0x7f02c581a800 >>>>>>>>>>>> 2016-11-08 23:11:42.880343 7f02e9405700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> <== osd.3 192.168.168.201:6829/893 53 ==== osd_op_reply(1020 >>>>>>>>>>>> XXXXXXXEDITEDXXXXXX [read 0~37] v0'0 uv7 ondisk = 0) v7 ==== 140+0+37 >>>>>>>>>>>> (3791206703 0 >>>>>>>>>>>> 1897156770) 0x7f02c9897280 con 0x7f02c581a800 >>>>>>>>>>>> 2016-11-08 23:11:42.880433 7f026b94f700 20 rados->read r=0 bl.length=37 >>>>>>>>>>>> 2016-11-08 23:11:42.880464 7f026b94f700 10 cache put: >>>>>>>>>>>> name=.users+XXXXXXXEDITEDXXXXXX info.flags=1 >>>>>>>>>>>> 2016-11-08 23:11:42.880476 7f026b94f700 10 moving .users+XXXXXXXEDITEDXXXXXX >>>>>>>>>>>> to cache LRU end >>>>>>>>>>>> 2016-11-08 23:11:42.880509 7f026b94f700 20 get_system_obj_state: >>>>>>>>>>>> rctx=0x7f026b94b3f0 obj=.users.uid:EDITED - client name state=0x7f02b70cede8 >>>>>>>>>>>> s->prefetch_data=0 >>>>>>>>>>>> 2016-11-08 23:11:42.880519 7f026b94f700 10 cache get: name=.users.uid+EDITED >>>>>>>>>>>> - client name : miss >>>>>>>>>>>> 2016-11-08 23:11:42.880587 7f026b94f700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> --> 192.168.168.203:6825/25725 -- osd_op(client.160747848.0:1021 15.d14b4318 >>>>>>>>>>>> EDITED - client name [call version.read,getxattrs,stat] >>>>>>>>>>>> snapc 0=[] ack+read+known_if_redirected e97148) v7 -- ?+0 0x7f02b70abd00 >>>>>>>>>>>> con 0x7f02b9057180 >>>>>>>>>>>> 2016-11-08 23:11:42.882156 7f02bbefe700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> <== osd.27 192.168.168.203:6825/25725 33 ==== osd_op_reply(1021 EDITED - >>>>>>>>>>>> client name [call,getxattrs,stat] v0'0 uv14016 ondisk = 0) >>>>>>>>>>>> v7 ==== 237+0+139 (1797997075 0 3786209825) 0x7f02bc847600 con >>>>>>>>>>>> 0x7f02b9057180 >>>>>>>>>>>> 2016-11-08 23:11:42.882322 7f026b94f700 10 cache put: name=.users.uid+EDITED >>>>>>>>>>>> - client name info.flags=22 >>>>>>>>>>>> 2016-11-08 23:11:42.882353 7f026b94f700 10 adding .users.uid+EDITED - client >>>>>>>>>>>> name to cache LRU end >>>>>>>>>>>> 2016-11-08 23:11:42.882363 7f026b94f700 20 get_system_obj_state: s->obj_tag >>>>>>>>>>>> was set empty >>>>>>>>>>>> 2016-11-08 23:11:42.882374 7f026b94f700 10 cache get: name=.users.uid+EDITED >>>>>>>>>>>> - client name : type miss (requested=17, cached=22) >>>>>>>>>>>> 2016-11-08 23:11:42.882383 7f026b94f700 20 get_system_obj_state: >>>>>>>>>>>> rctx=0x7f026b94b3f0 obj=.users.uid:EDITED - client name state=0x7f02b70cede8 >>>>>>>>>>>> s->prefetch_data=0 >>>>>>>>>>>> 2016-11-08 23:11:42.882427 7f026b94f700 20 rados->read ofs=0 len=524288 >>>>>>>>>>>> 2016-11-08 23:11:42.882492 7f026b94f700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> --> 192.168.168.203:6825/25725 -- osd_op(client.160747848.0:1022 15.d14b4318 >>>>>>>>>>>> EDITED - client name [call version.check_conds,call ver >>>>>>>>>>>> sion.read,read 0~524288] snapc 0=[] ack+read+known_if_redirected e97148) v7 >>>>>>>>>>>> -- ?+0 0x7f02b70ac400 con 0x7f02b9057180 >>>>>>>>>>>> 2016-11-08 23:11:42.883550 7f02bbefe700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> <== osd.27 192.168.168.203:6825/25725 34 ==== osd_op_reply(1022 EDITED - >>>>>>>>>>>> client name [call,call,read 0~401] v0'0 uv14016 ondisk = 0) >>>>>>>>>>>> v7 ==== 237+0+449 (131137144 0 372490823) 0x7f02bc847980 con 0x7f02b9057180 >>>>>>>>>>>> 2016-11-08 23:11:42.883653 7f026b94f700 20 rados->read r=0 bl.length=401 >>>>>>>>>>>> 2016-11-08 23:11:42.883695 7f026b94f700 10 cache put: name=.users.uid+EDITED >>>>>>>>>>>> - client name info.flags=17 >>>>>>>>>>>> 2016-11-08 23:11:42.883698 7f026b94f700 10 moving .users.uid+EDITED - client >>>>>>>>>>>> name to cache LRU end >>>>>>>>>>>> 2016-11-08 23:11:42.883726 7f026b94f700 10 chain_cache_entry: >>>>>>>>>>>> cache_locator=.users.uid+EDITED - client name >>>>>>>>>>>> 2016-11-08 23:11:42.883777 7f026b94f700 10 get_canon_resource(): >>>>>>>>>>>> dest=/my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.883782 7f026b94f700 10 auth_hdr: >>>>>>>>>>>> PUT >>>>>>>>>>>> >>>>>>>>>>>> Tue, 08 Nov 2016 23:11:37 GMT >>>>>>>>>>>> /my-new-bucket-31337/ >>>>>>>>>>>> 2016-11-08 23:11:42.883849 7f026b94f700 15 calculated >>>>>>>>>>>> digest=EDITEDXXXXXXXeWyiacaN26GcME >>>>>>>>>>>> 2016-11-08 23:11:42.883852 7f026b94f700 15 >>>>>>>>>>>> auth_sign=EDITEDXXXXXXXeWyiacaN26GcME >>>>>>>>>>>> 2016-11-08 23:11:42.883853 7f026b94f700 15 compare=0 >>>>>>>>>>>> 2016-11-08 23:11:42.883858 7f026b94f700 2 req 2:0.006739:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:normalizing buckets and tenants >>>>>>>>>>>> 2016-11-08 23:11:42.883864 7f026b94f700 10 s->object=<NULL> >>>>>>>>>>>> s->bucket=my-new-bucket-31337 >>>>>>>>>>>> 2016-11-08 23:11:42.883869 7f026b94f700 2 req 2:0.006750:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:init permissions >>>>>>>>>>>> 2016-11-08 23:11:42.883872 7f026b94f700 2 req 2:0.006753:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:recalculating target >>>>>>>>>>>> 2016-11-08 23:11:42.883875 7f026b94f700 2 req 2:0.006756:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:reading permissions >>>>>>>>>>>> 2016-11-08 23:11:42.883879 7f026b94f700 2 req 2:0.006760:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:init op >>>>>>>>>>>> 2016-11-08 23:11:42.883884 7f026b94f700 2 req 2:0.006765:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:verifying op mask >>>>>>>>>>>> 2016-11-08 23:11:42.883886 7f026b94f700 20 required_mask= 2 user.op_mask=7 >>>>>>>>>>>> 2016-11-08 23:11:42.883887 7f026b94f700 2 req 2:0.006769:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:verifying op permissions >>>>>>>>>>>> 2016-11-08 23:11:42.883939 7f026b94f700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> --> 192.168.168.203:6840/36134 -- osd_op(client.160747848.0:1023 15.efcbc969 >>>>>>>>>>>> EDITED - client name.buckets [call user.list_buckets] snapc 0=[] >>>>>>>>>>>> ack+read+known_if_redirected e97148) v7 -- ?+0 0x7f02b70acb00 con >>>>>>>>>>>> 0x7f02cd43a900 >>>>>>>>>>>> 2016-11-08 23:11:42.885157 7f02bbfff700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> <== osd.25 192.168.168.203:6840/36134 37 ==== osd_op_reply(1023 EDITED - >>>>>>>>>>>> client name.buckets [call] v0'0 uv0 ack = -2 ((2) No such file or >>>>>>>>>>>> directory)) v7 ==== 161+0+0 (2433927993 0 0) 0x7f02b886d280 con >>>>>>>>>>>> 0x7f02cd43a900 >>>>>>>>>>>> 2016-11-08 23:11:42.885307 7f026b94f700 2 req 2:0.008187:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:verifying op params >>>>>>>>>>>> 2016-11-08 23:11:42.885338 7f026b94f700 2 req 2:0.008219:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:pre-executing >>>>>>>>>>>> 2016-11-08 23:11:42.885342 7f026b94f700 2 req 2:0.008223:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:executing >>>>>>>>>>>> 2016-11-08 23:11:42.885377 7f026b94f700 20 get_system_obj_state: >>>>>>>>>>>> rctx=0x7f026b94c6d0 obj=.rgw:my-new-bucket-31337 state=0x7f02b70cdfe8 >>>>>>>>>>>> s->prefetch_data=0 >>>>>>>>>>>> 2016-11-08 23:11:42.885390 7f026b94f700 10 cache get: >>>>>>>>>>>> name=.rgw+my-new-bucket-31337 : miss >>>>>>>>>>>> 2016-11-08 23:11:42.885444 7f026b94f700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> --> 192.168.168.202:6821/14483 -- osd_op(client.160747848.0:1024 13.501be1c3 >>>>>>>>>>>> my-new-bucket-31337 [call version.read,getxattrs,stat] snapc 0=[] >>>>>>>>>>>> ack+read+known_if_redirected e97148) v7 -- ?+0 0x7f02b70ace80 con >>>>>>>>>>>> 0x7f02cd43b980 >>>>>>>>>>>> 2016-11-08 23:11:42.886940 7f02bfefe700 1 -- 192.168.168.201:0/3703211654 >>>>>>>>>>>> <== osd.11 192.168.168.202:6821/14483 11 ==== osd_op_reply(1024 >>>>>>>>>>>> my-new-bucket-31337 [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such >>>>>>>>>>>> file or directory)) v7 ==== 223+0+0 (4048274385 0 0) 0x7f02bf480280 con >>>>>>>>>>>> 0x7f02cd43b980 >>>>>>>>>>>> 2016-11-08 23:11:42.887120 7f026b94f700 10 cache put: >>>>>>>>>>>> name=.rgw+my-new-bucket-31337 info.flags=0 >>>>>>>>>>>> 2016-11-08 23:11:42.887135 7f026b94f700 10 adding .rgw+my-new-bucket-31337 >>>>>>>>>>>> to cache LRU end >>>>>>>>>>>> 2016-11-08 23:11:42.887144 7f026b94f700 0 rest connection is invalid >>>>>>>>>>> >>>>>>>>>>> Did you configure multi-site? Seems to me that it tries to connect to >>>>>>>>>>> a master zone, so if you have a single zonegroup, single zone >>>>>>>>>>> configuration you need to make sure that the zone (and zonegroup) are >>>>>>>>>>> set to be the master. >>>>>>>>>>> >>>>>>>>>>> Yehuda >>>>>>>>>>> >>>>>>>>>>>> 2016-11-08 23:11:42.887149 7f026b94f700 2 req 2:0.010030:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:completing >>>>>>>>>>>> 2016-11-08 23:11:42.887217 7f026b94f700 2 req 2:0.010098:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:op status=-22 >>>>>>>>>>>> 2016-11-08 23:11:42.887221 7f026b94f700 2 req 2:0.010102:s3:PUT >>>>>>>>>>>> /my-new-bucket-31337/:create_bucket:http status=400 >>>>>>>>>>>> 2016-11-08 23:11:42.887244 7f026b94f700 1 ====== req done >>>>>>>>>>>> req=0x7f02ba07b0e0 op status=-22 http_status=400 ====== >>>>>>>>>>>> 2016-11-08 23:11:42.887254 7f026b94f700 20 process_request() returned -22 >>>>>>>>>>>> >>>>>>>>>>>> ############################### >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Apache proxy logs show the following: >>>>>>>>>>>> >>>>>>>>>>>> 192.168.169.91 - - [08/Nov/2016:23:11:42 +0000] "PUT /my-new-bucket-31337/ >>>>>>>>>>>> HTTP/1.1" 400 4379 "-" "Boto/2.38.0 Python/2.7.12 >>>>>>>>>>>> Linux/4.8.4-040804-generic" >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The existing buckets work perfectly well, i can list and put objects. It's >>>>>>>>>>>> the creation of new buckets that i am having issues with. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Could someone please help me to figure out what the issue is? >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> Andrei >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list >>>>>>>>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> >>>>> >>>>> >>>>> -- >>>>> Yoann Moulin >>>>> EPFL IC-IT >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com