Hi all,
I'm having a bit of fun and games getting the Rados Gateway going. I
*think* I've got a working Ceph storage cluster. I've tried following
the documentation, but how one tests that the gateway works is unclear.
My structure is as follows:
- Two storage nodes, bnedevsn0 and bnedevsn1 with 2 OSDs
each (3TB spinners) run ceph-osd instances.
- Two management nodes, bnedevmn0 and bnedevmn1 run ceph-mon, ceph-mds,
Keystone and other OpenStack management services. The two management
nodes have a dr:bd block device for the MySQL database, and listen on a
virtual IP with hostname 'bnedevcm' ("development cluster management")
The platform being used is Ubuntu 12.04 LTS AMD64 with Ceph release 0.61.
In order to establish a quorum, I've told it there's a bnedevmn2 which
is down. It establishes a 2:3 quorum, and is happy. I'm told the
cluster is in good health. In production we will have 3 nodes.
I've tried both with and without Keystone, but something is clearly
wrong in my setup of the Rados Gateway. I tried following the guide as
best I understood it, but still can't get it to work it seems.
In my ceph.conf, I have the following:
> [client.radosgw.gateway]
> host = bnedevmn0
> keyring = /etc/ceph/keyring.radosgw.gateway
> rgw socket path = /tmp/radosgw.sock
> log file = /var/log/ceph/radosgw.log
> #rgw keystone url = http://bnedevcm.vrt.int:35357
> #rgw keystone admin token = YFmhzg0Kt5RHvDGx
> #rgw keystone accepted roles = admin, Member, swiftoperator
> #rgw keystone token cache size = 500
> #rgw keystone revocation interval = 600
> rgw enable usage log = true
> nss db path = /var/lib/ceph/nss
> rgw print continue = false
Shown above, are also the parameters I've tried for Keystone, commented
out here to take Keystone out of the equation.
If I try creating a user for Swift access... I can create the user itself:
> root@bnedevmn0:/var/lib/ceph/radosgw/ceph-radosgw.gateway# radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@xxxxxxxxxxx
> 2013-05-27 11:40:09.330457 7f7fb5832700 0 -- :/1019040 >> 10.87.168.252:6789/0 pipe(0x15d8400 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
> { "user_id": "johndoe",
> "display_name": "John Doe",
> "email": "john@xxxxxxxxxxx",
> "suspended": 0,
> "max_buckets": 1000,
> "auid": 0,
> "subusers": [],
> "keys": [
> { "user": "johndoe",
> "access_key": "P4OJFBD7X99YZKEXVQV2",
> "secret_key": "Qzn3PmL85wkQScoYJB1ksLF4DJi7owJNcy9Hvntv"}],
> "swift_keys": [],
> "caps": []}
So far so good, I create the subuser:
> root@bnedevmn0:/var/lib/ceph/radosgw/ceph-radosgw.gateway# radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift --access=full
> 2013-05-27 11:40:59.134460 7f1e092e4700 0 -- :/1019338 >> 10.87.168.252:6789/0 pipe(0x2b22400 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
> { "user_id": "johndoe",
> "display_name": "John Doe",
> "email": "john@xxxxxxxxxxx",
> "suspended": 0,
> "max_buckets": 1000,
> "auid": 0,
> "subusers": [
> { "id": "johndoe:swift",
> "permissions": "full-control"}],
> "keys": [
> { "user": "johndoe",
> "access_key": "P4OJFBD7X99YZKEXVQV2",
> "secret_key": "Qzn3PmL85wkQScoYJB1ksLF4DJi7owJNcy9Hvntv"}],
> "swift_keys": [],
> "caps": []}
Now I create a Swift key:
> root@bnedevmn0:/var/lib/ceph/radosgw/ceph-radosgw.gateway# radosgw-admin key create --subuser=johndoe:swift --key-type=swift
> 2013-05-27 11:41:25.550454 7f9bfa11e700 0 -- :/1019498 >> 10.87.168.252:6789/0 pipe(0x18b8400 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
> { "user_id": "johndoe",
> "display_name": "John Doe",
> "email": "john@xxxxxxxxxxx",
> "suspended": 0,
> "max_buckets": 1000,
> "auid": 0,
> "subusers": [
> { "id": "johndoe:swift",
> "permissions": "full-control"}],
> "keys": [
> { "user": "johndoe",
> "access_key": "P4OJFBD7X99YZKEXVQV2",
> "secret_key": "Qzn3PmL85wkQScoYJB1ksLF4DJi7owJNcy9Hvntv"}],
> "swift_keys": [
> { "user": "johndoe:swift",
> "secret_key": ""}],
> "caps": []}
... .Ooookay, so the key is the empty string?
> Auth version 1.0 requires ST_AUTH, ST_USER, and ST_KEY environment variables
> to be set or overridden with -A, -U, or -K.
>
> Auth version 2.0 requires OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, and
> OS_TENANT_NAME OS_TENANT_ID to be set or overridden with --os-auth-url,
> --os-username, --os-password, --os-tenant-name or os-tenant-id.
> root@bnedevmn0:~# swift -V 1.0 -A http://bnedevcm.vrt.int/auth/ -U johndoe:swift -K '' list
Nope.
If I uncomment the Keystone-related lines in ceph.conf... I don't get
very far trying to connect to Swift either:
> root@bnedevmn0:/var/lib/ceph/radosgw/ceph-radosgw.gateway# swift --verbose --debug --os-auth-url http://bnedevcm.vrt.int:35357/v2.0 --os-username username --os-password password --os-tenant-name tenant-name list
> REQ: curl -i http://bnedevcm.vrt.int:35357/v2.0/tokens -X POST -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient"
> DEBUG:keystoneclient.client:REQ: curl -i http://bnedevcm.vrt.int:35357/v2.0/tokens -X POST -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient"
> REQ BODY: {"auth": {"tenantName": "tenant-name", "passwordCredentials": {"username": "username", "password": "password"}}}
>
> DEBUG:keystoneclient.client:REQ BODY: {"auth": {"tenantName": "tenant-name", "passwordCredentials": {"username": "username", "password": "password"}}}
>
> INFO:urllib3.connectionpool:Starting new HTTP connection (1): bnedevcm.vrt.int
> DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 6230
> RESP: [200] {'date': 'Mon, 27 May 2013 01:47:27 GMT', 'content-type': 'application/json', 'content-length': '6230', 'vary': 'X-Auth-Token'}
> RESP BODY: {"access": {"token": {"issued_at": "2013-05-27T01:47:27.620430", "expires": "2013-05-28T01:47:27Z", "id": "MIIK...p6bo", "tenant": {"description": "VRT Network", "enabled": true, "id": "4fa251b2f2d24e72a69ba472ec77f06f", "name": "tenant-name"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:8774/v2/4fa251b2f2d24e72a69ba472ec77f06f";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:8774/v2/4fa251b2f2d24e72a69ba472ec77f06f";, "id": "75ee4381419a42d296d70577433a6d5b", "publicURL": "http://bnedevcm.vrt.int:8774/v2/4fa251b2f2d24e72a69ba472ec77f06f"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:9292";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:9292";, "id": "01c3a6c7f9a442c598cdde1891f94546", "publicURL": "http://bnedevcm.vrt.int:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int
:8776/v1/
4fa251b2f2d24e72a69ba472ec77f06f", "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:8776/v1/4fa251b2f2d24e72a69ba472ec77f06f";, "id": "58c34b6e16554447aa880277f0365978", "publicURL": "http://bnedevcm.vrt.int:8776/v1/4fa251b2f2d24e72a69ba472ec77f06f"}], "endpoints_links": [], "type": "volume", "name": "volume"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:8773/services/Admin";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:8773/services/Cloud";, "id": "31041d504c5143d6ac7103645f614229", "publicURL": "http://bnedevcm.vrt.int:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int/swift/v1";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int/swift/v1";, "id": "00d58c396d01479b887bb401298496aa", "publicURL": "http://bnedevcm.vrt.int/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:353
57/v2.0",
"region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:5000/v2.0";, "id": "087c1afebbf04bd19a644ffd5d6088d0", "publicURL": "http://bnedevcm.vrt.int:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "username", "roles_links": [], "id": "dc13dd9d013446b49ee3e564b990f5ec", "roles": [{"name": "admin"}, {"name": "_member_"}], "name": "username"}, "metadata": {"is_admin": 0, "roles": ["4c936ec7ec75426b98ef15cd6fd43eaf", "9fe2ff9ee4384b1894a90878d3e92bab"]}}}
>
> DEBUG:keystoneclient.client:RESP: [200] {'date': 'Mon, 27 May 2013 01:47:27 GMT', 'content-type': 'application/json', 'content-length': '6230', 'vary': 'X-Auth-Token'}
> RESP BODY: {"access": {"token": {"issued_at": "2013-05-27T01:47:27.620430", "expires": "2013-05-28T01:47:27Z", "id": "MIIK...p6bo", "tenant": {"description": "VRT Network", "enabled": true, "id": "4fa251b2f2d24e72a69ba472ec77f06f", "name": "tenant-name"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:8774/v2/4fa251b2f2d24e72a69ba472ec77f06f";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:8774/v2/4fa251b2f2d24e72a69ba472ec77f06f";, "id": "75ee4381419a42d296d70577433a6d5b", "publicURL": "http://bnedevcm.vrt.int:8774/v2/4fa251b2f2d24e72a69ba472ec77f06f"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:9292";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:9292";, "id": "01c3a6c7f9a442c598cdde1891f94546", "publicURL": "http://bnedevcm.vrt.int:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int
:8776/v1/
4fa251b2f2d24e72a69ba472ec77f06f", "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:8776/v1/4fa251b2f2d24e72a69ba472ec77f06f";, "id": "58c34b6e16554447aa880277f0365978", "publicURL": "http://bnedevcm.vrt.int:8776/v1/4fa251b2f2d24e72a69ba472ec77f06f"}], "endpoints_links": [], "type": "volume", "name": "volume"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:8773/services/Admin";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:8773/services/Cloud";, "id": "31041d504c5143d6ac7103645f614229", "publicURL": "http://bnedevcm.vrt.int:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "ec2"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int/swift/v1";, "region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int/swift/v1";, "id": "00d58c396d01479b887bb401298496aa", "publicURL": "http://bnedevcm.vrt.int/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://bnedevcm.vrt.int:353
57/v2.0",
"region": "bne-dev", "internalURL": "http://bnedevcm.vrt.int:5000/v2.0";, "id": "087c1afebbf04bd19a644ffd5d6088d0", "publicURL": "http://bnedevcm.vrt.int:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "username", "roles_links": [], "id": "dc13dd9d013446b49ee3e564b990f5ec", "roles": [{"name": "admin"}, {"name": "_member_"}], "name": "username"}, "metadata": {"is_admin": 0, "roles": ["4c936ec7ec75426b98ef15cd6fd43eaf", "9fe2ff9ee4384b1894a90878d3e92bab"]}}}
>
> DEBUG:swiftclient:RESP STATUS: 500
>
> DEBUG:swiftclient:RESP BODY: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>500 Internal Server Error</title>
> </head><body>
> <h1>Internal Server Error</h1>
> <p>The server encountered an internal error or
> misconfiguration and was unable to complete
> your request.</p>
> <p>Please contact the server administrator,
> {email.address} and inform them of the time the error occurred,
> and anything you might have done that may have
> caused the error.</p>
> <p>More information about this error may be available
> in the server error log.</p>
> </body></html>
In the logs, I see the following a lot:
> 2013-05-27 11:39:14.731355 7f78d7123780 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), proc
> ess radosgw, pid 18717
> 2013-05-27 11:39:14.739316 7f78d711f700 0 -- :/1018719 >> 10.87.168.253:6789/0 pipe(0x1e3b400 sd=11 :0 s=1 pgs
> =0 cs=0 l=1).fault
> 2013-05-27 11:39:15.739714 7f79e601b700 0 -- 10.87.168.254:0/1018695 >> 10.87.168.253:6789/0 pipe(0x7f79d4001d
> 60 sd=11 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-05-27 11:39:17.742484 7f78c9eef700 2 garbage collection: start
> 2013-05-27 11:39:17.744752 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.3
> 2013-05-27 11:39:17.746848 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.4
> 2013-05-27 11:39:17.747902 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.5
> 2013-05-27 11:39:17.749974 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.6
> 2013-05-27 11:39:17.751092 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.7
> 2013-05-27 11:39:17.752034 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.8
> 2013-05-27 11:39:17.753101 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.9
> 2013-05-27 11:39:17.754179 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.10
> 2013-05-27 11:39:17.755119 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.11
> 2013-05-27 11:39:17.756389 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.12
> 2013-05-27 11:39:17.757334 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.13
> 2013-05-27 11:39:17.758525 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.14
> 2013-05-27 11:39:17.759683 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.15
> 2013-05-27 11:39:17.760809 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.16
> 2013-05-27 11:39:17.761988 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.17
> 2013-05-27 11:39:17.763021 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.18
> 2013-05-27 11:39:17.764159 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.19
> 2013-05-27 11:39:17.765249 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.20
> 2013-05-27 11:39:17.766123 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.21
> 2013-05-27 11:39:17.767190 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.22
> 2013-05-27 11:39:17.768283 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.23
> 2013-05-27 11:39:17.769322 7f78c9eef700 0 RGWGC::process() failed to acquire lock on gc.24
and sometimes this too:
> 2013-05-27 11:39:44.829626 7f6a198f0700 0 RGWGC::process() failed to acquire lock on gc.21
> 2013-05-27 11:39:44.830784 7f6a198f0700 0 RGWGC::process() failed to acquire lock on gc.22
> 2013-05-27 11:39:44.831758 7f6a198f0700 -1 *** Caught signal (Segmentation fault) **
> in thread 7f6a198f0700
>
> ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
> 1: /usr/bin/radosgw() [0x4f19da]
> 2: (()+0xfcb0) [0x7f6a24db9cb0]
> 3: (()+0x1499d6) [0x7f6a238d59d6]
> 4: (std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x72) [0x7f6a240fc842]
> 5: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)+0x3c) [0x7f6a240fcf5c]
> 6: (RGWGC::process(int, int)+0x2c) [0x60106c]
> 7: (RGWGC::process()+0x5e) [0x601f3e]
> 8: (RGWGC::GCWorker::entry()+0x213) [0x602173]
> 9: (()+0x7e9a) [0x7f6a24db1e9a]
> 10: (clone()+0x6d) [0x7f6a2387fccd]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> --- begin dump of recent events ---
> -172> 2013-05-27 11:39:44.788653 7f6a26b24780 5 asok(0x2442660) register_command perfcounters_dump hook 0x2443a50
> -171> 2013-05-27 11:39:44.788684 7f6a26b24780 5 asok(0x2442660) register_command 1 hook 0x2443a50
> -170> 2013-05-27 11:39:44.788691 7f6a26b24780 5 asok(0x2442660) register_command perf dump hook 0x2443a50
> -169> 2013-05-27 11:39:44.788723 7f6a26b24780 5 asok(0x2442660) register_command perfcounters_schema hook 0x2443a50
> -168> 2013-05-27 11:39:44.788729 7f6a26b24780 5 asok(0x2442660) register_command 2 hook 0x2443a50
> -167> 2013-05-27 11:39:44.788733 7f6a26b24780 5 asok(0x2442660) register_command perf schema hook 0x2443a50
> -166> 2013-05-27 11:39:44.788740 7f6a26b24780 5 asok(0x2442660) register_command config show hook 0x2443a50
> -165> 2013-05-27 11:39:44.788745 7f6a26b24780 5 asok(0x2442660) register_command config set hook 0x2443a50
> -164> 2013-05-27 11:39:44.788750 7f6a26b24780 5 asok(0x2442660) register_command log flush hook 0x2443a50
> -163> 2013-05-27 11:39:44.788754 7f6a26b24780 5 asok(0x2442660) register_command log dump hook 0x2443a50
> -162> 2013-05-27 11:39:44.788758 7f6a26b24780 5 asok(0x2442660) register_command log reopen hook 0x2443a50
> -161> 2013-05-27 11:39:44.791129 7f6a26b24780 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process radosgw, pid 18902
> -160> 2013-05-27 11:39:44.791833 7f6a26b24780 1 finished global_init_daemonize
> -159> 2013-05-27 11:39:44.797687 7f6a26b24780 10 monclient(hunting): build_initial_monmap
> -158> 2013-05-27 11:39:44.797843 7f6a26b24780 1 librados: starting msgr at :/0
> -157> 2013-05-27 11:39:44.797858 7f6a26b24780 1 librados: starting objecter
> -156> 2013-05-27 11:39:44.797905 7f6a26b24780 1 -- :/0 messenger.start
> -155> 2013-05-27 11:39:44.797937 7f6a26b24780 1 librados: setting wanted keys
> -154> 2013-05-27 11:39:44.797943 7f6a26b24780 1 librados: calling monclient init
> -153> 2013-05-27 11:39:44.797948 7f6a26b24780 10 monclient(hunting): init
> -152> 2013-05-27 11:39:44.797966 7f6a26b24780 5 adding auth protocol: cephx
> -151> 2013-05-27 11:39:44.797975 7f6a26b24780 10 monclient(hunting): auth_supported 2 method cephx
> -150> 2013-05-27 11:39:44.798318 7f6a26b24780 2 auth: KeyRing::load: loaded key file /etc/ceph/keyring.radosgw.gateway
> -149> 2013-05-27 11:39:44.798391 7f6a26b24780 10 monclient(hunting): _reopen_session
> -148> 2013-05-27 11:39:44.798439 7f6a26b24780 10 monclient(hunting): _pick_new_mon picked mon.noname-a con 0x249d660 addr 10.87.168.254:6789/0
> -147> 2013-05-27 11:39:44.798467 7f6a26b24780 10 monclient(hunting): _send_mon_message to mon.noname-a at 10.87.168.254:6789/0
The following packages are installed:
> root@bnedevmn0:~# dpkg -l radosgw* ceph*
> Desired=Unknown/Install/Remove/Purge/Hold
> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name Version Description
> +++-=====================-=====================-==========================================================
> ii ceph 0.61.2-1precise distributed storage and file system
> un ceph-client-tools <none> (no description available)
> ii ceph-common 0.61.2-1precise common utilities to mount and interact with a ceph storage
> ii ceph-fs-common 0.61.2-1precise common utilities to mount and interact with a ceph file sy
> un ceph-fuse <none> (no description available)
> ii ceph-mds 0.61.2-1precise metadata server for the ceph distributed file system
> ii radosgw 0.61.2-1precise REST gateway for RADOS distributed object store
> ii radosgw-dbg 0.61.2-1precise debugging symbols for radosgw
Is there something I missed in setting this up?
--
## -,-''''-. ###### Stuart Longland, Contractor
##. : ## : ## 38b Douglas Street
## # ## -'` .#' Milton, QLD, 4064
'#' *' '-. *' http://www.vrt.com.au
S Y S T E M S T: +61 7 3535 9619 F: +61 7 3535 9699
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
