Hello again Mark, Thanks again for pointing this out. After some discussion, we would like to keep the 'client.rgw.*' naming convention, and we do need to update ceph-deploy to make sure people do not shoot themselves in the foot with this. It will still be totally fine to have custom names, but more along the lines of client.rgw.foo, client.rgw.bar, etc. I've filed http://tracker.ceph.com/issues/11561 to track this. - Travis On Wed, Apr 22, 2015 at 9:24 AM, Travis Rhoden <trhoden@xxxxxxxxx> wrote: > Hi Mark, > > Thanks for the detective work! I'll confirm whether the constraint on > the client name is really intended -- I believe that it is. Once > confirmed, I completely agree that ceph-deploy should not allow you to > create rgw daemons with names that we know Ceph is going to reject. > > - Travis > > On Wed, Apr 22, 2015 at 4:42 AM, Mark Kirkwood > <mark.kirkwood@xxxxxxxxxxxxxxx> wrote: >> On 13/04/15 15:33, Mark Kirkwood wrote: >>> >>> Hi, >>> >>> I've been experimenting with the new rgw creation in ceph-deploy, using >>> version 1.5.23 together with ceph 0.94 (-948-gd77de49). >>> >>> If simply run it without any args, then it works fine. e.g: >>> >>> >>> $ ceph-deploy rgw create ceph1 >>> >>> >>> However if I try to set a name for the daemon: >>> >>> >>> $ ceph-deploy rgw create ceph1:radosgw.ceph1 >>> >>> >>> then it fails to create and start a radosgw daemon at all. Poking >>> through the logs I see: >>> >>> $ tail ceph-client.radosgw.ceph1.log >>> 2015-04-13 14:13:20.513860 7f7e36c04840 0 librados: >>> client.radosgw.ceph1 authentication error (1) Operation not permitted >>> >>> >>> Not terribly informative. Looking at the audit log: >>> >>> $ tail ceph.audit.log >>> 015-04-13 15:10:37.880085 mon.0 192.168.122.21:6789/0 147 : audit [INF] >>> from='client.? 192.168.122.22:0/1004669' entity='client.bootstrap-rgw' >>> cmd=[{"prefix": "auth get-or-create", "entity": "client.radosgw.ceph2", >>> "caps": ["osd", "allow rwx", "mon", "allow rw"]}]: access denied >>> >>> >>> Looking at what ceph-deploy is doing I see that the failing command is: >>> >>> >>> $ sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring >>> /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create >>> client.radosgw.ceph2 osd 'allow rwx' mon 'allow rw' -o >>> /var/lib/ceph/radosgw/ceph-radosgw.ceph2/keyring >>> >>> >>> This is failing whereas the equivalent command with radosgw replaced by >>> rgw works. Looking at what privileges client.bootstrap-rgw has I see: >>> >>> $ sudo ceph auth get client.bootstrap-rgw >>> [client.bootstrap-rgw] >>> key = XXXXXXXXXXXXXXXXXX== >>> caps mon = "allow profile bootstrap-rgw" >>> >>> Ah, so need to look at how the profile bootstrap-rgw is setup: >>> >>> $ more src/mon/MonCap.cc >>> ... >>> if (profile == "bootstrap-rgw") { >>> profile_grants.push_back(MonCapGrant("mon", MON_CAP_R)); // read >>> monmap >>> profile_grants.push_back(MonCapGrant("osd", MON_CAP_R)); // read >>> osdmap >>> profile_grants.push_back(MonCapGrant("mon getmap")); >>> profile_grants.push_back(MonCapGrant("auth get-or-create")); // >>> FIXME: this can expose other mds keys >>> profile_grants.back().command_args["entity"] = StringConstraint("", >>> "client.rgw."); <====== >>> profile_grants.back().command_args["caps_mon"] = >>> StringConstraint("allow rw", ""); >>> profile_grants.back().command_args["caps_osd"] = >>> StringConstraint("allow rwx", ""); >>> } >>> ... >>> >>> I see - the ["entity"] check requires the name to match "client.rgw.". >>> Double checking back on the deployed host (ceph1), I see that: >>> >>> >>> $ ceph --cluster ceph --name client.bootstrap-rgw --keyring >>> /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create >>> client.rgw.ceph3 osd 'allow rwx' mon 'allow rw' >>> [client.rgw.ceph3] >>> key = AQCrNytVp9bqFBAAgTJVQDVQFJqiSkFqD68XsQ== >>> $ ceph --cluster ceph --name client.bootstrap-rgw --keyring >>> /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create >>> client.radosgw.ceph3 osd 'allow rwx' mon 'allow rw' >>> Error EACCES: access denied >>> >>> Ok so that is definitely the problem. >>> >>> Is this restriction intended? If so we really should make ceph-deploy >>> aware of this...as it is a bit of a foot gun (and not as easy as I made >>> it seem above to figure out *why* the deploy didn't work). >>> >>> Cheers >>> >>> Mark >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> ping? >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
