Ncolasc,
You said: "Just ran a fresh install of version Emperor on an empty cluster, and I am left clueless, trying to troubleshoot cephx. After ceph-deploy created the keys, I used ceph-authtool to generate the client.admin keyring and the monitor keyring, as indicated in the doc. The configuration is really out-of-the-box: 3 monitors, each with the keyring in /var/lib/ceph/mon/ceph-???/keyring, all keyrings have umask 644 and are owned by ceph."
You said: "Just ran a fresh install of version Emperor on an empty cluster, and I am left clueless, trying to troubleshoot cephx. After ceph-deploy created the keys, I used ceph-authtool to generate the client.admin keyring and the monitor keyring, as indicated in the doc. The configuration is really out-of-the-box: 3 monitors, each with the keyring in /var/lib/ceph/mon/ceph-???/keyring, all keyrings have umask 644 and are owned by ceph."
The ceph-deploy utility already generates the monitor keyring and the ceph.client.admin.keyring for you. I have a wip-doc branch for a manual deployment procedure without ceph-deploy, which I don't recommend for a first time user. However, it does detail what is going on.
Referring to steps 8-11, you'll notice that the manual process involves creating the monitor secret on step 8. Then, we generate a ceph.client.admin.keyring on step 9. See what happens on step 10 and 11? I add the ceph.client.admin.keyring contents to the monitor secret keyring; then, I feed that to step 11 for creating the monmap. Since ceph-deploy already creates the ceph.client.admin.keyring for you and populates it in the monmap, the fact that you are creating one after ceph-deploy has done this for you probably implies that you have overwritten the ceph.client.admin.keyring that was generated by ceph-deploy. The one you generated probably isn't in your monmap, so you are passing it the wrong key.
If you had the right key or if you turn off cephx, you could execute "ceph auth list" to see the client.admin key contents. It's likely different from what you have in your ceph.client.admin.keyring file.
ceph-deploy new generates a mon key. Then you deploy one or more monitors. Then, you use ceph-deploy gatherkeys. At that point, you should have a ceph.client.admin.keyring in the local/current directory after you executed ceph-deploy gatherkeys.
Let me know if this helps.
On Fri, Dec 6, 2013 at 7:59 AM, nicolasc <nicolas.canceill@xxxxxxxxxxx> wrote:
Hi Dan,
Thank you for the advice and indications. We have the exact same configuration, except I am only enabling "auth cluster", and I am using "ceph.client.admin.keyring" instead of simply "keyring".
Both locations "/etc/ceph/ceph.client.admin.keyring" and "/etc/ceph/keyring" are presented as default values for the "keyring" configuration setting. I will try "/etc/ceph/keyring", but I doubt this changes much.
I am curious of whether your setup still works if you remove the "keyring = /etc/ceph/keyring" setting (you are also using a default location, so you could remove that line safely, right?).
Thank you very much for answering. Best regards,
Nicolas Canceill
Scalable Storage Systems
SURFsara (Amsterdam, NL)
On 12/06/2013 11:42 AM, Dan Van Der Ster wrote:
Hi,
All of our clusters have this in ceph.conf:
[global]
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
keyring = /etc/ceph/keyring
and the client.admin secret in /etc/ceph/keyring:
# cat /etc/ceph/keyring
[client.admin]
key = ...
With that you should be able to do "ceph health" without passing —id or —keyring args. (this is with dumpling, not emperor, but I guess it didn’t change.)
If it still doesn’t work, check the capabilities that client.admin has (with ceph auth list). Should be
caps: [mds] allow
caps: [mon] allow *
caps: [osd] allow *
Cheers, Dan
On 06 Dec 2013, at 11:06, nicolasc <nicolas.canceill@xxxxxxxxxxx> wrote:
Hi every one,
I did not get any answer to my basic cephx question last week, so let me ask it one more time here, before I completely give up on Ceph and move on.
So, my issue is:
When all authentication settings are "none":
* The cluster works fine
* The file "/etc/ceph/ceph.client.admin.keyring " exists
Then I set "auth_cluster_required" to "cephx". When I try to connect to the cluster, it detects "client.admin" and denies access with "operation not supported", even for commands like "ceph health".
Finally, after I explicitly set the "keyring" parameter in the config (to the default value, because the keyring file was already in the default location), the cluster works fine again. So the behavior changes when I add those 2 default lines to the config:
[client.admin]
keyring = /etc/ceph/ceph.client.admin.keyring
From the ceph.com documentation [1], about this "keyring" parameter:
Description: The path to the keyring file.
Type: String
Required: No
Default: /etc/ceph/$cluster.$name.keyring
... so, I need help:
* maybe this is a real bug? (was it already reported ?)
* maybe I am deeply stupid, and I don't understand what "required" and "default" means? (can anyone send me a good dictionary ?)
* maybe obi-wan kenobi?
Thanks to anyone who will respond anything (at that point, even a three-letter e-mail reading "ACK" would make me feel better). Best wishes for the future of Ceph, and best regards.
Nicolas Canceill
Scalable Storage Systems
SURFsara (Amsterdam, NL)
[1] http://ceph.com/docs/master/rados/configuration/auth-config-ref/#keys
On 11/29/2013 03:09 PM, nicolasc wrote:
An update on this issue:_______________________________________________
Explicitly setting the "keyring" parameter to its default value, in the client section, like this:
[client.admin]
keyring = /etc/ceph/ceph.client.admin.keyring
solves the problem in the particular case when ONLY "auth_cluster_required" is set to "cephx", and the two remaining auth parameters are set to "none".
The documentation clearly states that "/etc/ceph/ceph.client.admin.keyring" is the default value of the "keyring" setting [1], so this looks like a bug. Should I report it on the tracker? (BTW, all of this is on v0.72.1.)
Also, does anyone have any idea about why this is not enough to enable the "auth_service_required" setting? That one still gives me the error:
client.admin authentication error (95) Operation not supported
Best regards,
Nicolas Canceill
Scalable Storage Systems
SURFsara (Amsterdam, NL)
[1] http://ceph.com/docs/master/rados/configuration/auth-config-ref/#keys
On 11/29/2013 10:22 AM, nicolasc wrote:
Hello every one,
Just ran a fresh install of version Emperor on an empty cluster, and I am left clueless, trying to troubleshoot cephx. After ceph-deploy created the keys, I used ceph-authtool to generate the client.admin keyring and the monitor keyring, as indicated in the doc. The configuration is really out-of-the-box: 3 monitors, each with the keyring in /var/lib/ceph/mon/ceph-???/keyring, all keyrings have umask 644 and are owned by ceph.
However, no matter which combination of "auth_cluster_", "auth_service_", or "auth_client_required", is set to cephx; no matter either the keyring options like "-k" and "--id" on the command line. Authentication fails every time with:
client.admin authentication error (95) Operation not supported
Error connecting to cluster: Error
A big thanks to any one who gives me a hint about what it means. (This message carries so little information, I feel it could be simply replaced by the "!" character.) I have looked in every ceph and system log file, nothing more.
Best regards,
Nicolas Canceill
Scalable Storage Systems
SURFsara (Amsterdam, NL)
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
John Wilkins
Senior Technical Writer
Intank
john.wilkins@xxxxxxxxxxx
(415) 425-9599
http://inktank.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
