On Friday, October 13, 2023 10:46:24 AM EDT Torkil Svensgaard wrote:
> On 13-10-2023 16:40, Torkil Svensgaard wrote:
> > On 13-10-2023 14:00, John Mulligan wrote:
> >> On Friday, October 13, 2023 6:11:18 AM EDT Torkil Svensgaard wrote:
> >>> Hi
> >>>
> >>> We have kerberos working with bare metal kernel NFS exporting RBDs. I
> >>> can see in the ceph documentation[1] that nfs-ganesha should work with
> >>> kerberos but I'm having little luck getting it to work.
> >>
> >> Could you clarify how you are deploying the ganesha instances? I
> >> think you
> >> may be asking about cephadm deployed containers but it is not clear to
> >> me.
> >>
> >> There are a couple of viable ways to deploy nfs-ganesha today:
> >> manually/"bare-
> >> metal", cephadm, and rook.
> >
> > Of course, my bad. Ceph version 17.2.6-100.el9cp, nfs-ganesha deployed
> > with cephadm as pr the documentation I linked:
> >
> > "
> > ceph nfs export create cephfs --cluster-id cephfs_noHA --pseudo-path
> > /testkrb5p_noHA --fsname cephfs_ssd --path=/test --sectype krb5p
> > --sectype sys
> > "
> >
> > "
> > [ceph: root@lazy /]# ceph nfs export info cephfs_noHA /testkrb5p_noHA
> > {
> >
> > "export_id": 1,
> > "path": "/test",
> > "cluster_id": "cephfs_noHA",
> > "pseudo": "/testkrb5p_noHA",
> > "access_type": "RW",
> > "squash": "none",
> > "security_label": true,
> > "protocols": [
> > 4
> > ],
> > "transports": [
> > "TCP"
> > ],
> > "fsal": {
> > "name": "CEPH",
> > "user_id": "nfs.cephfs_noHA.1",
> > "fs_name": "cephfs_ssd"
> > },
> > "clients": [],
> > "sectype": [
> > "krb5p",
> > "sys"
> > ]
> >
> > }
> > "
> >
> > Works with sys, not with krb5p.
> >
> > Thanks.
> >
> > Mvh.
> >
> > Torkil
> >
> >>> This bit from the container log seems to suggest that some plumbing is
> >>> missing?
> >>>
> >>> "
> >>> 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
> >>> ganesha.nfsd-2[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT
> >>>
> >>> :Callback creds directory (/var/run/ganesha) already exists
> >>>
> >>> 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
> >>> ganesha.nfsd-2[main] find_keytab_entry :NFS CB :WARN :Configuration file
> >>> does not specify default realm while getting default realm name
> >>> 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
> >>> ganesha.nfsd-2[main] gssd_refresh_krb5_machine_credential :NFS CB :CRIT
> >>>
> >>> :ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry
> >>>
> >>> found in keytab /etc/krb5.keytab for connection with host localhost
> >>> 13/10/2023 08:09:12 : epoch 6528fb25 : ceph-flash1 :
> >>> ganesha.nfsd-2[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN
> >>>
> >>> :gssd_refresh_krb5_machine_credential failed (-1765328160:0)
> >>>
> >>> "
> >>>
> >>> Thoughts?
> >>>
> >>> Mvh.
> >>>
> >>> Torkil
> >>>
> >>> [1] https://docs.ceph.com/en/quincy/mgr/nfs/#create-cephfs-export
> >>
> >> Based on your mention of "container log" above I assume this is on either
> >> cephadm or rook. The rook team has been actively working on adding
> >> kerberized
> >> nfs support and we added the `sectype` option for that work. Currently,
> >> cephadm doesn't have support for kerberos because it lacks the server
> >> side
> >> components needed to connect to krb5/ldap.
> >>
> >> I would like to see this support eventually come to cephadm but it's
> >> not there
> >> today IMO.
>
> Missed this part first time around. Interesting.
>
No worries. I know not everyone is used to inline/bottom posting. :-)
> >> A manual deployment of nfs-ganesha ought to also be able to make use
> >> of this
> >> option. Ultimately, this generates ganesha config blocks and is mostly
> >> agnostic
> >> of the cluster/deployment method, but I have not tried it out myself.
>
> The config block posted above seems fine though, or? So it wouldn't
> matter if deployed manually, the bits that are missing are how to get
> kerberos inside the container? The container host has all the bits.
>
Right, even if the container host is set up for kerberos, the needed
properties dont' propagate to the container. In fact, we probably *don't* want
the host's krb5/ldap config in the container - I'd prefer to have each
container have it's own set up. On the mgr module side that would mean every
"cluster" could have a different set of kerberos and/or ldap properties. I'm
working on SMB support which I intend to have a similar workflow: you can define
"clusters" (aka "virtual servers") that have their own optional domain
membership. Maybe at some point further in the future we can add similar
behaviors to nfs. Hope that helps!
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
