On 2/29/24 16:52, Scott Mayhew wrote: > On Thu, 29 Feb 2024, Orion Poplawski wrote: > >> We are starting to add some EL9 clients into the mix on our network. Autofs >> mounts are working fine when initiated by a regular user, but they are failing >> when initiated by root. This works fine from our EL8 clients. >> >> Client: >> kernel 5.14.0-362.18.1.el9_3.x86_64 >> nfs-utils-2.5.4-20.el9.x86_64 >> >> Keytab name: FILE:/etc/krb5.keytab >> KVNO Principal >> ---- -------------------------------------------------------------------------- >> 1 host/client.fqdn@xxxxxxxx (aes256-cts-hmac-sha1-96) >> 1 host/client.fqdn@xxxxxxxx (aes128-cts-hmac-sha1-96) >> >> Server: >> kernel 4.18.0-513.18.1.el8_9.x86_64 >> nfs-utils-2.3.3-59.el8.x86_64 >> >> Keytab name: FILE:/etc/krb5.keytab >> KVNO Principal >> ---- -------------------------------------------------------------------------- >> 1 host/server.fqdn@xxxxxxxx (aes256-cts-hmac-sha1-96) >> 1 host/server.fqdn@xxxxxxxx (aes128-cts-hmac-sha1-96) >> 1 nfs/server.fqdn@xxxxxxxx (aes256-cts-hmac-sha1-96) >> 1 nfs/server.fqdn@xxxxxxxx (aes128-cts-hmac-sha1-96) >> >> Client rpc.gssd: >> >> rpc.gssd[778]: >> handle_gssd_upcall(0x7f15a0299840): 'mech=krb5 >> uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt3) >> rpc.gssd[778]: start_upcall_thread(0x7f15a0299840): created thread id >> 0x7f159f1fe640 >> rpc.gssd[778]: krb5_use_machine_creds(0x7f159f1fe640): uid 0 tgtname (null) >> rpc.gssd[778]: No key table entry found for client$@NWRA.COM while getting >> keytab entry for 'client$@NWRA.COM' >> rpc.gssd[778]: No key table entry found for CLIENT$@NWRA.COM while getting >> keytab entry for 'SRV-MRY01$@NWRA.COM' >> rpc.gssd[778]: No key table entry found for root/client.fqdn@xxxxxxxx while >> getting keytab entry for 'root/client.fqdn@xxxxxxxx' >> rpc.gssd[778]: No key table entry found for nfs/client.fqdn@xxxxxxxx while >> getting keytab entry for 'nfs/client.fqdn@xxxxxxxx' >> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry >> for 'host/client.fqdn@xxxxxxxx' >> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC >> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 >> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC >> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 >> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for >> server server.fqdn >> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with >> server nfs@xxxxxxxxxxx >> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for >> server nfs@xxxxxxxxxxx >> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache >> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn >> rpc.gssd[778]: WARNING: Machine cache prematurely expired or corrupted trying >> to recreate cache for server server.fqdn >> rpc.gssd[778]: No key table entry found for client$@NWRA.COM while getting >> keytab entry for 'client$@NWRA.COM' >> rpc.gssd[778]: No key table entry found for CLIENT$@NWRA.COM while getting >> keytab entry for 'SRV-MRY01$@NWRA.COM' >> rpc.gssd[778]: No key table entry found for root/client.fqdn@xxxxxxxx while >> getting keytab entry for 'root/client.fqdn@xxxxxxxx' >> rpc.gssd[778]: No key table entry found for nfs/client.fqdn@xxxxxxxx while >> getting keytab entry for 'nfs/client.fqdn@xxxxxxxx' >> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry >> for 'host/client.fqdn@xxxxxxxx' >> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC >> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 >> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC >> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 >> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for >> server server.fqdn >> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with >> server nfs@xxxxxxxxxxx >> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for >> server nfs@xxxxxxxxxxx >> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache >> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn >> rpc.gssd[778]: ERROR: Failed to create machine krb5 context with any >> credentials cache for server server.fqdn >> rpc.gssd[778]: do_error_downcall(0x7f159f1fe640): uid 0 err -13 >> >> mount.nfs4: mount(2): Permission denied >> mount.nfs4: access denied by server while mounting >> >> I don't seem to be getting any useful information from rpc.gssd on the server. >> >> Please include me in replies. > > I'm pretty sure this is the same issue that would be addressed by the > patches that I sent to the list yesterday would address: > https://lore.kernel.org/linux-nfs/20240228222253.1080880-1-smayhew@xxxxxxxxxx/T/#t > > There's a couple things you could check. > > 1. On the NFS server, increase the rpc debug logging just a little: > # rpcdebug -m rpc -s auth > > and then after a failure, look for lines like this in the journal: > Feb 29 18:14:44 rhel8.smayhew.test kernel: svc: svc_authenticate (6) > Feb 29 18:14:44 rhel8.smayhew.test kernel: gss_kerberos_mech: unsupported krb5 enctype 20 > Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_import_sec_context_kerberos: returning -22 > Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_delete_sec_context deleting 0000000090f401ca > > 2. On the NFS client, increase the rpc-verbosity in the gssd stanza in > /etc/nfs.conf (I have mine set to 3 but I think 1 would suffice) and > then 'systemctl restart rpc-gssd'. > > then after a failure, look for a line like this in the journal: > Feb 29 18:14:44 rhel9.smayhew.test rpc.gssd[1700]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer > > If you see both of those, then it's almost certainly the same issue. Thank you! Yes, I see both of these messages. I guess this also explains why I was able to mount from an EL7 server. > A quick solution would be to do this on your NFS server: > > # echo "mac@Kerberos = -HMAC-SHA2-*" >/usr/share/crypto-policies/policies/modules/NFS.pmod > # update-crypto-policies --set DEFAULT:NFS > # systemctl restart gssproxy > > but note that would be turning off the SHA2 enctypes for everything > krb5-related, not just NFS. Note you can always revert that later > simply by: > > # update-crypto-policies --set DEFAULT > # systemctl restart gssproxy > > Or, you could test the patches I sent to the list yesterday (this would > be on the client, not the server). The problem is those patches don't > apply cleanly to the current version of nfs-utils shipped in EL9. At a > quick glance, it looks like nfs-utils would need: > > 49567e7d configure: check for rpc_gss_seccreate > 15cd5666 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials > 2bfb59c6 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials > 3abf6b52 gssd: switch to using rpc_gss_seccreate() > f05af7d9 gssd: revert commit 513630d720bd > 20c07979 gssd: revert commit a5f3b7ccb01c > 14ee4878 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials > 4b272471 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials > 75b04a9b gssd: fix handling DNS lookup failure > f066f87b gssd: enable forcing cred renewal using the keytab > > and you'd also need to patch libtirpc to include: > > 22b1c0c gssapi: fix rpc_gss_seccreate passed in cred I'll take a look at these more tomorrow. Since I see you have a @redhat.com address, I am hoping that these fixes will flow down to EL9 sooner rather than later. Is there an open issue I can subscribe to to follow along? -- Orion Poplawski he/him/his - surely the least important thing about me Manager of IT Systems 720-772-5637 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion@xxxxxxxx Boulder, CO 80301 https://www.nwra.com/
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
