On Thu, 29 Feb 2024, Orion Poplawski wrote: > We are starting to add some EL9 clients into the mix on our network. Autofs > mounts are working fine when initiated by a regular user, but they are failing > when initiated by root. This works fine from our EL8 clients. > > Client: > kernel 5.14.0-362.18.1.el9_3.x86_64 > nfs-utils-2.5.4-20.el9.x86_64 > > Keytab name: FILE:/etc/krb5.keytab > KVNO Principal > ---- -------------------------------------------------------------------------- > 1 host/client.fqdn@xxxxxxxx (aes256-cts-hmac-sha1-96) > 1 host/client.fqdn@xxxxxxxx (aes128-cts-hmac-sha1-96) > > Server: > kernel 4.18.0-513.18.1.el8_9.x86_64 > nfs-utils-2.3.3-59.el8.x86_64 > > Keytab name: FILE:/etc/krb5.keytab > KVNO Principal > ---- -------------------------------------------------------------------------- > 1 host/server.fqdn@xxxxxxxx (aes256-cts-hmac-sha1-96) > 1 host/server.fqdn@xxxxxxxx (aes128-cts-hmac-sha1-96) > 1 nfs/server.fqdn@xxxxxxxx (aes256-cts-hmac-sha1-96) > 1 nfs/server.fqdn@xxxxxxxx (aes128-cts-hmac-sha1-96) > > Client rpc.gssd: > > rpc.gssd[778]: > handle_gssd_upcall(0x7f15a0299840): 'mech=krb5 > uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt3) > rpc.gssd[778]: start_upcall_thread(0x7f15a0299840): created thread id > 0x7f159f1fe640 > rpc.gssd[778]: krb5_use_machine_creds(0x7f159f1fe640): uid 0 tgtname (null) > rpc.gssd[778]: No key table entry found for client$@NWRA.COM while getting > keytab entry for 'client$@NWRA.COM' > rpc.gssd[778]: No key table entry found for CLIENT$@NWRA.COM while getting > keytab entry for 'SRV-MRY01$@NWRA.COM' > rpc.gssd[778]: No key table entry found for root/client.fqdn@xxxxxxxx while > getting keytab entry for 'root/client.fqdn@xxxxxxxx' > rpc.gssd[778]: No key table entry found for nfs/client.fqdn@xxxxxxxx while > getting keytab entry for 'nfs/client.fqdn@xxxxxxxx' > rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry > for 'host/client.fqdn@xxxxxxxx' > rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC > 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 > rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC > 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 > rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for > server server.fqdn > rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with > server nfs@xxxxxxxxxxx > rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for > server nfs@xxxxxxxxxxx > rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache > FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn > rpc.gssd[778]: WARNING: Machine cache prematurely expired or corrupted trying > to recreate cache for server server.fqdn > rpc.gssd[778]: No key table entry found for client$@NWRA.COM while getting > keytab entry for 'client$@NWRA.COM' > rpc.gssd[778]: No key table entry found for CLIENT$@NWRA.COM while getting > keytab entry for 'SRV-MRY01$@NWRA.COM' > rpc.gssd[778]: No key table entry found for root/client.fqdn@xxxxxxxx while > getting keytab entry for 'root/client.fqdn@xxxxxxxx' > rpc.gssd[778]: No key table entry found for nfs/client.fqdn@xxxxxxxx while > getting keytab entry for 'nfs/client.fqdn@xxxxxxxx' > rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry > for 'host/client.fqdn@xxxxxxxx' > rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC > 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 > rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC > 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024 > rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for > server server.fqdn > rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with > server nfs@xxxxxxxxxxx > rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for > server nfs@xxxxxxxxxxx > rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache > FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn > rpc.gssd[778]: ERROR: Failed to create machine krb5 context with any > credentials cache for server server.fqdn > rpc.gssd[778]: do_error_downcall(0x7f159f1fe640): uid 0 err -13 > > mount.nfs4: mount(2): Permission denied > mount.nfs4: access denied by server while mounting > > I don't seem to be getting any useful information from rpc.gssd on the server. > > Please include me in replies. I'm pretty sure this is the same issue that would be addressed by the patches that I sent to the list yesterday would address: https://lore.kernel.org/linux-nfs/20240228222253.1080880-1-smayhew@xxxxxxxxxx/T/#t There's a couple things you could check. 1. On the NFS server, increase the rpc debug logging just a little: # rpcdebug -m rpc -s auth and then after a failure, look for lines like this in the journal: Feb 29 18:14:44 rhel8.smayhew.test kernel: svc: svc_authenticate (6) Feb 29 18:14:44 rhel8.smayhew.test kernel: gss_kerberos_mech: unsupported krb5 enctype 20 Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_import_sec_context_kerberos: returning -22 Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_delete_sec_context deleting 0000000090f401ca 2. On the NFS client, increase the rpc-verbosity in the gssd stanza in /etc/nfs.conf (I have mine set to 3 but I think 1 would suffice) and then 'systemctl restart rpc-gssd'. then after a failure, look for a line like this in the journal: Feb 29 18:14:44 rhel9.smayhew.test rpc.gssd[1700]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer If you see both of those, then it's almost certainly the same issue. A quick solution would be to do this on your NFS server: # echo "mac@Kerberos = -HMAC-SHA2-*" >/usr/share/crypto-policies/policies/modules/NFS.pmod # update-crypto-policies --set DEFAULT:NFS # systemctl restart gssproxy but note that would be turning off the SHA2 enctypes for everything krb5-related, not just NFS. Note you can always revert that later simply by: # update-crypto-policies --set DEFAULT # systemctl restart gssproxy Or, you could test the patches I sent to the list yesterday (this would be on the client, not the server). The problem is those patches don't apply cleanly to the current version of nfs-utils shipped in EL9. At a quick glance, it looks like nfs-utils would need: 49567e7d configure: check for rpc_gss_seccreate 15cd5666 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials 2bfb59c6 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials 3abf6b52 gssd: switch to using rpc_gss_seccreate() f05af7d9 gssd: revert commit 513630d720bd 20c07979 gssd: revert commit a5f3b7ccb01c 14ee4878 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials 4b272471 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials 75b04a9b gssd: fix handling DNS lookup failure f066f87b gssd: enable forcing cred renewal using the keytab and you'd also need to patch libtirpc to include: 22b1c0c gssapi: fix rpc_gss_seccreate passed in cred -Scott > > -- > Orion Poplawski > he/him/his - surely the least important thing about me > Manager of IT Systems 720-772-5637 > NWRA, Boulder/CoRA Office FAX: 303-415-9702 > 3380 Mitchell Lane orion@xxxxxxxx > Boulder, CO 80301 https://www.nwra.com/
