Hello, I'm not sure if your problem is like this. But I know there is a bug for selinux. It will prevent to mount cifs with krb5 when krb5 upgrade from 1.17 to 1.18 in RHEL8. Disabled selinux or executing "setsebool samba_export_all_rw=1" will fix this issue. https://bugzilla.redhat.com/show_bug.cgi?id=1847221 Thanks. ----- Original Message ----- > From: "Steve French via samba-technical" <samba-technical@xxxxxxxxxxxxxxx> > To: "Michael Keane, K1MK" <mkeane@xxxxxxxx> > Cc: "Shyam Prasad N" <nspmangalore@xxxxxxxxx>, "CIFS" <linux-cifs@xxxxxxxxxxxxxxx>, "samba-technical" > <samba-technical@xxxxxxxxxxxxxxx> > Sent: Wednesday, July 22, 2020 12:03:10 PM > Subject: Re: issue -- cifs automounts stopped working > > There are two common Kerberos libraries, MIT and Heimdal (Heimdal is > also packaged as an optional part of Samba as an alternative to MIT > Kerberos) ... would be interesting if there is a way to work around > this by switching to Heimdal krb5 libraries. > > Adding samba-technical as some of the developers on samba-technical > know the Kerberos maintainers and may have suggestions on how to track > down this possible regression in MIT kerberos. > > On Tue, Jul 21, 2020 at 10:48 PM Michael Keane, K1MK <mkeane@xxxxxxxx> wrote: > > > > Tried setting KRB5_CCNAME to KEYRING:persistent:1000 with no change > > (Also tried KRB5CCNAME as well, as that's the environment variable > > which seems to work with kinit & klist) > > > > Also tried setting default_ccahe_name in /etc/krb5.conf to a legacy > > value of > > > > default_ccache_name = /tmp/krb5_%{uid} > > > > in order to force it to use the filesystem rather than the kernel space > > for the credentials cache. And it still fails in the same way. I think > > that tends to makes this more likely a regression in krb5 than the kernel. > > > > I believe that I may have verified this by reverting from > > krb5-*/-/1.18.2-10 to krb5-*-1.17.1-5 which resolves the issue > > > > > > > > On 07/21/2020 3:59 PM, Steve French wrote: > > > Would setting KRB5_CCNAME (as would be expected in the past) for that > > > process fix the issue? > > > > > > On Tue, Jul 21, 2020 at 2:52 PM Michael Keane, K1MK <mkeane@xxxxxxxx> > > > wrote: > > >> KRB5_CCNAME is not set in the environment of the user for whom the > > >> automount is failing. > > >> > > >> In /etc/krb5.conf the default is set to: > > >> > > >> default_ccache_name = KEYRING:persistent:%{uid} > > >> > > >> The calls to keyctl() in the strace for cifs.upcall indicate that it's > > >> successfully locating the user's keyring in the kernel and that it is > > >> able to read key data from that keying > > >> > > >> I haven't gone through and tried unpacking and decoding the contents of > > >> those reads, but the reads aren't failing outright like it couldn't find > > >> the credentials cache or the contents that it's looking for > > >> > > >> > > >> On 07/21/2020 1:36 PM, Steve French wrote: > > >>> That is plausible but I also wonder about other whether other parts of > > >>> krb5 configuration are changed/broken eg KRB5_CCNAME environment > > >>> variable > > >>> > > >>> See https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html > > >>> > > >>> On Tue, Jul 21, 2020, 12:24 Michael Keane, K1MK <mkeane@xxxxxxxx > > >>> <mailto:mkeane@xxxxxxxx>> wrote: > > >>> > > >>> Thanks > > >>> > > >>> In trying to further debug this problem, I've stood up a separate > > >>> Fedora 31 instance for testing with the result that the same > > >>> configuration that is failing on Fedora 32 is working under Fedora > > >>> 31 > > >>> > > >>> Using strace it would appear the deviation occurs at the point of > > >>> "handle_krb5_mech" where the Fedora 31 instance proceeds though > > >>> several keyctl() calls to a successful return while the Fedora 32 > > >>> instance goes through a similar sequence of keyctl() calls but > > >>> rather than finishing up with a final call to keyctl() to > > >>> instantiate the key it appears that the krb5 library and/or sssd > > >>> is going through a series of steps to locate the KDC (and failing) > > >>> even though the KDCs have always been explicitly configured in > > >>> /etc/krb5.conf > > >>> > > >>> So the root cause of this issue may not be anything in cifs.upcall > > >>> but rather something in the sssd or kerberos that got changed as a > > >>> result of moving from Fedora 31 to Fedora 32 > > >>> > > >>> On 07/18/2020 8:40 PM, Steve French wrote: > > >>>> Looks like error obtaining the key (either keyutils package not > > >>>> installed or no Kerberos ticket found). > > >>>> > > >>>> This page has debug instructions for cifs.upcall. see of that > > >>>> information is helpful. > > >>>> > > >>>> http://sprabhu.blogspot.com/2014/12/debugging-calls-to-cifsupcall.html?m=1 > > >>>> > > >>>> Also try kinit as a local user and then mount with cruid mount > > >>>> option pointing to their uid to see if that helps. > > >>>> > > >>>> On Fri, Jul 17, 2020, 09:04 Michael Keane, K1MK <mkeane@xxxxxxxx > > >>>> <mailto:mkeane@xxxxxxxx>> wrote: > > >>>> > > >>>> CIFS automounts to local NAS devices have stopped working > > >>>> recently > > >>>> > > >>>> Fedora 32 > > >>>> > > >>>> kernel 5.7.8-200.fc32.x86_64 > > >>>> > > >>>> mount.cifs version: 6.9 > > >>>> > > >>>> auto.misc: it-share > > >>>> -fstype=cifs,multiuser,cruid=${UID},rw,vers=default,sec=krb5 > > >>>> ://filer5/IT_Share > > >>>> > > >>>> dmesg: > > >>>> > > >>>> [ 3428.883661] fs/cifs/cifsfs.c: Devname: > > >>>> //filer5.arrlhq.org/IT_Share > > >>>> <http://filer5.arrlhq.org/IT_Share> flags: 0 > > >>>> [ 3428.883702] fs/cifs/connect.c: Username: root > > >>>> [ 3428.883706] fs/cifs/connect.c: file mode: 0755 dir > > >>>> mode: 0755 > > >>>> [ 3428.883709] fs/cifs/connect.c: CIFS VFS: in > > >>>> mount_get_conns as > > >>>> Xid: 12 with uid: 0 > > >>>> [ 3428.883710] fs/cifs/connect.c: UNC: > > >>>> \\filer5.arrlhq.org <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3428.883721] fs/cifs/connect.c: Socket created > > >>>> [ 3428.883723] fs/cifs/connect.c: sndbuf 16384 rcvbuf > > >>>> 131072 > > >>>> rcvtimeo 0x1b58 > > >>>> [ 3428.884126] fs/cifs/fscache.c: > > >>>> cifs_fscache_get_client_cookie: > > >>>> (0x000000006d2b3226/0x0000000090edec3a) > > >>>> [ 3428.884130] fs/cifs/connect.c: CIFS VFS: in > > >>>> cifs_get_smb_ses as > > >>>> Xid: 13 with uid: 0 > > >>>> [ 3428.884131] fs/cifs/connect.c: Existing smb sess not > > >>>> found > > >>>> [ 3428.884135] fs/cifs/smb2pdu.c: Negotiate protocol > > >>>> [ 3428.884141] fs/cifs/connect.c: Demultiplex PID: 5778 > > >>>> [ 3428.884156] fs/cifs/transport.c: Sending smb: > > >>>> smb_len=252 > > >>>> [ 3429.010818] fs/cifs/connect.c: RFC1002 header 0x11c > > >>>> [ 3429.010830] fs/cifs/smb2misc.c: SMB2 data length 96 > > >>>> offset 128 > > >>>> [ 3429.010832] fs/cifs/smb2misc.c: SMB2 len 224 > > >>>> [ 3429.010835] fs/cifs/smb2misc.c: length of negcontexts > > >>>> 60 pad 0 > > >>>> [ 3429.010871] fs/cifs/transport.c: cifs_sync_mid_result: > > >>>> cmd=0 > > >>>> mid=0 state=4 > > >>>> [ 3429.010885] fs/cifs/misc.c: Null buffer passed to > > >>>> cifs_small_buf_release > > >>>> [ 3429.010891] fs/cifs/smb2pdu.c: mode 0x1 > > >>>> [ 3429.010893] fs/cifs/smb2pdu.c: negotiated smb3.1.1 > > >>>> dialect > > >>>> [ 3429.010903] fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 > > >>>> 0x348 0xbb92 > > >>>> [ 3429.010907] fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 > > >>>> 0x348 0x1bb92 > > >>>> [ 3429.010910] fs/cifs/asn1.c: OID len = 10 oid = 0x1 0x3 > > >>>> 0x6 0x1 > > >>>> [ 3429.010912] fs/cifs/smb2pdu.c: decoding 2 negotiate > > >>>> contexts > > >>>> [ 3429.010914] fs/cifs/smb2pdu.c: decode SMB3.11 > > >>>> encryption neg > > >>>> context of len 4 > > >>>> [ 3429.010916] fs/cifs/smb2pdu.c: SMB311 cipher type:1 > > >>>> [ 3429.010921] fs/cifs/connect.c: Security Mode: 0x1 > > >>>> Capabilities: > > >>>> 0x300046 TimeAdjust: 0 > > >>>> [ 3429.010923] fs/cifs/smb2pdu.c: Session Setup > > >>>> [ 3429.010926] fs/cifs/smb2pdu.c: sess setup type 5 > > >>>> [ 3429.010949] fs/cifs/cifs_spnego.c: key description = > > >>>> ver=0x2;host=filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org>;ip4=10.1.123.38;sec=krb5;uid=0x0;creduid=0x3e8;user=root;pid=0x1690 > > >>>> [ 3429.025053] CIFS VFS: \\filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org> Send error in SessSetup > > >>>> = -126 > > >>>> [ 3429.025056] fs/cifs/connect.c: CIFS VFS: leaving > > >>>> cifs_get_smb_ses > > >>>> (xid = 13) rc = -126 > > >>>> [ 3429.025059] fs/cifs/connect.c: build_unc_path_to_root: > > >>>> full_path=\\filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3429.025059] fs/cifs/connect.c: build_unc_path_to_root: > > >>>> full_path=\\filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3429.025060] fs/cifs/connect.c: build_unc_path_to_root: > > >>>> full_path=\\filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3429.025062] fs/cifs/dfs_cache.c: __dfs_cache_find: > > >>>> search path: > > >>>> \filer5.arrlhq.org <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3429.025063] fs/cifs/dfs_cache.c: get_dfs_referral: get > > >>>> an DFS > > >>>> referral for \filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3429.025065] fs/cifs/dfs_cache.c: dfs_cache_noreq_find: > > >>>> path: > > >>>> \filer5.arrlhq.org <http://filer5.arrlhq.org>\IT_Share > > >>>> [ 3429.025071] fs/cifs/fscache.c: > > >>>> cifs_fscache_release_client_cookie: > > >>>> (0x000000006d2b3226/0x0000000090edec3a) > > >>>> [ 3429.025076] fs/cifs/connect.c: CIFS VFS: leaving > > >>>> mount_put_conns > > >>>> (xid = 12) rc = 0 > > >>>> [ 3429.025077] CIFS VFS: cifs_mount failed w/return code > > >>>> = -2 > > >>>> > > >>>> journalctl: > > >>>> > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: key > > >>>> description: > > >>>> cifs.spnego;0;0;39010000;ver=0x2;host=filer5.arrlhq.org > > >>>> <http://filer5.arrlhq.org>;ip4=10.1.123.38;sec=krb5;uid=0x0;creduid=0x3e8;user=root;pid=0x1690 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: ver=2 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: > > >>>> host=filer5.arrlhq.org <http://filer5.arrlhq.org> > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: ip=10.1.123.38 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: sec=1 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: uid=0 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: creduid=1000 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: user=root > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: pid=5776 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: > > >>>> get_cachename_from_process_env: > > >>>> pathname=/proc/5776/environ > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: > > >>>> get_existing_cc: default ccache is > > >>>> KEYRING:persistent:1000:1000 > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: > > >>>> handle_krb5_mech: getting service ticket for > > >>>> filer5.arrlhq.org <http://filer5.arrlhq.org> > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: > > >>>> cifs_krb5_get_req: unable to get credentials for > > >>>> filer5.arrlhq.org <http://filer5.arrlhq.org> > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: > > >>>> handle_krb5_mech: failed to obtain service ticket > > >>>> (-1765328370) > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: Unable to > > >>>> obtain service ticket > > >>>> Jul 17 09:43:13 mkZ230.ARRLHQ.ORG > > >>>> <http://mkZ230.ARRLHQ.ORG> cifs.upcall[5779]: Exit status > > >>>> -1765328370 > > >>>> > > >>>> $ klist > > >>>> Ticket cache: KEYRING:persistent:1000:1000 > > >>>> Default principal: mkeane@xxxxxxxxxx > > >>>> <mailto:mkeane@xxxxxxxxxx> > > >>>> > > >>>> Valid starting Expires Service principal > > >>>> 07/17/2020 09:43:06 07/17/2020 19:43:06 > > >>>> krbtgt/ARRLHQ.ORG@xxxxxxxxxx <mailto:ARRLHQ.ORG@xxxxxxxxxx> > > >>>> renew until 07/24/2020 09:42:57 > > >>>> > > >>>> Filer5 is a QNAP TS-870U-RP Version 4.3.6.1070 (2019/09/10) > > >>>> NAS device > > >>>> but having this issue with other NAS device on LAN > > >>>> > > >>>> -- > > >>>> Michael Keane, K1MK > > >>>> IT Manager > > >>>> ARRL, The National Association for Amateur Radio™ > > >>>> 225 Main Street, Newington, CT 06111-1494 USA > > >>>> Telephone: (860) 594-0285 > > >>>> email: mkeane@xxxxxxxx <mailto:mkeane@xxxxxxxx> > > >>>> > > >>> -- > > >>> Michael Keane, K1MK > > >>> IT Manager > > >>> ARRL, The National Association for Amateur Radio™ > > >>> 225 Main Street, Newington, CT 06111-1494 USA > > >>> Telephone: (860) 594-0285 > > >>> email:mkeane@xxxxxxxx <mailto:mkeane@xxxxxxxx> > > >>> > > >> -- > > >> Michael Keane, K1MK > > >> IT Manager > > >> ARRL, The National Association for Amateur Radio™ > > >> 225 Main Street, Newington, CT 06111-1494 USA > > >> Telephone: (860) 594-0285 > > >> email: mkeane@xxxxxxxx > > >> > > > > > > > -- > > Michael Keane, K1MK > > IT Manager > > ARRL, The National Association for Amateur Radio™ > > 225 Main Street, Newington, CT 06111-1494 USA > > Telephone: (860) 594-0285 > > email: mkeane@xxxxxxxx > > > > > -- > Thanks, > > Steve > >
