On Fri, 03 May 2024, Rick Macklem wrote: > On Fri, May 3, 2024 at 11:45 AM Chuck Lever <chuck.lever@xxxxxxxxxx> wrote: > > > > On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote: > > > On Thu, 02 May 2024, Chuck Lever III wrote: > > > > > > > > > > > > > > > > On May 2, 2024, at 1:37 PM, Scott Mayhew <smayhew@xxxxxxxxxx> wrote: > > > > > > > > > > On Thu, 02 May 2024, Chuck Lever III wrote: > > > > > > > > > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <smayhew@xxxxxxxxxx> wrote: > > > > >>> > > > > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an > > > > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with > > > > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a > > > > >>> mount to its mount table - it can't actually access the mount). > > > > >>> > > > > >>> Here's an example using machines from the recent Bakeathon. > > > > >>> > > > > >>> Mounting a server with TLS enabled: > > > > >>> > > > > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt > > > > >>> # umount /mnt > > > > >>> > > > > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none": > > > > >>> > > > > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt > > > > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt > > > > >>> > > > > >>> Yet a v3 mount without "xprtsec=tls" works: > > > > >>> > > > > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt > > > > >>> # umount /mnt > > > > >>> > > > > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to > > > > >>> v3 and also "works": > > > > >>> > > > > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt > > > > >>> # grep ora /proc/mounts > > > > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs > > > > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0 > > > > >>> > > > > >>> Even though the filesystem is mounted, the client can't do anything with it: > > > > >>> > > > > >>> # ls /mnt > > > > >>> ls: cannot open directory '/mnt': Permission denied > > > > >>> > > > > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in > > > > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1). > > > > >>> The client compares that list with its own list of auth flavors parsed from the > > > > >>> mount request and returns -EACCES if no match is found (see > > > > >>> nfs_verify_authflavors()). > > > > >>> > > > > >>> Perhaps we should be doing something similar with xprtsec policies? > > > > >> > > > > >> The problem might be in how you've set up the exports. With NFSv3, > > > > >> the parent export needs the "crossmnt" export option in order for > > > > >> NFSv3 to behave like NFSv4 in this regard, although I could have > > > > >> missed something. > > > > > > > > > > I was mounting your server though :) > > > > > > > > OK, then not the same bug that Olga found last year. > > > > > > > > We should find out what FreeBSD does in this case. > > > > > > I thought about that. Rick's servers from the BAT are offline, and I > > > don't think he was exporting v3 anyway. > If you want me to leave the FreeBSD server up on tailscale configured to > allow NFSv3/RPC-with-TLS mounts, just email. > > I did a quick test using the FreeBSD client and the mount fails, but I know > that failure happens when it tries to access the server file system in > the kernel. > The replies fail with RPC Msg_denied. > (I've attached a packet trace, in case you are interested.) Thanks, Rick. Looking at that trace, freebsd is rejecting the GETATTR and FSINFO requests at the RPC layer, whereas Linux knfsd isn't. Digging a little further, both nfsd3_proc_fsinfo() and nfsd3_proc_getattr() are calling fh_verify() with a flag called NFSD_MAY_BYPASS_GSS_ON_ROOT ---8<--- resp->status = fh_verify(rqstp, &resp->fh, 0, NFSD_MAY_NOP | NFSD_MAY_BYPASS_GSS_ON_ROOT); ---8<--- and fh_verify() has ---8<--- /* * Clients may expect to be able to use auth_sys during mount, * even if they use gss for everything else; see section 2.3.2 * of rfc 2623. */ if (access & NFSD_MAY_BYPASS_GSS_ON_ROOT && exp->ex_path.dentry == dentry) goto skip_pseudoflavor_check; error = check_nfsd_access(exp, rqstp); ---8<--- so it's skipping check_nfsd_access(), which is where the xprtsec policy checking occurs. Looking at RFC 2623 section 2.3.2, it basically says a server can choose to not require authentication for GETATTR and FSINFO. Obviously, RFC 2623 predates RPC-with-TLS by 20+ years, so there's nothing there to dictate what to do here... but as Chuck reiterated, TLS isn't a security flavor so maybe knfsd shouldn't be bypassing those xprtsec policy checks at all. -Scott > > I haven't done anything with the userspace mountd daemon, which means it > will allow a Mount protocol RPC without TLS and does not return any different > info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard, > but I have tried hard to avoid adding RPC-with-TLS to the userspace > RPC libraries. > (Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.) > > > > > > > > > > > > > > > > >>> Should > > > > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for > > > > >>> pseudo-flavors corresponding to xprtsec policies? > > > > >> > > > > >> No. Transport-layer security is not an RPC security flavor or > > > > >> pseudo-flavor. These two things are not related. > > > > >> > > > > >> (And in fact, I proposed something like this for NFSv4 SECINFO, > > > > >> but it was rejected). > > > > > > > > > > I thought it might be a stretch to try to use mountres3.auth_flavors for > > > > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication > > > > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml > > > > > also lists TLS under the Flavor Name column I thought it might make > > > > > sense to treat xprtsec policies as if they were pseudo-flavors even > > > > > though they're not, if only to give the client a way to determine that > > > > > the mount should fail. > > > > > > > > RPC_AUTH_TLS is used only when a client probes a server to see if > > > > it supports RPC-with-TLS. At all other times, the client uses one > > > > of the normal, legitimate flavors. It does not represent a security > > > > flavor that can be used during regular operation. > > > > > > > > NFSv3 mount failover logic is still open for discussion (ie, incomplete). > > > > > > > > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors > > > > list? I think clients that don't recognize it should ignore it, > > > > but I'm not sure. What should a client do if it sees that flavor in > > > > the list? It's not one that can be used for any other procedure than > > > > a NULL RPC. > > > > > > Maybe? After the client gets the filehandle it's calling FSINFO and > > > PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't > > > checking for a negative return code from the PATHCONF operation. If it > > > did, it could maybe use the -EACCES coupled with the knowledge that the > > > server had RPC_AUTH_TLS enabled to emit an error message saying to check > > > the xprtsec policies (but I don't think that would be as definitive as > > > what I had in mind) and to fail the mount. > > > > If Linux is the only implementation of NFSv3 with TLS so far, then > > we have some latitude for innovation. > At this point, FreeBSD can change to whatever you guys think works best, > although I'd like to avoid adding RPC-with-TLS support to the Mount protocol. > > Have fun with it, rick > > > > > I would like to hear from the client maintainers about what they > > would prefer the client user experience to look like. Then NFSD's > > behavior can be adjusted to accommodate. > > > > In this case, Steve would have to sign off on an rpc.mountd change > > to return AUTH_TLS in the auth_flavors list. > > > > > > -- > > Chuck Lever > >
