On Wed, 19 Jun 2024, Chuck Lever wrote:
> On Tue, Jun 18, 2024 at 04:19:49PM -0400, Mike Snitzer wrote:
> > This document gives an overview of the LOCALIO protocol extension
> > added to the Linux NFS client and server (both v3 and v4) to allow a
> > client and server to reliably handshake to determine if they are on
> > the same host. The LOCALIO protocol extension follows the well-worn
> > pattern established by the ACL protocol extension.
> >
> > The robust handshake between local client and server is just the
> > beginning, the ultimate use-case this locality makes possible is the
> > client is able to issue reads, writes and commits directly to the
> > server without having to go over the network.
> >
> > Signed-off-by: Mike Snitzer <snitzer@xxxxxxxxxx>
> > ---
> > Documentation/filesystems/nfs/localio.rst | 101 ++++++++++++++++++++++
> > include/linux/nfslocalio.h | 2 +
> > 2 files changed, 103 insertions(+)
> > create mode 100644 Documentation/filesystems/nfs/localio.rst
> >
> > diff --git a/Documentation/filesystems/nfs/localio.rst b/Documentation/filesystems/nfs/localio.rst
> > new file mode 100644
> > index 000000000000..4b4595037a7f
> > --- /dev/null
> > +++ b/Documentation/filesystems/nfs/localio.rst
> > @@ -0,0 +1,101 @@
> > +===========
> > +NFS localio
> > +===========
> > +
> > +This document gives an overview of the LOCALIO protocol extension added
> > +to the Linux NFS client and server (both v3 and v4) to allow a client
> > +and server to reliably handshake to determine if they are on the same
> > +host. The LOCALIO protocol extension follows the well-worn pattern
> > +established by the ACL protocol extension.
> > +
> > +The LOCALIO protocol extension is needed to allow robust discovery of
> > +clients local to their servers. Prior to this extension a fragile
> > +sockaddr network address based match against all local network
> > +interfaces was attempted. But unlike the LOCALIO protocol extension,
> > +the sockaddr-based matching didn't handle use of iptables or containers.
> > +
> > +The robust handshake between local client and server is just the
> > +beginning, the ultimate use-case this locality makes possible is the
> > +client is able to issue reads, writes and commits directly to the server
> > +without having to go over the network. This is particularly useful for
> > +container usecases (e.g. kubernetes) where it is possible to run an IO
> > +job local to the server.
> > +
> > +The performance advantage realized from localio's ability to bypass
> > +using XDR and RPC for reads, writes and commits can be extreme, e.g.:
> > +fio for 20 secs with 24 libaio threads, 64k directio reads, qd of 8,
> > +- With localio:
> > + read: IOPS=691k, BW=42.2GiB/s (45.3GB/s)(843GiB/20002msec)
> > +- Without localio:
> > + read: IOPS=15.7k, BW=984MiB/s (1032MB/s)(19.2GiB/20013msec)
> > +
> > +RPC
> > +---
> > +
> > +The LOCALIO RPC protocol consists of a single "GETUUID" RPC that allows
> > +the client to retrieve a server's uuid. LOCALIOPROC_GETUUID encodes the
> > +server's uuid_t in terms of the fixed UUID_SIZE (16 bytes). The fixed
> > +size opaque encode and decode XDR methods are used instead of the less
> > +efficient variable sized methods.
>
> I'm reading between the lines ("well-worn pattern established by
> the [NFS]ACL protocol"). I'm guessing that the client and server
> will exchange this protocol on the same connection as NFS traffic?
>
> The use of the term "extension" in this Document might be atypical.
> An /extension/ means that the base RPC program (NFS in this case)
> is somehow modified. However, if LOCALIO is a distinct RPC program
> then this isn't an extension of the NFS protocol, per se.
>
> A protocol spec needs to include:
>
> o The RPC program and version number
>
> o A description of each its procedures, along with an XDR definition
> of its arguments and results
>
> o Any related constants or bit mask values
Note that providing this information in the format of a ".x" file as
understood by rpcgen is a good approach.
It isn't clear to me why you implement both v3 and v4 of the LOCALIO
program. I don't see how they relate to the NFS protocol version. Just
implement v1 which simply returns the UUID.
Thanks,
NeilBrown
