We (open source SMB3 developers) have been thinking a lot about SMB3.1.1 over QUIC especially after the interesting talks on this at the last few SDC conferences e.g. https://www.snia.org/sites/default/files/SDC/2019/presentations/SMB/Li_Frank_Surfing_the_World_Wide_File_SMB3_Improvements_for_Safe_and_Efficient_Internet_Access.pdf (slides 8 and following). And some of the broader coverage: https://redmondmag.com/articles/2020/03/02/microsoft-smb-over-quic-to-windows.aspx I have gotten a chance to talk with some of the developers of the recently open sourced quic libraries (https://github.com/microsoft/msquic) and there are various code style issues that would prevent it being merged into Linux kernel without significant changes (not just to fix camelCase) but ... it is a very very promising place to start. One of the ideas being bounced around is to do this in stages - first try to cleanup the kernel style issues with the open sourced msquic project and merge it experimentally into the Linux client (cifs.ko) since there should be SMB3.1.1. servers to test it against reasonably soon (presumably we will know a lot more after the 2020 SNIA SDC talk on SMB3.1.1 over QUIC in a few weeks updating us on status https://www.snia.org/events/storage-developer/2020/abstracts#smb-Dantuluri) after we are convinced that the quic driver interoperates ok then move it into the Linux networking stack and let the networking experts clean it up more and remove the 'experimental' tag from it when satisfied. More discussion is needed on this so we should add samba-technical and some of the network experts as well. I am very excited about the possibilities opened up by a decent quic kernel driver, but the current cross platform open sourced driver is fairly large - ~90KLOC so will have to be trimmed down (and a lot of boring changes to convert to kernel style needed). -----Original Message----- From: Chuck Lever <chucklever@xxxxxxxxx> Sent: Thursday, September 3, 2020 7:53 PM To: Bruce Fields <bfields@xxxxxxxxxxxx> Cc: Linux NFS Mailing List <linux-nfs@xxxxxxxxxxxxxxx>; NFSv4 <nfsv4@xxxxxxxx>; Steven French <Steven.French@xxxxxxxxxxxxx> Subject: [EXTERNAL] Re: [nfsv4] NFS over QUIC > On Sep 3, 2020, at 8:32 PM, Bruce Fields <bfields@xxxxxxxxxxxx> wrote: > > On Thu, Sep 03, 2020 at 07:48:19PM -0400, Chuck Lever wrote: >> Hi Bruce- >> >>> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <bfields@xxxxxxxxxxxx> wrote: >>> >>> I've been thinking about what might be required for NFS to run over >>> QUIC. >>> >>> Also cc'ing Steve French in case he's thought about this for CIFS/SMB. >>> >>> I don't have real plans. For Linux, I don't even know if there's a >>> kernel QUIC implementation planned yet. >>> >>> QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft: >>> >>> >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fda >>> tatracker.ietf.org%2Fdoc%2Fdraft-cel-nfsv4-rpc-tls%2F&data=02%7C >>> 01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee6 >>> 3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&am >>> p;sdata=pOnlPJcTkyJiqQm%2BCbo7dbOAqHMKFIi01qrShVZTH%2Bk%3D&reser >>> ved=0 >> >> The link to the latest version of that document is >> >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdat >> atracker.ietf.org%2Fdoc%2Fdraft-ietf-nfsv4-rpc-tls%2F&data=02%7C0 >> 1%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63% >> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&s >> data=YYBPryN5sFJApWm%2BkQvmjYzGXxjxI1fXRUWVXvVGE24%3D&reserved=0 >> >>> For example, section 4.3, which explains how to authenticate on top >>> of an already-encrypted session, should also apply to QUIC. >> >> Most of the document's content will be re-used for defining >> RPC-over-QUIC, for example the ALPN defined in Section 8.2. >> Lars Eggert, a chair of the QUIC WG, has been helping guide the >> RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC >> becomes more mature. >> >> I thought the plan was to write a specification of RPC-over- QUIC as >> a new RPC transport type with a netid and uaddr along with a >> definition of the transport semantics (a la TI-RPC). >> The document would need to explain record marking, peer >> authentication, how to use multi-path and multi-stream support, and >> so on. >> >> Making NFS work on that transport should then be straightforward >> enough that perhaps additional standards work wouldn't be necessary. > > Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know > there was a plan. That's all there is for the moment! :-) > --b. > >>> QUIC runs over UDP, so I think all that would be required to >>> negotiate support would be to attempt a QUIC connection to port 2049. >>> >>> The "Transport Layers" section in the NFS RFCs: >>> >>> >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto >>> ols.ietf.org%2Fhtml%2Frfc5661%23section-2.9&data=02%7C01%7CSteve >>> n.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%7C72f988 >>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&sdata=ej >>> gN6OuHrcDMzejeEKbaP5UiD1GAxHfQcPyLZZ45vbo%3D&reserved=0 >>> >>> requires transports support reliable and in-order transmission, >>> forbids clients from retrying a request unless a connection is lost, >>> and forbids servers from dropping a request without closing a >>> connection. I'm still vague on how those requirements interact with >>> QUIC's connection management and 0-RTT reconnection. >>> >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww >>> w.ietf.org%2Fid%2Fdraft-ietf-quic-applicability-07.txt&data=02%7 >>> C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee >>> 63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&sdata=TZeoR%2FT22hg%2BkG0LllDEttlgdzv%2FAsH8PeXbdEHQ9Mk%3D&reserved=0 looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly. >>> Sounds like that's what they do for HTTP/QUIC to make server push >>> notifications work: >>> >>> >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto >>> ols.ietf.org%2Fhtml%2Fdraft-ietf-quic-http-09%23section-5&data=0 >>> 2%7C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506 >>> cee63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63734777607589360 >>> 4&sdata=rF91Tq8OXHeN7W5iuSbtYotq1XBC4Zc8DGEGzwPUuTg%3D&reser >>> ved=0 >>> >>> HTTP clients are expected to use QUIC PING frames to keep >>> connections open. Servers SHOULD NOT use PING frames to keep a >>> connection open. A client SHOULD NOT use PING frames for this >>> purpose unless there are responses outstanding for requests or >>> server pushes. >>> >>> QUIC allows multiple streams per connection--I wonder how we might >>> use that. RFC 5661 justifies the requirement for an ordered transport with: >>> >>> Ordered delivery simplifies detection of transmit errors, and >>> simplifies the sending of arbitrary sized requests and responses >>> via the record marking protocol. >>> >>> So as long as we don't try to split a single RPC among streams, I >>> think we're OK. Would a stream per session slot be reasonable? I'm >>> not sure what the cost of a stream is. >>> >>> Do we need to add a new universal address type so the protocol can >>> specify QUIC endpoints when necessary? (For server-to-server-copy, >>> pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP >>> address and maybe a port, so maybe the existing UDP/TCP addresses are enough? >> >> -- >> Chuck Lever >> chucklever@xxxxxxxxx -- Chuck Lever chucklever@xxxxxxxxx
