On Thu, 2018-02-01 at 10:52 -0700, Jason Gunthorpe wrote:
> On Thu, Feb 01, 2018 at 09:18:10AM +0100, Stefan Metzmacher wrote:
>
> > The first goal is to provide a socket fd to userspace (or in kernel
> > consumers)
> > which provides semantics like a TCP socket which is used as transport
> > for SMB3. Basically frames are submitted with a 4 byte length header.
>
> Part of the point of RDMA is that we don't need to make protocol
> specific kernel modules like this - is there a specific reason this
> needs to be in the kernel like this?
Although your question was partially answered, I think the precise cause
is that to behave like an SMB3 server, the model involves opening a TCP
connection, doing auth over that, then negotiating whether or not RDMA
connections are possible, then opening the RDMA connections. On our
end, once the auth happens, smbd forks and the new user process handles
the connection from that point on (which is integral to smbd's security
model). Because of this, and because the client initiates the RDMA
connection and will only use a single well known port to connect to,
incoming RDMA connections after negotiation would always go to the
master smbd process and not the forked client authed smbd process.
Because we don't have a way of then handing the RDMA connection from one
process to another forked process, the model breaks down. Possible
solutions are:
1) Don't make the master process listen on the well known RDMA port.
Let each client register to listen on the port only after the
negotiation has completed, listen until it finds the proper client
connection, then stop listening on the port. This obviously makes
multiple client connections race for which server is listening. I have
no idea if the Microsoft client would even retry a connection if it went
to the wrong process the first time and was rejected.
2) Create a new listen model where a process could register to listen on
a specific port + remote IP address(es). That way all connection
attempts from the specific remote IP address(es) and the local port
would go to the specific process. This could be contained within the
CMA and need not impact normal sockets processes. I would envision that
the smbd would get the negotiation packet from the client with a list of
the RDMA devices it can connect across, would work out which of those
devices it could accept connections from, would initiate all necessary
listens, and finally would send the reply packet to the client.
3) Make a kernel module as they are doing to handle the incoming
connections and multiplex between processes.
4) Make smbd threaded so that it can listen on the well known port and
then hand a connection off to a specific thread once established.
I'm partial to either 2 or 4 myself.
--
Doug Ledford <dledford@xxxxxxxxxx>
GPG KeyID: B826A3330E572FDD
Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDDAttachment:
signature.asc
Description: This is a digitally signed message part
