On Thu, 2018-02-01 at 10:52 -0700, Jason Gunthorpe wrote: > On Thu, Feb 01, 2018 at 09:18:10AM +0100, Stefan Metzmacher wrote: > > > The first goal is to provide a socket fd to userspace (or in kernel > > consumers) > > which provides semantics like a TCP socket which is used as transport > > for SMB3. Basically frames are submitted with a 4 byte length header. > > Part of the point of RDMA is that we don't need to make protocol > specific kernel modules like this - is there a specific reason this > needs to be in the kernel like this? Although your question was partially answered, I think the precise cause is that to behave like an SMB3 server, the model involves opening a TCP connection, doing auth over that, then negotiating whether or not RDMA connections are possible, then opening the RDMA connections. On our end, once the auth happens, smbd forks and the new user process handles the connection from that point on (which is integral to smbd's security model). Because of this, and because the client initiates the RDMA connection and will only use a single well known port to connect to, incoming RDMA connections after negotiation would always go to the master smbd process and not the forked client authed smbd process. Because we don't have a way of then handing the RDMA connection from one process to another forked process, the model breaks down. Possible solutions are: 1) Don't make the master process listen on the well known RDMA port. Let each client register to listen on the port only after the negotiation has completed, listen until it finds the proper client connection, then stop listening on the port. This obviously makes multiple client connections race for which server is listening. I have no idea if the Microsoft client would even retry a connection if it went to the wrong process the first time and was rejected. 2) Create a new listen model where a process could register to listen on a specific port + remote IP address(es). That way all connection attempts from the specific remote IP address(es) and the local port would go to the specific process. This could be contained within the CMA and need not impact normal sockets processes. I would envision that the smbd would get the negotiation packet from the client with a list of the RDMA devices it can connect across, would work out which of those devices it could accept connections from, would initiate all necessary listens, and finally would send the reply packet to the client. 3) Make a kernel module as they are doing to handle the incoming connections and multiplex between processes. 4) Make smbd threaded so that it can listen on the well known port and then hand a connection off to a specific thread once established. I'm partial to either 2 or 4 myself. -- Doug Ledford <dledford@xxxxxxxxxx> GPG KeyID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
Attachment:
signature.asc
Description: This is a digitally signed message part