On Thu, Jan 9, 2025 at 8:02 AM Christoph Hellwig <hch@xxxxxxxxxxxxx> wrote: > > On Tue, Jan 07, 2025 at 10:51:19PM +0000, Kangjing Huang wrote: > > This is good to know. However, since the SMB protocol explicitly calls > > for enumeration of all network interfaces on the server host, > > including their RDMA capabilities, I believe this is a sensible > > exception to the layering rule. Or is there anyway else to do this > > enumeration from the kernel space? > > No, it's not a sensible exception. It's a massive data leak and a > completely idiotic protocol feasture Linux should not support. If the > protocol requires a lsit of network interfaces, a Linux server should > require explicitly require that list to be configured and not expose > private information to the untrusted network. > I see the point. I wasn't thinking about it from the security perspective too much and would agree with you on this argument about data leaks. The protocol itself is not necessarily unsecured, since in the spec it calls for enumeration in an "implementation-specific manner"[1]. I would interpret that as the implementation could enumerate interfaces as it sees fit to its functional and security perspectives. The spec only requires the server to always return a list of client-usable interfaces, that does not need to always be the full interface list. That being said, from ksmbd's implementation perspective, this definitely needs to be something to be explicitly configured. ref: [1](https://winprotocoldoc.z19.web.core.windows.net/MS-SMB2/%5bMS-SMB2%5d-240729.pdf) sec 3.3.5.15.11 On Thu, Jan 9, 2025 at 7:59 AM Leon Romanovsky <leon@xxxxxxxxxx> wrote: ... > Except SMB spec, why do you need to provide "RDMA-capable" information? > Is it must? What will it give to the users? Why can't you treat IPoIB > like any other netdevice? ... > Let's start with an answer to more simple question: "why do you need > this capability flag?" The short answer is: a Windows client requires this flag to be present to actually utilize RDMA. Long answer: Although SMB-Direct (Microsoft speak for SMB over RDMA) protocol technically is just wrapping SMB packets in RDMA transport, allowing for a SMB connection to be completely handshaked and established in RDMA context, without any need for transferring ip packets. This kind of connection seems to be never happening for the actual client implementation in Windows. Under Windows, SMB Direct is only enabled when SMB multichannel is enabled. And there is no way for the user to specify if they want to connect to the IPoIB endpoint or RDMA endpoint - the client simply connects to the ip interface first, uses the aforementioned IOCTL request to query the RDMA capabilities for all interfaces, and upon finding usable RDMA-capable interfaces, opens preferred data channels on RDMA transport to carry the actual data. As far as I know there is no way from a Windows user's perspective to modify this behavior and upon disabling SMB multichannel, SMB will just disable RDMA support and transfer everything over IP. The RDMA transport is only "automatically upgraded to", no explicit configurations are available whatsoever. So if ksmbd does not send out capability flags to indicate the Windows client to initiate RDMA transport as a side channel, the client will just keep working with connections on the ip stack and the user can never utilize RDMA. To be clear: I would strongly agree that the Windows client's behavior here is pretty stupid and is just outright dumb design. In configuring my two-workstation setup utilizing this protocol, the no-config design of the Windows side has created far more problems than it has solved. It also somehow brought me here submitting my first kernel patch. However if we want any interoperability of ksmbd with Windows clients on the RDMA front, we would need to implement the IOCTL response to some degree and send out the RDMA capability flags. My apologies for not making this context earlier, I was too hyper-focused on the code itself to fall into the classic XY problem trap. Disclaimer: all my knowledge and research about windows client is limited to the edition I have access to (Windows 10 Pro Workstation), I have no idea if Windows server edition's client is different. ... > > Yes, ib_device_get_by_netdev and get_netdev are intended to perform > lookup of ib device based on underlying netdev, but in IPoIB case you > are interested in ib device based on upper netdev. > > So this leads to another question, if user didn't ask to connect SMB to > IB device and chose netdevice (IPoIB) instead, why are you still forcing > him to take IB path? If it is not user's decision and you are choosing > devices from the list, you already have in your list the IB device which > is connected to IPoIB. This is related to the long answer to the first question. Basically using a Windows client the user cannot make decisions to use RDMA or not, it is not the decision of the server, either - the client will enumerate the interfaces and make decisions for the user. And due to the presence of SMB multichannel, the ip interface receiving the interface enumeration request is not even guaranteed to be the connected IPoIB interface - the multichannel protocol is designed to work with service discovery, where it is totally possible that the initial connection will be established on another data path, before the client discovers another better path potentially with RDMA capability. The only way for ksmbd to be fully compatible with these is to implement the interface enumeration to some degree. Although I am indeed convinced that this needs to be explicitly configured to avoid any security risks. Thanks > > Thanks -- Kangjing "Chaser" Huang
