On Mon, Sep 25, 2023 at 01:50:22PM +0200, Alexandra Winter wrote: > > >On 24.09.23 17:16, Wen Gu wrote: >> This patch introduces a kind of loopback device for SMC-D. The device >> is created when SMC module is loaded and destroyed when the SMC module >> is unloaded. The loopback device is a kernel device used only by the >> SMC module and is not restricted by net namespace, so it can be used >> for local inter-process or inter-container communication. >> >> Signed-off-by: Wen Gu <guwen@xxxxxxxxxxxxxxxxx> >> --- >> net/smc/Kconfig | 13 ++++ >> net/smc/Makefile | 2 +- >> net/smc/af_smc.c | 12 +++- >> net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++ >> net/smc/smc_loopback.h | 33 ++++++++++ >> 5 files changed, 223 insertions(+), 2 deletions(-) >> create mode 100644 net/smc/smc_loopback.c >> create mode 100644 net/smc/smc_loopback.h > > >Hello Wen Gu, > >thank you for adding the Kconfig, so the distributions can decide when to offer this feature. > >I propose you add some kind of runtime switch as well. Not every user who loads the SMC module >may want to exploit smcd-loopback. Especially in native environements without containers. > >If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. >If loopback is always created unconditionally, there is no way to opt-out. Hi Sandy, After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c Without this, we cannot do: 1. Enable/disable those devices dynamically Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally. If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices. 2. Do statistics per device Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device. Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications. So I think those are better to support in the SMC layer. Best regards! Dust
