On 2024/2/6 20:18, Alexandra Winter wrote:
On 11.01.24 13:00, Wen Gu wrote:
This patch set acts as the second part of the new version of [1] (The first
part can be referred from [2]), the updated things of this version are listed
at the end.
# Background
SMC-D is now used in IBM z with ISM function to optimize network interconnect
for intra-CPC communications. Inspired by this, we try to make SMC-D available
on the non-s390 architecture through a software-implemented virtual ISM device,
that is the loopback-ism device here, to accelerate inter-process or
inter-containers communication within the same OS instance.
Hello Wen Gu,
thank you very much for this patchset. I have been looking at it a bit.
I installed in on a testserver, but did not yet excercise the loopback-ism device.
I want to continue investigations, but daily work interferes, so I thought I
send you some comments now. So this is not at all a code review, but some
thoughts and observations about the general concept.
Thank you very much, Sandy.
In [1] there was a discussion about an abstraction layer between smc-d and the
ism devices.
I am not sure what you are proposing now, is it an smc-d feature or independent of smc?
In 3/15 you say it is part of the SMC module, but then it has its own device entry.
Didn't you want to use it for other things as well? Or is it an SMC-D only feature?
Is it a device (Config help: "kind of virtual device")? Or an SMC-D feature?
This patchset aims to propose an SMC feature, which is SMC-D loopback. The main work
to achieve this feature is to implement an Emulated-ISM, which is loopback-ism. The
loopback-ism is a 'built-in' dummy device of SMC and only serves SMC.
SMC-D protocol + 'built-in dummy device' (loopback-ism device) = SMC-D loopback feature.
To provide the runtime switch and statistics of loopback-ism, I need to find a sysfs
entry for it, since it doesn't belong to any class (e.g. pci_bus), I created an 'smc'
entry under /sys/devices/virtual/ and put loopback-ism under it.
The other SMC devices, such as RoCE, s390 ISM, virtio-ism will be in their own sysfs
entry, not under the /sys/devices/*virtual*/smc/.
The Config help is somewhat inaccurate. To be more precise, the SMC_LO config is used to
configure whether to enable this built-in dummy device for intra-OS communication.
Will we have a class of ism devices (s390 ism, ism-loopback, virtio-ism)
That share common properties (internal API?)
and smc-d will work with any of those? > But they all can exist without smc ?! BTW: This is what we want for s390-ism.
The client-registration interface [2] is currently the way to achieve this.
But maybe we need a more general concept?
I didn't mean to create a class to cover all the ISM devices. It is only for
loopback-ism. Because loopback-ism can not be classified, so I create an entry
under /sys/devices/virtual/.
Maybe first a preparation patchset that introduces a class/ism
Together with an improved API?
In case you want to use ISM devices for other purposes as well..
But then the whole picture of ism-loopback in one patchset (RFC?)
has its benefits as well.
Sorry for causing, I didn't mean to create a class to cover all the ISM devices.
They should be in their own sysfs entries (e.g. pci_bus), since they will be used
out of SMC. Only loopback-ism belongs only to SMC.
Other points that I noticed:
Naming: smc loopback, ism-loopback, loopback-ism ?
config: why not tristate? Why under net/smc?
'SMC-D loopback' or 'SMC loopback' is used to indicate the feature or capability.
'loopback-ism' is the emulated-ISM device that 'SMC/SMC-D loopback' used.
('ism-loopback' doesn't seem to appear in my patchset)
If we all agree with these, I will check all the terms in the patch and unify them.
SMC_LO is used to configure whether SMC is allowed to use loopback-ism (CONFIG_SMC_LO),
it acts as a check in the code, so I defined it as a bool.
And loopback-ism only serves SMC-D loopback, as a feature of SMC, so the implementation
(net/smc/smc_loopback.{c|h}) is under net/smc.
/sys/devices/virtual/smc does not initially show up in my installation!!!
root@t35lp50:/sys/devices/virtual/> ls
3270/ bdi/ block/ graphics/ iommu/ mem/ memory_tiering/ misc/ net/ tty/ vc/ vtconsole/ workqueue/
root@t35lp50:/sys/devices/virtual/> ls smc/loopback-ism
active dmb_copy dmbs_cnt dmb_type subsystem@ uevent xfer_bytes
root@t35lp50:/sys/devices/virtual/> ls
3270/ bdi/ block/ graphics/ iommu/ mem/ memory_tiering/ misc/ net/ smc/ tty/ vc/ vtconsole/ workqueue/
Is that normal behaviour?
/sys/devices/virtual/smc is created after SMC module initialization.
During the SMC module initialization, smc_loopback_init() is called, and the
/sys/devices/virtual/smc entry is created.
You introduced a class/smc
Maybe class/ism would be better?
The other smc interfaces do not show up in class/smc!! Not so good
Sorry for causing, I didn't mean to create a class to cover all the ISM devices.
They should be in their own sysfs entries (e.g. pci_bus), since they can be used
out of SMC. But loopback-ism is a SMC 'built-in' dummy device, it belongs only
to SMC and can't be classified to other entries.
Why doesn't it show in smc_rnics?
(Maybe some deficiency of smc_rnics?)
smc_rnics can't be used on the arch other than s390.
# ./smc_rnics -a
Error: s390/s390x supported only
But then it shows in other smc-tools:
root@t35lp50:/sys/> smcd device
FID Type PCI-ID PCHID InUse #LGs PNET-ID
0000 0 loopback-ism ffff No 0
0029 ISM 0000:00:00.0 07c1 No 0 NET1
Nice!
Yes, this is did on patch 01/15.
Best regards,
Wen Gu
Kind regards
Sandy
[1] https://lore.kernel.org/lkml/e3819550-7b10-4f9c-7347-dcf1f97b8e6b@xxxxxxxxxxxxxxxxx/
[2] 89e7d2ba61b7 ("net/ism: Add new API for client registration")
