On Fri, Jan 17, 2025 at 05:57:10PM +0100, Niklas Schnelle wrote: > On Fri, 2025-01-17 at 17:33 +0100, Andrew Lunn wrote: > > > Conceptually kind of but the existing s390 specific ISM device is a bit > > > special. But let me start with some background. On s390 aka Mainframes > > > OSs including Linux runs in so called logical partitions (LPARs) which > > > are machine hypervisor VMs which use partitioned non-paging memory. The > > > fact that memory is partitioned is important because this means LPARs > > > can not share physical memory by mapping it. > > > > > > Now at a high level an ISM device allows communication between two such > > > Linux LPARs on the same machine. The device is discovered as a PCI > > > device and allows Linux to take a buffer called a DMB map that in the > > > IOMMU and generate a token specific to another LPAR which also sees an > > > ISM device sharing the same virtual channel identifier (VCHID). This > > > token can then be transferred out of band (e.g. as part of an extended > > > TCP handshake in SMC-D) to that other system. With the token the other > > > system can use its ISM device to securely (authenticated by the token, > > > LPAR identity and the IOMMU mapping) write into the original systems > > > DMB at throughput and latency similar to doing a memcpy() via a > > > syscall. > > > > > > On the implementation level the ISM device is actually a piece of > > > firmware and the write to a remote DMB is a special case of our PCI > > > Store Block instruction (no real MMIO on s390, instead there are > > > special instructions). Sadly there are a few more quirks but in > > > principle you can think of it as redirecting writes to a part of the > > > ISM PCI devices' BAR to the DMB in the peer system if that makes sense. > > > There's of course also a mechanism to cause an interrupt on the > > > receiver as the write completes. > > > > So the s390 details are interesting, but as you say, it is > > special. Ideally, all the special should be hidden away inside the > > driver. > > Yes and it will be. There are some exceptions e.g. for vfio-pci pass- > through but that's not unusual and why there is already the concept of > vfio-pci extension module. > > > > > So please take a step back. What is the abstract model? > > I think my high level description may be a good start. The abstract > model is the ability to share a memory buffer (DMB) for writing by a > communication partner, authenticated by a DMB Token. Plus stuff like > triggering an interrupt on write or explicit trigger. Then Alibaba > added optional support for what they called attaching the buffer which > means it becomes truly shared between the peers but which IBM's ISM > can't support. Plus a few more optional pieces such as VLANs, PNETIDs > don't ask. The idea for the new layer then is to define this interface > with operations and documentation. > > > > > Can the abstract model be mapped onto CLX? Could it be used with a GPU > > vRAM? SoC with real shared memory between a pool of CPUs. > > > > Andrew > > I'd think that yes, one could implement such a mechanism on top of CXL > as well as on SoC. Or even with no special hardware between a host and > a DPU (e.g. via PCIe endpoint framework). Basically anything that can > DMA and IRQs between two OS instances. Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA. Maybe take a look at subsystems like USB, I2C. usb_submit_urb(struct urb *urb, gfp_t mem_flags) An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device. i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num) *msgs points to num of messages which get transferred to/from the I2C device. Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct i2c_msg has some I2C concepts in it. A struct ism_msg would follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared? Andrew
