On Tue, Jul 06, 2021 at 02:07:16PM +0200, Daniel Vetter wrote: > On the "rdma-core" idea, afaik rdma NIC do not have fully programmable > cores in their hw, for which you'd need some kind of compiler to make > use of the hardware and the interfaces the kernel provides? So not > really compareable, but also my understanding is that rdma-core does > actually allow you to reasonable use&drive all the hw features and > kernel interfaces fully. The whole HPC stack has speciality compilers of course. OpenMP, PGAS, etc. These compilers map onto library primitives that eventually boil down into rdma-core calls. Even the HW devices have various programmability that are being targetted with compilers now. People are making NIC devices with ARM cores/etc - P4 is emerging for some packet processing tasks. rdma-core can drive all the kernel interfaces with at least an ioctl wrapper, and it has a test suite that tries to cover this. It does not exercise the full HW capability, programmability, etc of every single device. I actually don't entirely know what everyone has built on top of rdma-core, or how I'd try to map it the DRI ideas you are trying to explain. Should we ban all Intel RDMA drivers because they are shipping proprietary Intel HPC compilers and proprietary Intel MPI which drives their RDMA HW? Or is that OK because there are open analogs for some of that stuff? And yes, the open versions are inferior in various metrics. Pragmatically what I want to see is enough RDMA common/open user space to understand the uAPI and thus more about how the kernel driver works. Forcing everyone into rdma-core has already prevented a number of uAPI mistakes in drivers that would have been bad - so at least this level really is valuable. > So we actually want less on dri-devel, because for compute/accel chips > we're currently happy with a vendor userspace. It just needs to be > functional and complete, and open in its entirety. In a sense yes: DRI doesn't insist on a single code base to act as the kernel interface, but that is actually the thing that has brought the most value to RDMA, IMHO. We've certainly had some interesting successes because of this. The first submission for AWS's EFA driver proposed to skip the rdma-core step, which was rejected. However since EFA has been in that ecosystem it has benefited greatly, I think. However, in another sense no: RDMA hasn't been blocking, say Intel, just because they have built proprietary stuff on top of our open stack. Honestly, I think GPU is approaching this backwards. Wayland should have been designed to prevent proprietary userspace stacks. Jason
