This is my attempt to migrate the module loading functions from RedHat's
rdma.service into common code suitable for all distros.
The approach is completly hotplug safe and relies heavily on udev and systemd
to do the actual decision making and work, there are no shell scripts involved.
I do not have nearly enough hardware to exhaustively test this, I have checked
that the basic elements are working, but there is lots of room for little
errors, particularly in module names and udev matching. It would be great of
someone could try this on iwarp hardware and confirm all the autoloading works
as expected.
As should the active roce vendors. I expect that all roce hardware will
autoload the RDMA modules with this approach.
Compared to redhat/rdma.kernel-init:
- Config of what ULP modules to load is split into files in
/etc/rdma/modules/*.conf, each conf reflects a certain configuration. Eg
infiniband.conf is loaded when IB hardware is present, iwarp.conf is loaded
for iWarp, etc.
This allows each techology to load appropriate modules automatically, ie
roce hardware will not autoload ipoib out of the box
Users can configure these files to turn things on/off as appropriate, or just
use the usual /etc/modules-load.d/ mechanism. systemd-modules-load is used
to load the modules lists so behavior is identical to the standard loader.
eg blacklists work, etc.
This obsoletes the /etc/rdma/rdma.conf file
- Load of HW support drivers (eg the RDMA component of an ethernet NIC) is
done completely inside udev using the udev built in module load, based on
the hotplug event of the ethernet nic.
This means RDMA support modules are only loaded if the rdma-core package is
installed and only if the actual base ethernet hardware is present. (rather
than triggering on loaded modules)
- More roce hardware is supported now, I filled in entries for all of our
new providers (eg qedr).
- srp_daemon and iwpmd support kernel modules autoload automatically as part of
the .service file, using systemd dependencies. If those packages are not
installed/running then those modules do not load.
- iwpmd now automatically activates if iwarp hardware is present, systemd does
not run it automatically.
- None of the obsolete mtrr or pci config meddling is included
- Nothing to do with sriov
- udev has more information now about rdma devices:
$ udevadm info /sys/class/infiniband/mlx4_0
P: /devices/pci0000:00/0000:00:04.0/infiniband/mlx4_0
E: DEVPATH=/devices/pci0000:00/0000:00:04.0/infiniband/mlx4_0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=MT27500 Family [ConnectX-3]
E: ID_MODEL_ID=0x1003
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Network controller
E: ID_RDMA_INFINIBAND=1
E: ID_RDMA_ROCE=1
E: ID_VENDOR_FROM_DATABASE=Mellanox Technologies
E: ID_VENDOR_ID=0x15b3
E: NAME=mlx4_0
E: SUBSYSTEM=infiniband
E: SYSTEMD_WANTS=rdma-ndd.service rdma-load-modules@rdma.service rdma-load-modules@infiniband.service rdma-load-modules@roce.service
E: TAGS=:systemd:
E: USEC_INITIALIZED=7954413272
A successful test should show appropriate ID_RDMA_* entries.
I did not go ahead and remove now duplicate functionality from the
redhat/scripts, I think that Jarod should be involved in that. This approach
is not incompatible with the scripts in redhat/, but modules loads will be
attempted twice if both systems are present.
https://github.com/jgunthorpe/rdma-plumbing/tree/systemd
Jason Gunthorpe (5):
Common infrastructure for auto loading rdma modules
srp: Autoload the SRP kernel module if required
iwpmd: Autoload iwpmd if required
redhat: Remove cxgb3/4.modprobe files
Move umad udev rules to rdma-core
CMakeLists.txt | 1 +
Documentation/udev.md | 83 ++++++++++++++++++++++
debian/control | 2 +-
debian/iwpmd.install | 2 +
debian/libibumad3.install | 1 -
debian/rdma-core.install | 10 +++
debian/srptools.install | 1 +
iwpmd/CMakeLists.txt | 8 +++
iwpmd/iwpmd.rules | 1 +
iwpmd/iwpmd.service.in | 6 +-
iwpmd/modules-iwpmd.conf | 2 +
kernel-boot/CMakeLists.txt | 28 ++++++++
kernel-boot/modules/infiniband.conf | 12 ++++
kernel-boot/modules/iwarp.conf | 1 +
kernel-boot/modules/opa.conf | 10 +++
kernel-boot/modules/rdma.conf | 21 ++++++
kernel-boot/modules/roce.conf | 2 +
kernel-boot/rdma-description.rules | 41 +++++++++++
kernel-boot/rdma-hw-modules.rules | 39 ++++++++++
kernel-boot/rdma-load-modules@xxxxxxxxxxx | 15 ++++
kernel-boot/rdma-ulp-modules.rules | 11 +++
.../rdma-umad.rules | 0
libibumad/CMakeLists.txt | 5 --
redhat/rdma-core.spec | 20 ++++--
redhat/rdma.cxgb3.sys.modprobe | 1 -
redhat/rdma.cxgb4.sys.modprobe | 1 -
srp_daemon/CMakeLists.txt | 4 ++
srp_daemon/modules-srp_daemon.conf | 2 +
srp_daemon/srp_daemon.service.in | 2 +-
srp_daemon/srp_daemon_port@xxxxxxxxxxx | 3 +-
30 files changed, 315 insertions(+), 20 deletions(-)
create mode 100644 Documentation/udev.md
create mode 100644 iwpmd/iwpmd.rules
create mode 100644 iwpmd/modules-iwpmd.conf
create mode 100644 kernel-boot/CMakeLists.txt
create mode 100644 kernel-boot/modules/infiniband.conf
create mode 100644 kernel-boot/modules/iwarp.conf
create mode 100644 kernel-boot/modules/opa.conf
create mode 100644 kernel-boot/modules/rdma.conf
create mode 100644 kernel-boot/modules/roce.conf
create mode 100644 kernel-boot/rdma-description.rules
create mode 100644 kernel-boot/rdma-hw-modules.rules
create mode 100644 kernel-boot/rdma-load-modules@xxxxxxxxxxx
create mode 100644 kernel-boot/rdma-ulp-modules.rules
rename libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules (100%)
delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
delete mode 100644 redhat/rdma.cxgb4.sys.modprobe
create mode 100644 srp_daemon/modules-srp_daemon.conf
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
