On 12/24/2017 3:47 PM, Or Gerlitz wrote: > On Sun, Dec 24, 2017 at 2:57 PM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: > >> From Daniel: >> >> This feature allows RDMA resources (pd, mr, cq, qp, etc) to be used with >> both physical ports of capable mlx5 devices. When enabled a single IB >> device with two ports will be registered instead of two single port >> devices. >> >> There are still two PCI devices underlying the two port device, the >> capabilities indicate which device is the "master" device and which is >> the slave. >> >> When the add callback function is called for a slave device a list of IB >> devices is searched for matching master device, indicated by the >> capabilities >> and the system_image_guid. If a match is found the slave is bound to the >> master device, otherwise it's placed on a list, in case it's master becomes >> available in the future. When a master device is added it searches the list >> of available slaves for a matching slave device. If a match is found it >> binds >> the slave as its 2nd port. If no match as found the device still appears >> as a dual port device, with the 2nd port down. RDMA resources can still >> created that use the yet unavailable 2nd port. >> >> Commands related to IB resources are all routed through the master >> mlx5_core >> device. Port specific commands, like those for hardware counters are >> routed to >> their respective port mlx5_core device. Since devices can appear and >> disappear >> asynchronously a reference count on the underlying mlx5_core device is >> maintained. Getting and putting this reference is only necessary for >> commands >> destined to a specific port, the master core device can be used freely, >> as it will exist while the IB device exists. >> >> > Daniel, so far you only talked on IB devices and a master/slave relation > you introduce for the > core devices > > > >> SR-IOV devices follow the same pattern as the physical ones. VFs of a >> master >> port can bind VFs of slave ports, if available, and operate as dual >> port devices. >> > Not following. > > We still have two mlx5 EN devices, right?! and from libvirt point of view > business are just as usual, Yes, 2 netdevs. > a VM will be assigned with one VF from any of the core devices (master > only, slave only or both) > and this will be an IB device with one port - correct? No. If the PCI device passed to the VM is a master the VM will show an IB device with 2 ports, one of them down. If the PCI device were a slave there would be no IB device for it in the VM. If the VM is passed one master and one slave then there will be a 2 port IB device with both ports available. > Note that most of us are ooo till early January (== early next week), so > should be able to follow on your proposal then. > > Or. > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
