On Wed, Nov 13, 2024 at 12:20:20PM +0000, Shijith Thotton wrote: > >> >> This patch introduces a PCI hotplug controller driver for the OCTEON > >> >> PCIe device. The OCTEON PCIe device is a multi-function device where the > >> >> first function serves as the PCI hotplug controller. > >> >> > >> >> +--------------------------------+ > >> >> | Root Port | > >> >> +--------------------------------+ > >> >> | > >> >> PCIe > >> >> | > >> >> +---------------------------------------------------------------+ > >> >> | OCTEON PCIe Multifunction Device | > >> >> +---------------------------------------------------------------+ > >> >> | | | | > >> >> | | | | > >> >> +---------------------+ +----------------+ +-----+ +----------------+ > >> >> | Function 0 | | Function 1 | | ... | | Function 7 | > >> >> | (Hotplug controller)| | (Hotplug slot) | | | | (Hotplug slot) | > >> >> +---------------------+ +----------------+ +-----+ +----------------+ > >> >> | > >> >> | > >> >> +-------------------------+ > >> >> | Controller Firmware | > >> >> +-------------------------+ > >> >> > >> >> The hotplug controller driver enables hotplugging of non-controller > >> >> functions within the same device. During probing, the driver removes > >> >> the non-controller functions and registers them as PCI hotplug slots. > >> >> These slots are added back by the driver, only upon request from the > >> >> device firmware. > >> >> > >> >> The controller uses MSI-X interrupts to notify the host of hotplug > >> >> events initiated by the OCTEON firmware. Additionally, the driver > >> >> allows users to enable or disable individual functions via sysfs slot > >> >> entries, as provided by the PCI hotplug framework. > >> > > >> >Can we say something here about what the benefit of this driver is? > >> >For example, does it save power? > >> > >> The driver enables hotplugging of non-controller functions within the device > >> without requiring a fully implemented switch, reducing both power > >consumption > >> and product cost. > > > >Reduced product cost is motivation for the hardware design, not for > >this hotplug driver. > > > >You didn't explicitly say that when function 0 hot-removes another > >function, it reduces overall power consumption. But I assume that's > >the case? > > > > Yes, I will explain it in detail below > > >> >What causes the function 0 firmware to request a hot-add or > >> >hot-removal of another function? > >> > >> The firmware will enable the required number of non-controller > >> functions based on runtime demand, allowing control over these > >> functions. For example, in a vDPA scenario, each function could act > >> as a different type of device (such as net, crypto, or storage) > >> depending on the firmware configuration. > > > >What is the path for this runtime demand? I assume function 0 > >provides some interface to request a specific kind of functionality > >(net, crypo, storage, etc)? > > > > Right now, it done via firmware management console. > > >I don't know anything about vDPA, so if that's important here, it > >needs a little more context. > > > >> Hot removal is useful in cases of live firmware updates. > > > >So the idea is that function X is hot-removed, which forces the driver > >to let go of it, the firmware is updated, and X is hot-added again, > >and the driver binds to it again? > > > > I will explain the process in detail, which should also address the questions > below. > > >And somewhere in there is a reset of function X, and after the reset > >X is running the new firmware? > > > >Who/what initiates this whole path? Some request to function 0, > >saying "please remove function X"? > > > >But I guess maybe it doesn't go through function 0, since octeon_hp > >claims function 0, and it doesn't provide that functionality. Maybe > >the individual drivers for *other* functions know how to initiate > >these things, and those functions internally communicate with function > >0 to ask it to start a hot-remove/hot-add sequence? > > > >That wouldn't explain the power reduction plan, though. A driver for > >function X could conceivably tell its device "I'm no longer needed" > >and function X could tell function 0 to remove it. That might enable > >some power savings. But that doesn't have a path to *re-enable* > >function X, since function X has been removed and there's no driver to > >ask for it to be hot-added again. > > > >Maybe there's some out-of-band management path that can tell function > >0 to do things, independent of PCIe? > > > > Our implementation aims to achieve two main objectives: > > 1. Enable changing a function's personality at runtime. > 2. Reduce power consumption. > > The OCTEON PCI device has multiple ARM cores running Linux, with its firmware > composed of multiple components. For example, the firmware includes components > like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any function > at runtime. The device firmware is accessible via a management console, allowing > components to be started or stopped. For each component, an associated function > is hot-added on the host to expose its functionality. Initially, after boot, only > Function 0 and the controller firmware are active. > > Here's a breakdown: > > At Time 0: > - Linux boots on the device, starting the controller firmware. > > At Time 1: > - The hotplug driver loads on the host, temporarily removing other functions. > > At Time 2: > - A network device firmware component starts on an ARM core (initiated through > a console command). > - This component sets up the Function 1 configuration space, data, and other > request handlers for network processing. > - The firmware issues a hot-add request to Function 0 (hotplug driver) on the > host to enable Function 1. > > At Time 3: > - The Function 0 hotplug driver on the host receives the hot-add request and > enables Function 1 on the host. > - A network driver binds to Function 1 based on device class and ID. > > At Time 4: > - The network device firmware component receives a stop signal. > - The firmware issues a hot-remove request for Function 1 on the host. > - The firmware component halts, reducing the device's power consumption. > > At Time 5: > - The Function 0 hotplug driver on the host receives the hot-remove request and > disables Function 1 on the host. > > At Time 6: > - A crypto device firmware component starts on an ARM core. > - This component configures the Function 1 configuration space for crypto > processing and sets up the required firmware handlers. > - The firmware issues a hot-add request to enable Function 1 on the host. > > At Time 7: > - The Function 0 hotplug driver on the host receives the hot-add request and enables Function 1 on the host. > - A crypto driver binds to Function 1 based on device class and ID. > > The firmware component for each function only runs and is hot-added when > needed. Only Function 0 and the controller firmware remain active > continuously. This dynamic control reduces power usage by keeping unnecessary > components off. Additionally, a single function can adapt its personality based > on the associated firmware component, enhancing flexibility. > > I hope this clarifies the implementation. Let me know if you have any > questions. Thanks very much! I propose adding text like this to the commit log: There is an out-of-band management console interface to firmware running on function 0 whereby an administrator can disable functions to save power or enable them with one of several personalities (virtio-net, virtio-crypto, NVMe, etc) for the other functions. Function 0 initiates hotplug events handled by this driver when the other functions are enabled or disabled. I provisionally applied this to pci/hotplug-octeon, but will be happy to update the text if necessary. Bjorn
