>> >> >> This patch introduces a PCI hotplug controller driver for the OCTEON >> >> >> PCIe device. The OCTEON PCIe device is a multi-function device where >the >> >> >> first function serves as the PCI hotplug controller. >> >> >> >> >> >> +--------------------------------+ >> >> >> | Root Port | >> >> >> +--------------------------------+ >> >> >> | >> >> >> PCIe >> >> >> | >> >> >> +---------------------------------------------------------------+ >> >> >> | OCTEON PCIe Multifunction Device | >> >> >> +---------------------------------------------------------------+ >> >> >> | | | | >> >> >> | | | | >> >> >> +---------------------+ +----------------+ +-----+ +----------------+ >> >> >> | Function 0 | | Function 1 | | ... | | Function 7 | >> >> >> | (Hotplug controller)| | (Hotplug slot) | | | | (Hotplug slot) | >> >> >> +---------------------+ +----------------+ +-----+ +----------------+ >> >> >> | >> >> >> | >> >> >> +-------------------------+ >> >> >> | Controller Firmware | >> >> >> +-------------------------+ >> >> >> >> >> >> The hotplug controller driver enables hotplugging of non-controller >> >> >> functions within the same device. During probing, the driver removes >> >> >> the non-controller functions and registers them as PCI hotplug slots. >> >> >> These slots are added back by the driver, only upon request from the >> >> >> device firmware. >> >> >> >> >> >> The controller uses MSI-X interrupts to notify the host of hotplug >> >> >> events initiated by the OCTEON firmware. Additionally, the driver >> >> >> allows users to enable or disable individual functions via sysfs slot >> >> >> entries, as provided by the PCI hotplug framework. >> >> > >> >> >Can we say something here about what the benefit of this driver is? >> >> >For example, does it save power? >> >> >> >> The driver enables hotplugging of non-controller functions within the >device >> >> without requiring a fully implemented switch, reducing both power >> >consumption >> >> and product cost. >> > >> >Reduced product cost is motivation for the hardware design, not for >> >this hotplug driver. >> > >> >You didn't explicitly say that when function 0 hot-removes another >> >function, it reduces overall power consumption. But I assume that's >> >the case? >> > >> >> Yes, I will explain it in detail below >> >> >> >What causes the function 0 firmware to request a hot-add or >> >> >hot-removal of another function? >> >> >> >> The firmware will enable the required number of non-controller >> >> functions based on runtime demand, allowing control over these >> >> functions. For example, in a vDPA scenario, each function could act >> >> as a different type of device (such as net, crypto, or storage) >> >> depending on the firmware configuration. >> > >> >What is the path for this runtime demand? I assume function 0 >> >provides some interface to request a specific kind of functionality >> >(net, crypo, storage, etc)? >> > >> >> Right now, it done via firmware management console. >> >> >I don't know anything about vDPA, so if that's important here, it >> >needs a little more context. >> > >> >> Hot removal is useful in cases of live firmware updates. >> > >> >So the idea is that function X is hot-removed, which forces the driver >> >to let go of it, the firmware is updated, and X is hot-added again, >> >and the driver binds to it again? >> > >> >> I will explain the process in detail, which should also address the questions >> below. >> >> >And somewhere in there is a reset of function X, and after the reset >> >X is running the new firmware? >> > >> >Who/what initiates this whole path? Some request to function 0, >> >saying "please remove function X"? >> > >> >But I guess maybe it doesn't go through function 0, since octeon_hp >> >claims function 0, and it doesn't provide that functionality. Maybe >> >the individual drivers for *other* functions know how to initiate >> >these things, and those functions internally communicate with function >> >0 to ask it to start a hot-remove/hot-add sequence? >> > >> >That wouldn't explain the power reduction plan, though. A driver for >> >function X could conceivably tell its device "I'm no longer needed" >> >and function X could tell function 0 to remove it. That might enable >> >some power savings. But that doesn't have a path to *re-enable* >> >function X, since function X has been removed and there's no driver to >> >ask for it to be hot-added again. >> > >> >Maybe there's some out-of-band management path that can tell function >> >0 to do things, independent of PCIe? >> > >> >> Our implementation aims to achieve two main objectives: >> >> 1. Enable changing a function's personality at runtime. >> 2. Reduce power consumption. >> >> The OCTEON PCI device has multiple ARM cores running Linux, with its >firmware >> composed of multiple components. For example, the firmware includes >components >> like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any >function >> at runtime. The device firmware is accessible via a management console, >allowing >> components to be started or stopped. For each component, an associated >function >> is hot-added on the host to expose its functionality. Initially, after boot, only >> Function 0 and the controller firmware are active. >> >> Here's a breakdown: >> >> At Time 0: >> - Linux boots on the device, starting the controller firmware. >> >> At Time 1: >> - The hotplug driver loads on the host, temporarily removing other functions. >> >> At Time 2: >> - A network device firmware component starts on an ARM core (initiated >through >> a console command). >> - This component sets up the Function 1 configuration space, data, and other >> request handlers for network processing. >> - The firmware issues a hot-add request to Function 0 (hotplug driver) on the >> host to enable Function 1. >> >> At Time 3: >> - The Function 0 hotplug driver on the host receives the hot-add request and >> enables Function 1 on the host. >> - A network driver binds to Function 1 based on device class and ID. >> >> At Time 4: >> - The network device firmware component receives a stop signal. >> - The firmware issues a hot-remove request for Function 1 on the host. >> - The firmware component halts, reducing the device's power consumption. >> >> At Time 5: >> - The Function 0 hotplug driver on the host receives the hot-remove request >and >> disables Function 1 on the host. >> >> At Time 6: >> - A crypto device firmware component starts on an ARM core. >> - This component configures the Function 1 configuration space for crypto >> processing and sets up the required firmware handlers. >> - The firmware issues a hot-add request to enable Function 1 on the host. >> >> At Time 7: >> - The Function 0 hotplug driver on the host receives the hot-add request and >enables Function 1 on the host. >> - A crypto driver binds to Function 1 based on device class and ID. >> >> The firmware component for each function only runs and is hot-added when >> needed. Only Function 0 and the controller firmware remain active >> continuously. This dynamic control reduces power usage by keeping >unnecessary >> components off. Additionally, a single function can adapt its personality based >> on the associated firmware component, enhancing flexibility. >> >> I hope this clarifies the implementation. Let me know if you have any >> questions. > >Thanks very much! I propose adding text like this to the commit log: > > There is an out-of-band management console interface to firmware > running on function 0 whereby an administrator can disable functions > to save power or enable them with one of several personalities > (virtio-net, virtio-crypto, NVMe, etc) for the other functions. > Function 0 initiates hotplug events handled by this driver when the > other functions are enabled or disabled. > >I provisionally applied this to pci/hotplug-octeon, but will be happy >to update the text if necessary. > Thank you. The commit log looks good. I hope no further action is required from my side. Shijith
