>> >> This patch introduces a PCI hotplug controller driver for the OCTEON
>> >> PCIe device. The OCTEON PCIe device is a multi-function device where the
>> >> first function serves as the PCI hotplug controller.
>> >>
>> >> +--------------------------------+
>> >> | Root Port |
>> >> +--------------------------------+
>> >> |
>> >> PCIe
>> >> |
>> >> +---------------------------------------------------------------+
>> >> | OCTEON PCIe Multifunction Device |
>> >> +---------------------------------------------------------------+
>> >> | | | |
>> >> | | | |
>> >> +---------------------+ +----------------+ +-----+ +----------------+
>> >> | Function 0 | | Function 1 | | ... | | Function 7 |
>> >> | (Hotplug controller)| | (Hotplug slot) | | | | (Hotplug slot) |
>> >> +---------------------+ +----------------+ +-----+ +----------------+
>> >> |
>> >> |
>> >> +-------------------------+
>> >> | Controller Firmware |
>> >> +-------------------------+
>> >>
>> >> The hotplug controller driver enables hotplugging of non-controller
>> >> functions within the same device. During probing, the driver removes
>> >> the non-controller functions and registers them as PCI hotplug slots.
>> >> These slots are added back by the driver, only upon request from the
>> >> device firmware.
>> >>
>> >> The controller uses MSI-X interrupts to notify the host of hotplug
>> >> events initiated by the OCTEON firmware. Additionally, the driver
>> >> allows users to enable or disable individual functions via sysfs slot
>> >> entries, as provided by the PCI hotplug framework.
>> >
>> >Can we say something here about what the benefit of this driver is?
>> >For example, does it save power?
>>
>> The driver enables hotplugging of non-controller functions within the device
>> without requiring a fully implemented switch, reducing both power
>consumption
>> and product cost.
>
>Reduced product cost is motivation for the hardware design, not for
>this hotplug driver.
>
>You didn't explicitly say that when function 0 hot-removes another
>function, it reduces overall power consumption. But I assume that's
>the case?
>
Yes, I will explain it in detail below
>> >What causes the function 0 firmware to request a hot-add or
>> >hot-removal of another function?
>>
>> The firmware will enable the required number of non-controller
>> functions based on runtime demand, allowing control over these
>> functions. For example, in a vDPA scenario, each function could act
>> as a different type of device (such as net, crypto, or storage)
>> depending on the firmware configuration.
>
>What is the path for this runtime demand? I assume function 0
>provides some interface to request a specific kind of functionality
>(net, crypo, storage, etc)?
>
Right now, it done via firmware management console.
>I don't know anything about vDPA, so if that's important here, it
>needs a little more context.
>
>> Hot removal is useful in cases of live firmware updates.
>
>So the idea is that function X is hot-removed, which forces the driver
>to let go of it, the firmware is updated, and X is hot-added again,
>and the driver binds to it again?
>
I will explain the process in detail, which should also address the questions
below.
>And somewhere in there is a reset of function X, and after the reset
>X is running the new firmware?
>
>Who/what initiates this whole path? Some request to function 0,
>saying "please remove function X"?
>
>But I guess maybe it doesn't go through function 0, since octeon_hp
>claims function 0, and it doesn't provide that functionality. Maybe
>the individual drivers for *other* functions know how to initiate
>these things, and those functions internally communicate with function
>0 to ask it to start a hot-remove/hot-add sequence?
>
>That wouldn't explain the power reduction plan, though. A driver for
>function X could conceivably tell its device "I'm no longer needed"
>and function X could tell function 0 to remove it. That might enable
>some power savings. But that doesn't have a path to *re-enable*
>function X, since function X has been removed and there's no driver to
>ask for it to be hot-added again.
>
>Maybe there's some out-of-band management path that can tell function
>0 to do things, independent of PCIe?
>
Our implementation aims to achieve two main objectives:
1. Enable changing a function's personality at runtime.
2. Reduce power consumption.
The OCTEON PCI device has multiple ARM cores running Linux, with its firmware
composed of multiple components. For example, the firmware includes components
like Virtio-net, NVMe, and Virtio-Crypto, which can be assigned to any function
at runtime. The device firmware is accessible via a management console, allowing
components to be started or stopped. For each component, an associated function
is hot-added on the host to expose its functionality. Initially, after boot, only
Function 0 and the controller firmware are active.
Here's a breakdown:
At Time 0:
- Linux boots on the device, starting the controller firmware.
At Time 1:
- The hotplug driver loads on the host, temporarily removing other functions.
At Time 2:
- A network device firmware component starts on an ARM core (initiated through
a console command).
- This component sets up the Function 1 configuration space, data, and other
request handlers for network processing.
- The firmware issues a hot-add request to Function 0 (hotplug driver) on the
host to enable Function 1.
At Time 3:
- The Function 0 hotplug driver on the host receives the hot-add request and
enables Function 1 on the host.
- A network driver binds to Function 1 based on device class and ID.
At Time 4:
- The network device firmware component receives a stop signal.
- The firmware issues a hot-remove request for Function 1 on the host.
- The firmware component halts, reducing the device's power consumption.
At Time 5:
- The Function 0 hotplug driver on the host receives the hot-remove request and
disables Function 1 on the host.
At Time 6:
- A crypto device firmware component starts on an ARM core.
- This component configures the Function 1 configuration space for crypto
processing and sets up the required firmware handlers.
- The firmware issues a hot-add request to enable Function 1 on the host.
At Time 7:
- The Function 0 hotplug driver on the host receives the hot-add request and enables Function 1 on the host.
- A crypto driver binds to Function 1 based on device class and ID.
The firmware component for each function only runs and is hot-added when
needed. Only Function 0 and the controller firmware remain active
continuously. This dynamic control reduces power usage by keeping unnecessary
components off. Additionally, a single function can adapt its personality based
on the associated firmware component, enhancing flexibility.
I hope this clarifies the implementation. Let me know if you have any
questions.
Thanks,
Shijith
