On 9/13/22 10:41, Frank Rowand wrote:
On 9/13/22 12:10, Lizhi Hou wrote:
On 9/13/22 00:00, Frank Rowand wrote:
On 8/29/22 16:43, Lizhi Hou wrote:
This patch series introduces OF overlay support for PCI devices which
primarily addresses two use cases. First, it provides a data driven method
to describe hardware peripherals that are present in a PCI endpoint and
hence can be accessed by the PCI host. An example device is Xilinx/AMD
Alveo PCIe accelerators. Second, it allows reuse of a OF compatible
driver -- often used in SoC platforms -- in a PCI host based system. An
example device is Microchip LAN9662 Ethernet Controller.
This patch series consolidates previous efforts to define such an
infrastructure:
https://lore.kernel.org/lkml/20220305052304.726050-1-lizhi.hou@xxxxxxxxxx/
https://lore.kernel.org/lkml/20220427094502.456111-1-clement.leger@xxxxxxxxxxx/
Normally, the PCI core discovers PCI devices and their BARs using the
PCI enumeration process. However, the process does not provide a way to
discover the hardware peripherals that are present in a PCI device, and
which can be accessed through the PCI BARs. Also, the enumeration process
does not provide a way to associate MSI-X vectors of a PCI device with the
hardware peripherals that are present in the device. PCI device drivers
often use header files to describe the hardware peripherals and their
resources as there is no standard data driven way to do so. This patch
series proposes to use flattened device tree blob to describe the
peripherals in a data driven way. Based on previous discussion, using
device tree overlay is the best way to unflatten the blob and populate
platform devices. To use device tree overlay, there are three obvious
problems that need to be resolved.
First, we need to create a base tree for non-DT system such as x86_64. A
patch series has been submitted for this:
https://lore.kernel.org/lkml/20220624034327.2542112-1-frowand.list@xxxxxxxxx/
https://lore.kernel.org/lkml/20220216050056.311496-1-lizhi.hou@xxxxxxxxxx/
Second, a device tree node corresponding to the PCI endpoint is required
for overlaying the flattened device tree blob for that PCI endpoint.
Because PCI is a self-discoverable bus, a device tree node is usually not
created for PCI devices. This series adds support to generate a device
tree node for a PCI device which advertises itself using PCI quirks
infrastructure.
Third, we need to generate device tree nodes for PCI bridges since a child
PCI endpoint may choose to have a device tree node created.
This patch series is made up of two patches.
The first patch is adding OF interface to allocate an OF node. It is copied
from:
https://lore.kernel.org/lkml/20220620104123.341054-5-clement.leger@xxxxxxxxxxx/
The second patch introduces a kernel option, CONFIG_PCI_OF. When the option
is turned on, the kernel will generate device tree nodes for all PCI
bridges unconditionally. The patch also shows how to use the PCI quirks
infrastructure, DECLARE_PCI_FIXUP_FINAL to generate a device tree node for
a device. Specifically, the patch generates a device tree node for Xilinx
Alveo U50 PCIe accelerator device. The generated device tree nodes do not
have any property. Future patches will add the necessary properties.
Clément Léger (1):
of: dynamic: add of_node_alloc()
Lizhi Hou (1):
pci: create device tree node for selected devices
drivers/of/dynamic.c | 50 +++++++++++++----
drivers/pci/Kconfig | 11 ++++
drivers/pci/bus.c | 2 +
drivers/pci/msi/irqdomain.c | 6 +-
drivers/pci/of.c | 106 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci-driver.c | 3 +-
drivers/pci/pci.h | 16 ++++++
drivers/pci/quirks.c | 11 ++++
drivers/pci/remove.c | 1 +
include/linux/of.h | 7 +++
10 files changed, 200 insertions(+), 13 deletions(-)
The patch description leaves out the most important piece of information.
The device located at the PCI endpoint is implemented via FPGA
- which is programmed after Linux boots (or somewhere late in the boot process)
- (A) and thus can not be described by static data available pre-boot because
it is dynamic (and the FPGA program will often change while the Linux
kernel is already booted
- (B) can be described by static data available pre-boot because the FPGA
program will always be the same for this device on this system
I am not positive what part of what I wrote above is correct and would appreciate
some confirmation of what is correct or incorrect.
There are 2 series devices rely on this patch:
1) Xilinx Alveo Accelerator cards (FPGA based device)
2) lan9662 PCIe card
please see: https://lore.kernel.org/lkml/20220427094502.456111-1-clement.leger@xxxxxxxxxxx/
Thanks. Please include this information in future versions of the patch series.
For device 2 I have strongly recommended using pre-boot apply of the overlay to the base
device tree. I realize that this suggestion is only a partial solution if one wants to
use hotplug to change system configuration (as opposed to using hotplug only to replace
an existing device (eg a broken device) with another instance of the same device). I
also realize that this increased the system administration overhead. On the other hand
an overlay based solution is likely to be fragile and possibly flaky.
Can you clarify the pre-boot apply approach? How will it work for PCI
devices?
For Xilinx Alveo device, it is (A). The FPGA partitions can be programmed dynamically after boot.
I looked at the Xilinx Alveo web page, and there are a variety of types of Alveo cards
available. So the answer to my next question may vary by type of card.
Is it expected that the fpga program on a given card will change frequently (eg multiple
times per day), where the changed program results in a new device that would require a
different hardware description in the device tree?
Different images may be loaded to a FPGA partition several times a day.
The PCI topology (Device IDs, BARs, MSIx, etc) does not change. New IPs
may appear (and old IPs may disappear) on the BARs when a new image is
loaded. We would like to use flattened device tree to describe the IPs
on the BARs.
Thanks,
Lizhi
Or is the fpga program expected to change on an infrequent basis (eg monthly, quarterly,
annually), in the same way as device firmware and operating systems are updated on a regular
basis for bug fixes and new functionality?
Thanks,
Lzhi
-Frank
