Hi Bjorn
Many thanks for your review.
在 2017/9/6 8:04, Bjorn Helgaas 写道:
On Wed, Aug 30, 2017 at 07:09:35PM +0800, Dongdong Liu wrote:
Current code is broken as calling pci_free_irq_vectors()
invalidates the IRQ numbers returned before by pci_irq_vectors();
so we need to move all the assignment of the Linux IRQ numbers at
the bottom of the function.
After removing and adding back the PCI root port device,
we see the PCIe port service drivers request irq failed.
What exactly is the connection between the root port removal/addition
and the request IRQ failure? The current code does:
pci_alloc_irq_vectors(dev, ...)
irqs[x] = pci_irq_vector(dev, entry)
pci_free_irq_vectors(dev)
pci_alloc_irq_vectors(dev, ...)
So I guess the problem is that the second pci_alloc_irq_vectors() call
gets different vectors than the first one?
Yes, you are right.
[ 477.055527] pcieport 0000:00:00.0: irq [33-64] for MSI
[ 477.122631] pcieport 0000:00:00.0: irq [23-25] for MSI
[ 477.149352] pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22
[ 477.193697] pcieport 0000:00:00.0: request AER IRQ 34 failed
[ 477.211684] aer: probe of 0000:00:00.0:pcie002 failed with error -22
How is this related to removal/addition?
-[0000:00]-+-00.0-[01-02]--+-00.0 Device 8086:10fb
| \-00.1 Device 8086:10fb
After we boot the os.
ixgbe 0000:01:00.0: irq 23-24.
ixgbe 0000:01:00.1: irq 25-26.
pcieport 0000:00:00.0: irq [27-29].
After we remove the rp(echo 1 > /sys/devices/pci0000:00/0000:00:00.0/remove).
irq 23-29 are free, but the number of irqs does not have 32 (23-29 ) .
After we rescan the rp (echo 1 > /sys/bus/pci/rescan)
[ 477.055527] pcieport 0000:00:00.0: irq [33-64] for MSI
[ 477.122631] pcieport 0000:00:00.0: irq [23-25] for MSI
I debug this, the detail information, please see the attchment boot.log.
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 48eadf4..888230d 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -382,9 +382,9 @@ int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
for_each_msi_entry(desc, dev) {
virq = desc->irq;
if (desc->nvec_used == 1)
- dev_dbg(dev, "irq %d for MSI\n", virq);
+ dev_info(dev, "irq %d for MSI\n", virq);
else
- dev_dbg(dev, "irq [%d-%d] for MSI\n",
+ dev_info(dev, "irq [%d-%d] for MSI\n",
virq, virq + desc->nvec_used - 1);
> Is there some cleanup we're missing during the removal?
I have not found this.
pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22
aer: probe of 0000:00:00.0:pcie002 failed with error -22
pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd-
PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
pciehp 0000:00:00.0:pcie004: Cannot get irq 20 for the hotplug controller
pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1)
dpc 0000:00:00.0:pcie010: request IRQ22 failed: -22
dpc: probe of 0000:00:00.0:pcie010 failed with error -22
Cc: <stable@xxxxxxxxxxxxxxx>
Fixes: 3674cc4 ("PCI/portdrv: Use pci_irq_alloc_vectors()")
Please use a 12-character SHA1.
Ok, I will change it as below.
Fixes: 3674cc49da9a ("PCI/portdrv: Use pci_irq_alloc_vectors()")
Signed-off-by: Dongdong Liu <liudongdong3@xxxxxxxxxx>
Signed-off-by: Gabriele Paoloni <gabriele.paoloni@xxxxxxxxxx>
---
v1->v2:
- Fix comments on PATCH v1.
- Simplify implementation.
---
drivers/pci/pcie/portdrv_core.c | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index 313a21d..89f4cf5 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -55,7 +55,8 @@ static void release_pcie_device(struct device *dev)
static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)
{
int nr_entries, entry, nvec = 0;
-
+ int i;
+ int idx[PCIE_PORT_DEVICE_MAXSERVICES];
/*
* Allocate as many entries as the port wants, so that we can check
* which of them will be useful. Moreover, if nr_entries is correctly
This is not so much a question about your patch, but about the whole
interrupt vector allocation strategy in pcie_port_enable_irq_vec().
The current code basically does this:
- allocate 32 vectors
- figure out vector used by PME and hotplug
- figure out vector used by AER
- figure out vector used by DPC
- free the 32 vectors we allocated
- allocate only as many vectors as we need
Correct, that is the pcie_port_enable_irq_vec() do now.
What's the point of allocating those 32 vectors up front? Why can't
we first look up the interrupt message numbers used by each service
(PME, hotplut, AER, DPC), figure out how many vectors we need,
allocate them, and then fill in the slots in the irqs[] table?
Agree.
For example, something like this:
if (mask & (PCIE_PORT_SERVICE_PME | PCIE_PORT_SERVICE_HP)) {
pcie_capability_read_word(dev, PCI_EXP_FLAGS, ®16);
pme_msg = (reg16 & PCI_EXP_FLAGS_IRQ) >> 9;
}
if (mask & PCIE_PORT_SERVICE_AER) {
pci_read_config_dword(dev, dev->aer_cap + PCI_ERR_ROOT_STATUS, ®32);
aer_msg = reg32 >> 27;
}
if (mask & PCIE_PORT_SERVICE_DPC) {
pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_DPC);
pci_read_config_word(dev, pos + PCI_EXP_DPC_CAP, ®16);
dpc_msg = reg16 & 0x1f;
}
nvec = max(pme_msg, max(aer_msg, dpc_msg)) + 1;
pci_alloc_irq_vectors(dev, nvec, nvec, PCI_IRQ_MSIX | PCI_IRQ_MSI);
if (mask & PCIE_PORT_SERVICE_PME)
irqs[PCIE_PORT_SERVICE_PME_SHIFT] = pci_irq_vector(dev, pme_msg);
if (mask & PCIE_PORT_SERVICE_HP)
irqs[PCIE_PORT_SERVICE_HP_SHIFT] = pci_irq_vector(dev, pme_msg);
if (mask & PCIE_PORT_SERVICE_AER)
irqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, aer_msg);
if (mask & PCIE_PORT_SERVICE_DPC)
irqs[PCIE_PORT_SERVICE_DPC_SHIFT] = pci_irq_vector(dev, dpc_msg);
We currently only look at each service once, and what I'm proposing
would require looking at each service twice (once to read the message
number from the hardware and again (after allocating the vectors) to
fill in the irqs[] table). But I think it would be simpler overall.
It looks good to me.
Thanks,
Dongdong
[ 0.000000] Booting Linux on physical CPU 0x30000
[ 0.000000] Linux version 4.13.0-rc7-dirty (l00290354@linux-ioko) (gcc version 4.9.3 20150211 (prerelease) (20150316)) #49 SMP PREEMPT Wed Sep 6 10:29:35 CST 2017
[ 0.000000] Boot CPU: AArch64 Processor [480fd010]
[ 0.000000] earlycon: pl11 at MMIO32 0x0000000094080000 (options '')
[ 0.000000] bootconsole [pl11] enabled
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: EFI v2.60 by EDK II
[ 0.000000] efi: ACPI 2.0=0x39ce0000 MEMATTR=0x3e8c4018
[ 0.000000] cma: Reserved 16 MiB at 0x000000003b400000
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x0000000039CE0000 000024 (v02 HISI )
[ 0.000000] ACPI: XSDT 0x0000000039CD0000 00007C (v01 HISI HIP08 00000000 01000013)
[ 0.000000] ACPI: FACP 0x0000000039C10000 000114 (v06 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: DSDT 0x0000000039BB0000 000971 (v02 HISI HIP08 00000000 INTL 20170119)
[ 0.000000] ACPI: GTDT 0x0000000039C00000 000098 (v02 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: DBG2 0x0000000039BF0000 00005A (v00 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: MCFG 0x0000000039BE0000 00003C (v01 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: SPCR 0x0000000039BD0000 000050 (v02 HISI HIP07 00000000 INTL 20151124)
[ 0.000000] ACPI: APIC 0x0000000039BC0000 0000C0 (v04 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: IORT 0x0000000039BA0000 000120 (v00 HISI D08 00000000 INTL 20170119)
[ 0.000000] ACPI: HEST 0x0000000039B80000 00013C (v01 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: EINJ 0x0000000039B60000 000170 (v01 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: ERST 0x0000000031A90000 000230 (v01 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: BERT 0x0000000031A80000 008030 (v01 HISI HIP08 00000000 INTL 20151124)
[ 0.000000] ACPI: SPCR: console: pl011,mmio32,0x94080000,115200
[ 0.000000] ACPI: NUMA: Failed to initialise from firmware
[ 0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000003fbfffff]
[ 0.000000] NUMA: NODE_DATA [mem 0x3fbee500-0x3fbeffff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000003fbfffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000031a5ffff]
[ 0.000000] node 0: [mem 0x0000000031a60000-0x0000000031a9ffff]
[ 0.000000] node 0: [mem 0x0000000031aa0000-0x0000000039b5ffff]
[ 0.000000] node 0: [mem 0x0000000039b60000-0x0000000039b6ffff]
[ 0.000000] node 0: [mem 0x0000000039b70000-0x0000000039b79fff]
[ 0.000000] node 0: [mem 0x0000000039b7a000-0x0000000039b8ffff]
[ 0.000000] node 0: [mem 0x0000000039b90000-0x0000000039b91fff]
[ 0.000000] node 0: [mem 0x0000000039b92000-0x000000003a0cffff]
[ 0.000000] node 0: [mem 0x000000003a0d0000-0x000000003a0d1fff]
[ 0.000000] node 0: [mem 0x000000003a0d2000-0x000000003a0dbfff]
[ 0.000000] node 0: [mem 0x000000003a0dc000-0x000000003f0bffff]
[ 0.000000] node 0: [mem 0x000000003f0c0000-0x000000003f0effff]
[ 0.000000] node 0: [mem 0x000000003f0f0000-0x000000003fbfffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fbfffff]
[ 0.000000] On node 0 totalpages: 261120
[ 0.000000] DMA zone: 4080 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 261120 pages, LIFO batch:31
[ 0.000000] psci: probing for conduit method from ACPI.
NOTICE: [psci_smc_handler]:[397L] PSCI_VERSION CALL
[ 0.000000] psci: PSCIv1.0 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] percpu: Embedded 24 pages/cpu @ffff80003fbd3000 s59264 r8192 d30848 u98304
[ 0.000000] pcpu-alloc: s59264 r8192 d30848 u98304 alloc=24*4096
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 257040
[ 0.000000] Policy zone: DMA
[ 0.000000] Kernel command line: earlycon=pl011,mmio32,0x94080000 initrd=filesystem.cpio.gz acpi=force loglevel=8
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory: 959336K/1044480K available (9148K kernel code, 1108K rwdata, 4136K rodata, 1088K init, 398K bss, 68760K reserved, 16384K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] modules : 0xffff000000000000 - 0xffff000008000000 ( 128 MB)
[ 0.000000] vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000 (129022 GB)
[ 0.000000] .text : 0xffff000008080000 - 0xffff000008970000 ( 9152 KB)
[ 0.000000] .rodata : 0xffff000008970000 - 0xffff000008d80000 ( 4160 KB)
[ 0.000000] .init : 0xffff000008d80000 - 0xffff000008e90000 ( 1088 KB)
[ 0.000000] .data : 0xffff000008e90000 - 0xffff000008fa5200 ( 1109 KB)
[ 0.000000] .bss : 0xffff000008fa5200 - 0xffff000009008c70 ( 399 KB)
[ 0.000000] fixed : 0xffff7dfffe7fd000 - 0xffff7dfffec00000 ( 4108 KB)
[ 0.000000] PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000 ( 16 MB)
[ 0.000000] vmemmap : 0xffff7e0000000000 - 0xffff800000000000 ( 2048 GB maximum)
[ 0.000000] 0xffff7e0000000000 - 0xffff7e0000ff0000 ( 15 MB actual)
[ 0.000000] memory : 0xffff800000000000 - 0xffff80003fc00000 ( 1020 MB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[ 0.000000] ACPI: SRAT not present
[ 0.000000] ITS [mem 0xa8100000-0xa811ffff]
[ 0.000000] ITS@0x00000000a8100000: allocated 8192 Devices @3a100000 (indirect, esz 8, psz 16K, shr 1)
[ 0.000000] ITS@0x00000000a8100000: allocated 1024 Virtual CPUs @3a0e4000 (flat, esz 16, psz 4K, shr 1)
[ 0.000000] ITS@0x00000000a8100000: allocated 256 Interrupt Collections @3a0e8000 (flat, esz 16, psz 4K, shr 1)
[ 0.000000] GIC: using LPI property table @0x000000003a110000
[ 0.000000] ITS: Allocated 1792 chunks for LPIs
[ 0.000000] GICv3: CPU0: found redistributor 30000 region 0:0x00000000aa100000
[ 0.000000] CPU0: using LPI pending table @0x000000003a120000
[ 0.000000] arch_timer: cp15 timer(s) running at 6.25MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e6b, max_idle_ns: 440795202301 ns
[ 0.000329] sched_clock: 56 bits at 6MHz, resolution 160ns, wraps every 4398046511040ns
[ 0.022603] Console: colour dummy device 80x25
[ 0.034500] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.50 BogoMIPS (lpj=25000)
[ 0.053277] pid_max: default: 32768 minimum: 301
[ 0.062980] ACPI: Core revision 20170531
[ 0.119428] ACPI: 1 ACPI AML tables successfully acquired and loaded
[ 0.136157] Security Framework initialized
[ 0.190025] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.225984] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.240352] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.253407] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.362723] ASID allocator initialised with 65536 entries
[ 0.379364] Hierarchical SRCU implementation.
[ 0.409156] PCI/MSI: ITS@0xa8100000 domain created
[ 0.420450] Platform MSI: ITS@0xa8100000 domain created
[ 0.437286] Remapping and enabling EFI services.
[ 0.448258] EFI remap 0x0000000031a60000 => 0000000020000000
[ 0.459979] EFI remap 0x0000000039c30000 => 0000000020020000
[ 0.471578] EFI remap 0x0000000039c80000 => 0000000020070000
[ 0.483099] EFI remap 0x0000000039cf0000 => 00000000200c0000
[ 0.494579] EFI remap 0x0000000039d40000 => 0000000020110000
[ 0.506102] EFI remap 0x0000000039d90000 => 0000000020160000
[ 0.517611] EFI remap 0x0000000039de0000 => 00000000201b0000
[ 0.529248] EFI remap 0x0000000039e30000 => 0000000020200000
[ 0.540737] EFI remap 0x0000000039e80000 => 0000000020250000
[ 0.552216] EFI remap 0x0000000039ed0000 => 00000000202a0000
[ 0.563700] EFI remap 0x0000000039f20000 => 00000000202f0000
[ 0.575540] EFI remap 0x0000000039f70000 => 0000000020340000
[ 0.587015] EFI remap 0x0000000039fe0000 => 00000000203b0000
[ 0.598629] EFI remap 0x000000003a030000 => 0000000020400000
[ 0.610118] EFI remap 0x000000003a080000 => 0000000020450000
[ 0.621288] EFI remap 0x000000003f0c0000 => 00000000204a0000
[ 0.650692] smp: Bringing up secondary CPUs ...
[ 0.659943] smp: Brought up 1 node, 1 CPU
[ 0.667468] SMP: Total of 1 processors activated.
[ 0.676346] CPU features: detected feature: GIC system register CPU interface
[ 0.689110] CPU features: detected feature: Privileged Access Never
[ 0.700384] CPU features: detected feature: User Access Override
[ 0.711175] CPU features: detected feature: Virtualization Host Extensions
[ 0.731364] CPU: All CPU(s) started at EL2
[ 0.739396] alternatives: patching kernel code
[ 0.920360] devtmpfs: initialized
[ 0.990978] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 1.031879] futex hash table entries: 256 (order: 3, 32768 bytes)
[ 1.100210] pinctrl core: initialized pinctrl subsystem
[ 1.231764] random: fast init done
[ 1.441138] DMI not present or invalid.
[ 1.531527] NET: Registered protocol family 16
[ 1.954694] cpuidle: using governor menu
[ 1.997861] vdso: 2 pages (1 code @ ffff000008976000, 1 data @ ffff000008e95000)
[ 2.014553] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 2.086616] DMA: preallocated 256 KiB pool for atomic allocations
[ 2.169828] ACPI: bus type PCI registered
[ 2.269948] Serial: AMBA PL011 UART driver
[ 6.266490] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 6.442895] ACPI: Added _OSI(Module Device)
[ 6.452727] ACPI: Added _OSI(Processor Device)
[ 6.461662] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 6.470751] ACPI: Added _OSI(Processor Aggregator Device)
[ 6.567598] ACPI: Interpreter enabled
[ 6.575160] ACPI: Using GIC for interrupt routing
[ 6.595078] ACPI: MCFG table detected, 1 entries
[ 7.427562] ARMH0011:00: ttyAMA0 at MMIO 0x94080000 (irq = 5, base_baud = 0) is a SBSA
[ 7.446220] console [ttyAMA0] enabled
[ 7.446220] console [ttyAMA0] enabled
[ 7.458472] bootconsole [pl11] disabled
[ 7.458472] bootconsole [pl11] disabled
[ 7.536276] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3f])
[ 7.544988] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[ 7.584851] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[ 7.617867] acpi PNP0A08:00: [Firmware Bug]: ECAM area [mem 0xd0000000-0xd3ffffff] not reserved in ACPI namespace
[ 7.630773] acpi PNP0A08:00: ECAM at [mem 0xd0000000-0xd3ffffff] for [bus 00-3f]
[ 7.645534] Remapped I/O 0x00000000e0000000 to [io 0x0000-0xffff window]
[ 7.669434] PCI host bridge to bus 0000:00
[ 7.682031] pci_bus 0000:00: root bus resource [mem 0x80000000000-0x83fffffffff pref window]
[ 7.700166] pci_bus 0000:00: root bus resource [mem 0xe0010000-0xe7ffffff window]
[ 7.709762] pci_bus 0000:00: root bus resource [io 0x0000-0xffff window]
[ 7.718134] pci_bus 0000:00: root bus resource [bus 00-3f]
[ 7.725526] pci_bus 0000:00: scanning bus
[ 7.732374] pci 0000:00:00.0: [19e5:a120] type 01 class 0x060400
[ 7.748480] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 7.756070] pci 0000:00:00.0: PME# disabled
[ 7.781356] pci 0000:00:08.0: [19e5:a120] type 01 class 0x060400
[ 7.809322] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 7.817775] pci 0000:00:08.0: PME# disabled
[ 7.842221] pci_bus 0000:00: fixups for bus
[ 7.857795] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 0
[ 7.881935] pci_bus 0000:01: scanning bus
[ 7.894711] pci 0000:01:00.0: [8086:10fb] type 00 class 0x020000
[ 7.905129] pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x84
[ 7.914365] pci 0000:01:00.0: reg 0x10: [mem 0x80000c00000-0x80000ffffff 64bit pref]
[ 7.923845] pci 0000:01:00.0: reg 0x18: [io 0x0000-0x001f]
[ 7.931977] pci 0000:01:00.0: reg 0x20: [mem 0x80001404000-0x80001407fff 64bit pref]
[ 7.941331] pci 0000:01:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[ 7.954316] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 7.960786] pci 0000:01:00.0: PME# disabled
[ 7.967206] pci 0000:01:00.0: reg 0x184: [mem 0x80001304000-0x80001307fff 64bit pref]
[ 7.975904] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x80001304000-0x80001403fff 64bit pref] (contains BAR0 for 64 VFs)
[ 7.988004] pci 0000:01:00.0: reg 0x190: [mem 0x80001204000-0x80001207fff 64bit pref]
[ 7.996493] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x80001204000-0x80001303fff 64bit pref] (contains BAR3 for 64 VFs)
[ 8.036499] pci 0000:01:00.1: [8086:10fb] type 00 class 0x020000
[ 8.055046] pci 0000:01:00.1: calling quirk_f0_vpd_link+0x0/0x84
[ 8.064331] pci 0000:01:00.1: reg 0x10: [mem 0x80000800000-0x80000bfffff 64bit pref]
[ 8.074492] pci 0000:01:00.1: reg 0x18: [io 0x0000-0x001f]
[ 8.082047] pci 0000:01:00.1: reg 0x20: [mem 0x80001200000-0x80001203fff 64bit pref]
[ 8.091130] pci 0000:01:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[ 8.103548] pci 0000:01:00.1: PME# supported from D0 D3hot
[ 8.109878] pci 0000:01:00.1: PME# disabled
[ 8.115950] pci 0000:01:00.1: reg 0x184: [mem 0x80001100000-0x80001103fff 64bit pref]
[ 8.124516] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x80001100000-0x800011fffff 64bit pref] (contains BAR0 for 64 VFs)
[ 8.136598] pci 0000:01:00.1: reg 0x190: [mem 0x80001000000-0x80001003fff 64bit pref]
[ 8.145071] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0x80001000000-0x800010fffff 64bit pref] (contains BAR3 for 64 VFs)
[ 8.186151] pci_bus 0000:01: fixups for bus
[ 8.200134] pci_bus 0000:01: bus scan returning with max=01
[ 8.209477] pci 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 0
[ 8.231444] pci_bus 0000:03: scanning bus
[ 8.244266] pci 0000:03:00.0: [8086:1521] type 00 class 0x020000
[ 8.255004] pci 0000:03:00.0: calling quirk_f0_vpd_link+0x0/0x84
[ 8.264702] pci 0000:03:00.0: reg 0x10: [mem 0xe0300000-0xe03fffff]
[ 8.273645] pci 0000:03:00.0: reg 0x1c: [mem 0xe040c000-0xe040ffff]
[ 8.282075] pci 0000:03:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[ 8.295226] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
[ 8.302214] pci 0000:03:00.0: PME# disabled
[ 8.308536] pci 0000:03:00.0: reg 0x184: [mem 0x800015e0000-0x800015e3fff 64bit pref]
[ 8.317179] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0x800015e0000-0x800015fffff 64bit pref] (contains BAR0 for 8 VFs)
[ 8.329199] pci 0000:03:00.0: reg 0x190: [mem 0x800015c0000-0x800015c3fff 64bit pref]
[ 8.337636] pci 0000:03:00.0: VF(n) BAR3 space: [mem 0x800015c0000-0x800015dffff 64bit pref] (contains BAR3 for 8 VFs)
[ 8.368526] pci 0000:03:00.1: [8086:1521] type 00 class 0x020000
[ 8.386885] pci 0000:03:00.1: calling quirk_f0_vpd_link+0x0/0x84
[ 8.395857] pci 0000:03:00.1: reg 0x10: [mem 0xe0200000-0xe02fffff]
[ 8.404993] pci 0000:03:00.1: reg 0x1c: [mem 0xe0408000-0xe040bfff]
[ 8.413251] pci 0000:03:00.1: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[ 8.426293] pci 0000:03:00.1: PME# supported from D0 D3hot D3cold
[ 8.433273] pci 0000:03:00.1: PME# disabled
[ 8.439321] pci 0000:03:00.1: reg 0x184: [mem 0x800015a0000-0x800015a3fff 64bit pref]
[ 8.447914] pci 0000:03:00.1: VF(n) BAR0 space: [mem 0x800015a0000-0x800015bffff 64bit pref] (contains BAR0 for 8 VFs)
[ 8.459963] pci 0000:03:00.1: reg 0x190: [mem 0x80001580000-0x80001583fff 64bit pref]
[ 8.468410] pci 0000:03:00.1: VF(n) BAR3 space: [mem 0x80001580000-0x8000159ffff 64bit pref] (contains BAR3 for 8 VFs)
[ 8.499856] pci 0000:03:00.2: [8086:1521] type 00 class 0x020000
[ 8.518734] pci 0000:03:00.2: calling quirk_f0_vpd_link+0x0/0x84
[ 8.527539] pci 0000:03:00.2: reg 0x10: [mem 0xe0100000-0xe01fffff]
[ 8.536650] pci 0000:03:00.2: reg 0x1c: [mem 0xe0404000-0xe0407fff]
[ 8.550703] pci 0000:03:00.2: PME# supported from D0 D3hot D3cold
[ 8.557839] pci 0000:03:00.2: PME# disabled
[ 8.563852] pci 0000:03:00.2: reg 0x184: [mem 0x80001560000-0x80001563fff 64bit pref]
[ 8.572450] pci 0000:03:00.2: VF(n) BAR0 space: [mem 0x80001560000-0x8000157ffff 64bit pref] (contains BAR0 for 8 VFs)
[ 8.584466] pci 0000:03:00.2: reg 0x190: [mem 0x80001540000-0x80001543fff 64bit pref]
[ 8.592948] pci 0000:03:00.2: VF(n) BAR3 space: [mem 0x80001540000-0x8000155ffff 64bit pref] (contains BAR3 for 8 VFs)
[ 8.623880] pci 0000:03:00.3: [8086:1521] type 00 class 0x020000
[ 8.643212] pci 0000:03:00.3: calling quirk_f0_vpd_link+0x0/0x84
[ 8.652565] pci 0000:03:00.3: reg 0x10: [mem 0xe0000000-0xe00fffff]
[ 8.661184] pci 0000:03:00.3: reg 0x1c: [mem 0xe0400000-0xe0403fff]
[ 8.675795] pci 0000:03:00.3: PME# supported from D0 D3hot D3cold
[ 8.683456] pci 0000:03:00.3: PME# disabled
[ 8.689539] pci 0000:03:00.3: reg 0x184: [mem 0x80001520000-0x80001523fff 64bit pref]
[ 8.698171] pci 0000:03:00.3: VF(n) BAR0 space: [mem 0x80001520000-0x8000153ffff 64bit pref] (contains BAR0 for 8 VFs)
[ 8.710194] pci 0000:03:00.3: reg 0x190: [mem 0x80001500000-0x80001503fff 64bit pref]
[ 8.718661] pci 0000:03:00.3: VF(n) BAR3 space: [mem 0x80001500000-0x8000151ffff 64bit pref] (contains BAR3 for 8 VFs)
[ 8.751770] pci_bus 0000:03: fixups for bus
[ 8.767326] pci_bus 0000:03: bus scan returning with max=03
[ 8.775771] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 1
[ 8.784088] pci 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 1
[ 8.792775] pci_bus 0000:00: bus scan returning with max=04
[ 8.802531] pci 0000:00:00.0: BAR 14: assigned [mem 0xe0400000-0xe0bfffff]
[ 8.811578] pci 0000:00:00.0: BAR 15: assigned [mem 0x80000000000-0x80000dfffff 64bit pref]
[ 8.820739] pci 0000:00:08.0: BAR 14: assigned [mem 0xe0c00000-0xe12fffff]
[ 8.828338] pci 0000:00:08.0: BAR 15: assigned [mem 0x80000e00000-0x80000ffffff 64bit pref]
[ 8.837354] pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 8.844176] pci 0000:00:08.0: BAR 13: assigned [io 0x2000-0x2fff]
[ 8.851420] pci 0000:01:00.0: BAR 0: assigned [mem 0x80000000000-0x800003fffff 64bit pref]
[ 8.860790] pci 0000:01:00.0: BAR 6: assigned [mem 0xe0400000-0xe07fffff pref]
[ 8.868711] pci 0000:01:00.1: BAR 0: assigned [mem 0x80000400000-0x800007fffff 64bit pref]
[ 8.878024] pci 0000:01:00.1: BAR 6: assigned [mem 0xe0800000-0xe0bfffff pref]
[ 8.885861] pci 0000:01:00.0: BAR 4: assigned [mem 0x80000800000-0x80000803fff 64bit pref]
[ 8.895108] pci 0000:01:00.0: BAR 7: assigned [mem 0x80000804000-0x80000903fff 64bit pref]
[ 8.904170] pci 0000:01:00.0: BAR 10: assigned [mem 0x80000904000-0x80000a03fff 64bit pref]
[ 8.913312] pci 0000:01:00.1: BAR 4: assigned [mem 0x80000a04000-0x80000a07fff 64bit pref]
[ 8.922554] pci 0000:01:00.1: BAR 7: assigned [mem 0x80000a08000-0x80000b07fff 64bit pref]
[ 8.931594] pci 0000:01:00.1: BAR 10: assigned [mem 0x80000b08000-0x80000c07fff 64bit pref]
[ 8.940710] pci 0000:01:00.0: BAR 2: assigned [io 0x1000-0x101f]
[ 8.947512] pci 0000:01:00.1: BAR 2: assigned [io 0x1020-0x103f]
[ 8.956549] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[ 8.962641] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
[ 8.969541] pci 0000:00:00.0: bridge window [mem 0xe0400000-0xe0bfffff]
[ 8.977069] pci 0000:00:00.0: bridge window [mem 0x80000000000-0x80000dfffff 64bit pref]
[ 8.986830] pci 0000:03:00.0: BAR 0: assigned [mem 0xe0c00000-0xe0cfffff]
[ 8.994486] pci 0000:03:00.0: BAR 6: assigned [mem 0xe0d00000-0xe0dfffff pref]
[ 9.002391] pci 0000:03:00.1: BAR 0: assigned [mem 0xe0e00000-0xe0efffff]
[ 9.009901] pci 0000:03:00.1: BAR 6: assigned [mem 0xe0f00000-0xe0ffffff pref]
[ 9.017770] pci 0000:03:00.2: BAR 0: assigned [mem 0xe1000000-0xe10fffff]
[ 9.025268] pci 0000:03:00.3: BAR 0: assigned [mem 0xe1100000-0xe11fffff]
[ 9.032763] pci 0000:03:00.0: BAR 3: assigned [mem 0xe1200000-0xe1203fff]
[ 9.040281] pci 0000:03:00.0: BAR 7: assigned [mem 0x80000e00000-0x80000e1ffff 64bit pref]
[ 9.049319] pci 0000:03:00.0: BAR 10: assigned [mem 0x80000e20000-0x80000e3ffff 64bit pref]
[ 9.058434] pci 0000:03:00.1: BAR 3: assigned [mem 0xe1204000-0xe1207fff]
[ 9.065936] pci 0000:03:00.1: BAR 7: assigned [mem 0x80000e40000-0x80000e5ffff 64bit pref]
[ 9.074953] pci 0000:03:00.1: BAR 10: assigned [mem 0x80000e60000-0x80000e7ffff 64bit pref]
[ 9.084074] pci 0000:03:00.2: BAR 3: assigned [mem 0xe1208000-0xe120bfff]
[ 9.091562] pci 0000:03:00.2: BAR 7: assigned [mem 0x80000e80000-0x80000e9ffff 64bit pref]
[ 9.100587] pci 0000:03:00.2: BAR 10: assigned [mem 0x80000ea0000-0x80000ebffff 64bit pref]
[ 9.109690] pci 0000:03:00.3: BAR 3: assigned [mem 0xe120c000-0xe120ffff]
[ 9.117187] pci 0000:03:00.3: BAR 7: assigned [mem 0x80000ec0000-0x80000edffff 64bit pref]
[ 9.126233] pci 0000:03:00.3: BAR 10: assigned [mem 0x80000ee0000-0x80000efffff 64bit pref]
[ 9.135511] pci 0000:00:08.0: PCI bridge to [bus 03-04]
[ 9.141357] pci 0000:00:08.0: bridge window [io 0x2000-0x2fff]
[ 9.148217] pci 0000:00:08.0: bridge window [mem 0xe0c00000-0xe12fffff]
[ 9.155707] pci 0000:00:08.0: bridge window [mem 0x80000e00000-0x80000ffffff 64bit pref]
[ 9.560015] vgaarb: loaded
[ 9.640424] SCSI subsystem initialized
[ 9.698093] libata version 3.00 loaded.
[ 9.780441] ACPI: bus type USB registered
[ 9.826557] usbcore: registered new interface driver usbfs
[ 9.854891] usbcore: registered new interface driver hub
[ 9.878112] usbcore: registered new device driver usb
[ 10.024058] pps_core: LinuxPPS API ver. 1 registered
[ 10.040988] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@xxxxxxxx>
[ 10.058492] PTP clock support registered
[ 10.136679] dmi: Firmware registration failed.
[ 10.150629] Registered efivars operations
[ 10.218533] Advanced Linux Sound Architecture Driver Initialized.
[ 10.372769] clocksource: Switched to clocksource arch_sys_counter
[ 10.396969] VFS: Disk quotas dquot_6.6.0
[ 10.406240] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 10.454282] pnp: PnP ACPI init
[ 10.496852] pnp: PnP ACPI: found 0 devices
[ 12.594530] NET: Registered protocol family 2
[ 12.641412] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[ 12.653888] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[ 12.671318] TCP: Hash tables configured (established 8192 bind 8192)
[ 12.687407] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 12.696177] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 12.713559] NET: Registered protocol family 1
[ 12.751919] RPC: Registered named UNIX socket transport module.
[ 12.760732] RPC: Registered udp transport module.
[ 12.766108] RPC: Registered tcp transport module.
[ 12.772643] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 12.783117] pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x1c0
[ 12.791253] pci 0000:01:00.1: calling quirk_e100_interrupt+0x0/0x1c0
[ 12.798827] pci 0000:03:00.0: calling quirk_e100_interrupt+0x0/0x1c0
[ 12.805985] PCI: CLS mismatch (64 != 128), using 128 bytes
[ 12.812477] pci 0000:03:00.1: calling quirk_e100_interrupt+0x0/0x1c0
[ 12.819852] pci 0000:03:00.2: calling quirk_e100_interrupt+0x0/0x1c0
[ 12.827500] pci 0000:03:00.3: calling quirk_e100_interrupt+0x0/0x1c0
[ 12.853503] Unpacking initramfs...
[ 93.115023] Initramfs unpacking failed: junk in compressed archive
[ 94.520143] Freeing initrd memory: 28172K
[ 94.541265] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 13 counters available
[ 94.551107] kvm [1]: 16-bit VMID
[ 94.555180] kvm [1]: VHE mode initialized successfully
[ 94.561474] kvm [1]: GICv3: no GICV resource entry
[ 94.566985] kvm [1]: disabling GICv2 emulation
[ 94.573285] kvm [1]: GIC system register CPU interface enabled
[ 94.583554] kvm [1]: vgic interrupt IRQ1
[ 94.589094] kvm [1]: virtual timer IRQ3
[ 94.980154] audit: initializing netlink subsys (disabled)
[ 95.009820] audit: type=2000 audit(92.583:1): state=initialized audit_enabled=0 res=1
[ 95.127062] workingset: timestamp_bits=44 max_order=18 bucket_order=0
[ 96.798116] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 96.969554] NFS: Registering the id_resolver key type
[ 96.979188] Key type id_resolver registered
[ 96.985193] Key type id_legacy registered
[ 96.991028] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 97.018200] 9p: Installing v9fs 9p2000 file system support
[ 97.330580] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[ 97.351086] io scheduler noop registered
[ 97.403202] io scheduler cfq registered (default)
[ 97.418177] io scheduler mq-deadline registered
[ 97.426968] io scheduler kyber registered
[ 98.764128] acpi-ged ACPI0013:00: GED listening GSI 129 @ IRQ 6
[ 100.898280] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 101.310451] SuperH (H)SCI(F) driver initialized
[ 101.381953] msm_serial: driver initialized
[ 101.533722] arm-smmu-v3 arm-smmu-v3.0.auto: option mask 0x0
[ 101.545745] arm-smmu-v3 arm-smmu-v3.0.auto: ias 48-bit, oas 48-bit (features 0x00000fef)
[ 101.578260] arm-smmu-v3 arm-smmu-v3.0.auto: failed to allocate MSIs
[ 101.622191] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 102.507569] loop: module loaded
[ 102.734046] hisi_sas: driver version v1.6
[ 103.361973] libphy: Fixed MDIO Bus: probed
[ 103.582183] tun: Universal TUN/TAP device driver, 1.6
[ 103.811192] libphy: Hisilicon MII Bus: probed
[ 103.886133] hns-nic HISI00C2:00: not find ae-handle
[ 103.902399] hns-nic: probe of HISI00C2:00 failed with error -22
[ 103.920648] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[ 103.937988] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[ 103.961915] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 103.981971] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 104.026744] iommu: Adding device 0000:03:00.0 to group 0
[ 104.039941] igb 0000:03:00.0: runtime IRQ mapping not provided by arch
[ 104.060977] igb 0000:03:00.0: enabling device (0000 -> 0002)
[ 104.069357] igb 0000:03:00.0: enabling bus mastering
[ 104.107285] igb 0000:03:00.0: irq 9 for MSI
[ 104.137704] igb 0000:03:00.0: irq 9 for MSI
[ 104.143757] igb 0000:03:00.0: irq 10 for MSI
[ 104.149258] igb 0000:03:00.0: irq 11 for MSI
[ 104.762718] igb 0000:03:00.0: added PHC on eth0
[ 104.778186] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 104.790609] igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:61
[ 104.800924] igb 0000:03:00.0: eth0: PBA No: 106300-000
[ 104.807883] igb 0000:03:00.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[ 104.839294] iommu: Adding device 0000:03:00.1 to group 1
[ 104.853148] igb 0000:03:00.1: runtime IRQ mapping not provided by arch
[ 104.873162] igb 0000:03:00.1: enabling device (0000 -> 0002)
[ 104.881690] igb 0000:03:00.1: enabling bus mastering
[ 104.919958] igb 0000:03:00.1: irq 13 for MSI
[ 104.947522] igb 0000:03:00.1: irq 13 for MSI
[ 104.954060] igb 0000:03:00.1: irq 14 for MSI
[ 104.959320] igb 0000:03:00.1: irq 15 for MSI
[ 105.493974] igb 0000:03:00.1: added PHC on eth1
[ 105.507684] igb 0000:03:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 105.520897] igb 0000:03:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:62
[ 105.530370] igb 0000:03:00.1: eth1: PBA No: 106300-000
[ 105.536968] igb 0000:03:00.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[ 105.567646] iommu: Adding device 0000:03:00.2 to group 2
[ 105.581190] igb 0000:03:00.2: runtime IRQ mapping not provided by arch
[ 105.601115] igb 0000:03:00.2: enabling device (0000 -> 0002)
[ 105.609543] igb 0000:03:00.2: enabling bus mastering
[ 105.646074] igb 0000:03:00.2: irq 17 for MSI
[ 105.674381] igb 0000:03:00.2: irq 17 for MSI
[ 105.683278] igb 0000:03:00.2: irq 18 for MSI
[ 105.688778] igb 0000:03:00.2: irq 19 for MSI
[ 106.217634] igb 0000:03:00.2: added PHC on eth2
[ 106.233860] igb 0000:03:00.2: Intel(R) Gigabit Ethernet Network Connection
[ 106.243657] igb 0000:03:00.2: eth2: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:63
[ 106.254372] igb 0000:03:00.2: eth2: PBA No: 106300-000
[ 106.263316] igb 0000:03:00.2: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[ 106.295470] iommu: Adding device 0000:03:00.3 to group 3
[ 106.308924] igb 0000:03:00.3: runtime IRQ mapping not provided by arch
[ 106.321877] pci 0000:00:08.0: can't derive routing for PCI INT D
[ 106.328793] igb 0000:03:00.3: PCI INT D: no GSI
[ 106.334925] igb 0000:03:00.3: enabling device (0000 -> 0002)
[ 106.342987] igb 0000:03:00.3: enabling bus mastering
[ 106.381164] igb 0000:03:00.3: irq 20 for MSI
[ 106.395108] pci 0000:00:08.0: can't derive routing for PCI INT D
[ 106.402247] igb 0000:03:00.3: PCI INT D: no GSI
[ 106.423980] igb 0000:03:00.3: irq 20 for MSI
[ 106.430404] igb 0000:03:00.3: irq 21 for MSI
[ 106.435604] igb 0000:03:00.3: irq 22 for MSI
[ 106.969195] igb 0000:03:00.3: added PHC on eth3
[ 106.986599] igb 0000:03:00.3: Intel(R) Gigabit Ethernet Network Connection
[ 106.997192] igb 0000:03:00.3: eth3: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:64
[ 107.006052] igb 0000:03:00.3: eth3: PBA No: 106300-000
[ 107.014340] igb 0000:03:00.3: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[ 107.034774] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[ 107.051759] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 107.073731] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[ 107.093439] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 107.147616] iommu: Adding device 0000:01:00.0 to group 4
[ 107.161275] ixgbe 0000:01:00.0: runtime IRQ mapping not provided by arch
[ 107.176976] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 107.185395] ixgbe 0000:01:00.0: enabling bus mastering
[ 108.634988] ixgbe 0000:01:00.0: irq 23 for MSI
[ 108.643436] ixgbe 0000:01:00.0: irq 24 for MSI
[ 108.651323] ixgbe 0000:01:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 108.663349] ixgbe 0000:01:00.0: PCI Express bandwidth of 8GT/s available
[ 108.670828] ixgbe 0000:01:00.0: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[ 108.678675] ixgbe 0000:01:00.0: This is not sufficient for optimal performance of this card.
[ 108.687764] ixgbe 0000:01:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[ 108.697626] ixgbe 0000:01:00.0: A slot with more lanes and/or higher speed is suggested.
[ 108.706705] ixgbe 0000:01:00.0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[ 108.713609] ixgbe 0000:01:00.0: 9c:37:f4:90:cd:23
[ 110.602795] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[ 110.637473] iommu: Adding device 0000:01:00.1 to group 5
[ 110.650676] ixgbe 0000:01:00.1: runtime IRQ mapping not provided by arch
[ 110.665725] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[ 110.674161] ixgbe 0000:01:00.1: enabling bus mastering
[ 111.150789] ixgbe 0000:01:00.1: irq 25 for MSI
[ 111.159276] ixgbe 0000:01:00.1: irq 26 for MSI
[ 111.167407] ixgbe 0000:01:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 111.179704] ixgbe 0000:01:00.1: PCI Express bandwidth of 8GT/s available
[ 111.187549] ixgbe 0000:01:00.1: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[ 111.195360] ixgbe 0000:01:00.1: This is not sufficient for optimal performance of this card.
[ 111.204463] ixgbe 0000:01:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[ 111.214037] ixgbe 0000:01:00.1: A slot with more lanes and/or higher speed is suggested.
[ 111.223129] ixgbe 0000:01:00.1: MAC: 2, PHY: 17, SFP+: 10, PBA No: FFFFFF-0FF
[ 111.230947] ixgbe 0000:01:00.1: 9c:37:f4:90:cd:24
[ 113.207022] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[ 113.322096] sky2: driver version 1.30
[ 113.462071] VFIO - User Level meta-driver version: 0.3
[ 113.771122] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 113.787380] ehci-pci: EHCI PCI platform driver
[ 113.806857] ehci-platform: EHCI generic platform driver
[ 113.839730] ehci-orion: EHCI orion driver
[ 113.870892] ehci-exynos: EHCI EXYNOS driver
[ 113.901359] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[ 113.935225] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 113.954224] ohci-pci: OHCI PCI platform driver
[ 113.975549] ohci-platform: OHCI generic platform driver
[ 114.010294] ohci-exynos: OHCI EXYNOS driver
[ 114.141756] usbcore: registered new interface driver usb-storage
[ 114.773837] i2c /dev entries driver
[ 115.641863] sdhci: Secure Digital Host Controller Interface driver
[ 115.663068] sdhci: Copyright(c) Pierre Ossman
[ 115.729786] Synopsys Designware Multimedia Card Interface Driver
[ 115.918274] sdhci-pltfm: SDHCI platform and OF driver helper
[ 116.189809] ledtrig-cpu: registered to indicate activity on CPUs
[ 116.554506] usbcore: registered new interface driver usbhid
[ 116.569432] usbhid: USB HID core driver
[ 116.995307] NET: Registered protocol family 17
[ 117.008752] 9pnet: Installing 9P2000 support
[ 117.025356] Key type dns_resolver registered
[ 117.170735] registered taskstats version 1
[ 117.231555] iommu: Adding device 0000:00:00.0 to group 6
[ 117.245350] pcieport 0000:00:00.0: runtime IRQ mapping not provided by arch
[ 117.297232] pcieport 0000:00:00.0: irq [27-58] for MSI
[ 117.361464] pcieport 0000:00:00.0: irq [27-29] for MSI
[ 117.389455] pcieport 0000:00:00.0: Signaling PME with IRQ 27
[ 117.446120] pcieport 0000:00:00.0: AER enabled with IRQ 28
[ 117.475969] pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
[ 117.506102] pci_bus 0000:01: dev 00, created physical slot 0
[ 117.592065] dpc 0000:00:00.0:pcie010: DPC error containment capabilities: Int Msg #2, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
[ 117.632624] iommu: Adding device 0000:00:08.0 to group 7
[ 117.645307] pcieport 0000:00:08.0: runtime IRQ mapping not provided by arch
[ 117.693898] pcieport 0000:00:08.0: irq [30-61] for MSI
[ 117.757333] pcieport 0000:00:08.0: irq [30-32] for MSI
[ 117.785514] pcieport 0000:00:08.0: Signaling PME with IRQ 30
[ 117.833459] pcieport 0000:00:08.0: AER enabled with IRQ 31
[ 117.870840] pciehp 0000:00:08.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
[ 117.898814] pci_bus 0000:03: dev 00, created physical slot 0-1
[ 117.966720] dpc 0000:00:08.0:pcie010: DPC error containment capabilities: Int Msg #2, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
[ 118.017679] hctosys: unable to open rtc device (rtc0)
[ 118.609748] ALSA device list:
[ 118.614432] No soundcards found.
[ 118.691983] Freeing unused kernel memory: 1088K
root@(none)$
root@(none)$ cd /home/
root@(none)$ ./lspci -tv
-[0000:00]-+-00.0-[01-02]--+-00.0 Device 8086:10fb
| \-00.1 Device 8086:10fb
\-08.0-[03-04]--+-00.0 Device 8086:1521
+-00.1 Device 8086:1521
+-00.2 Device 8086:1521
\-00.3 Device 8086:1521
root@(none)$ echo1 1 > /sys/devices/pci0000:00/0000:00:00.0/remove
[ 447.630066] ixgbe 0000:01:00.1: PME# disabled
[ 449.503490] ixgbe 0000:01:00.1: complete
[ 449.515380] ixgbe 0000:01:00.0: PME# disabled
[ 451.323235] ixgbe 0000:01:00.0: complete
[ 451.335069] pcieport 0000:00:00.0: PME# disabled
[ 451.398016] pci_bus 0000:01: dev 00, dec refcount to 0
[ 451.414247] pci_bus 0000:01: dev 00, released physical slot 0
[ 451.523677] iommu: Removing device 0000:01:00.0 from group 4
[ 451.563037] iommu: Removing device 0000:01:00.1 from group 5
[ 451.596134] pci_bus 0000:01: busn_res: [bus 01-02] is released
[ 451.643249] iommu: Removing device 0000:00:00.0 from group 6
root@(none)$ echo 1 > /sys/bus/pci/rescan
[ 476.261084] pci_bus 0000:00: scanning bus
[ 476.268738] pci 0000:00:00.0: [19e5:a120] type 01 class 0x060400
[ 476.285212] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 476.292934] pci 0000:00:00.0: PME# disabled
[ 476.323146] pcieport 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 0
[ 476.342380] pci_bus 0000:03: scanning bus
[ 476.355514] pci_bus 0000:03: bus scan returning with max=03
[ 476.363649] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 0
[ 476.389654] pci_bus 0000:01: scanning bus
[ 476.403577] pci 0000:01:00.0: [8086:10fb] type 00 class 0x020000
[ 476.418389] pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x84
[ 476.427450] pci 0000:01:00.0: reg 0x10: [mem 0x80000000000-0x800003fffff 64bit pref]
[ 476.438299] pci 0000:01:00.0: reg 0x18: [io 0x1000-0x101f]
[ 476.446411] pci 0000:01:00.0: reg 0x20: [mem 0x80000800000-0x80000803fff 64bit pref]
[ 476.456342] pci 0000:01:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[ 476.469753] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 476.476429] pci 0000:01:00.0: PME# disabled
[ 476.482955] pci 0000:01:00.0: reg 0x184: [mem 0x80000804000-0x80000807fff 64bit pref]
[ 476.491789] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x80000804000-0x80000903fff 64bit pref] (contains BAR0 for 64 VFs)
[ 476.504795] pci 0000:01:00.0: reg 0x190: [mem 0x80000904000-0x80000907fff 64bit pref]
[ 476.513379] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x80000904000-0x80000a03fff 64bit pref] (contains BAR3 for 64 VFs)
[ 476.555511] pci 0000:01:00.1: [8086:10fb] type 00 class 0x020000
[ 476.578766] pci 0000:01:00.1: calling quirk_f0_vpd_link+0x0/0x84
[ 476.589784] pci 0000:01:00.1: reg 0x10: [mem 0x80000400000-0x800007fffff 64bit pref]
[ 476.599717] pci 0000:01:00.1: reg 0x18: [io 0x1020-0x103f]
[ 476.608373] pci 0000:01:00.1: reg 0x20: [mem 0x80000a04000-0x80000a07fff 64bit pref]
[ 476.617225] pci 0000:01:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[ 476.629753] pci 0000:01:00.1: PME# supported from D0 D3hot
[ 476.636368] pci 0000:01:00.1: PME# disabled
[ 476.642766] pci 0000:01:00.1: reg 0x184: [mem 0x80000a08000-0x80000a0bfff 64bit pref]
[ 476.651533] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x80000a08000-0x80000b07fff 64bit pref] (contains BAR0 for 64 VFs)
[ 476.664255] pci 0000:01:00.1: reg 0x190: [mem 0x80000b08000-0x80000b0bfff 64bit pref]
[ 476.672824] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0x80000b08000-0x80000c07fff 64bit pref] (contains BAR3 for 64 VFs)
[ 476.717233] pci_bus 0000:01: fixups for bus
[ 476.734843] pci_bus 0000:01: bus scan returning with max=01
[ 476.743552] pcieport 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 1
[ 476.753981] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 1
[ 476.762484] pci_bus 0000:00: bus scan returning with max=04
[ 476.771676] pci 0000:00:00.0: bridge window [mem 0x00200000-0x00bfffff 64bit pref] to [bus 01-02] add_size 400000 add_align 200000
[ 476.786395] pci 0000:00:00.0: bridge window [mem 0x00400000-0x00bfffff] to [bus 01-02] add_size 400000 add_align 400000
[ 476.798969] pci 0000:00:00.0: BAR 14: assigned [mem 0xe1400000-0xe1ffffff]
[ 476.806837] pci 0000:00:00.0: BAR 15: assigned [mem 0x80000000000-0x80000dfffff 64bit pref]
[ 476.816253] pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 476.825090] pci 0000:01:00.0: BAR 0: assigned [mem 0x80000000000-0x800003fffff 64bit pref]
[ 476.834626] pci 0000:01:00.0: BAR 6: assigned [mem 0xe1400000-0xe17fffff pref]
[ 476.842698] pci 0000:01:00.1: BAR 0: assigned [mem 0x80000400000-0x800007fffff 64bit pref]
[ 476.852353] pci 0000:01:00.1: BAR 6: assigned [mem 0xe1800000-0xe1bfffff pref]
[ 476.860311] pci 0000:01:00.0: BAR 4: assigned [mem 0x80000800000-0x80000803fff 64bit pref]
[ 476.869637] pci 0000:01:00.0: BAR 7: assigned [mem 0x80000804000-0x80000903fff 64bit pref]
[ 476.878800] pci 0000:01:00.0: BAR 10: assigned [mem 0x80000904000-0x80000a03fff 64bit pref]
[ 476.888354] pci 0000:01:00.1: BAR 4: assigned [mem 0x80000a04000-0x80000a07fff 64bit pref]
[ 476.897688] pci 0000:01:00.1: BAR 7: assigned [mem 0x80000a08000-0x80000b07fff 64bit pref]
[ 476.906856] pci 0000:01:00.1: BAR 10: assigned [mem 0x80000b08000-0x80000c07fff 64bit pref]
[ 476.916342] pci 0000:01:00.0: BAR 2: assigned [io 0x1000-0x101f]
[ 476.923231] pci 0000:01:00.1: BAR 2: assigned [io 0x1020-0x103f]
[ 476.930426] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[ 476.936429] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
[ 476.943383] pci 0000:00:00.0: bridge window [mem 0xe1400000-0xe1ffffff]
[ 476.950994] pci 0000:00:00.0: bridge window [mem 0x80000000000-0x80000dfffff 64bit pref]
[ 476.989468] iommu: Adding device 0000:00:00.0 to group 4
[ 477.001974] pcieport 0000:00:00.0: runtime IRQ mapping not provided by arch
[ 477.055527] pcieport 0000:00:00.0: irq [33-64] for MSI
[ 477.122631] pcieport 0000:00:00.0: irq [23-25] for MSI
[ 477.149352] pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22
[ 477.193697] pcieport 0000:00:00.0: request AER IRQ 34 failed
[ 477.211684] aer: probe of 0000:00:00.0:pcie002 failed with error -22
[ 477.237964] pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
[ 477.267030] pci_bus 0000:01: dev 00, created physical slot 0
[ 477.286902] pciehp 0000:00:00.0:pcie004: Cannot get irq 33 for the hotplug controller
[ 477.305707] pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1)
[ 477.317644] pci_bus 0000:01: dev 00, dec refcount to 0
[ 477.332681] pci_bus 0000:01: dev 00, released physical slot 0
[ 477.392148] dpc 0000:00:00.0:pcie010: request IRQ35 failed: -22
[ 477.410632] dpc: probe of 0000:00:00.0:pcie010 failed with error -22
[ 477.430064] pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x1c0
[ 477.464772] iommu: Adding device 0000:01:00.0 to group 5
[ 477.477650] ixgbe 0000:01:00.0: runtime IRQ mapping not provided by arch
[ 477.494175] ixgbe 0000:01:00.0: enabling bus mastering
[ 478.942614] ixgbe 0000:01:00.0: irq 26 for MSI
[ 478.952806] ixgbe 0000:01:00.0: irq 27 for MSI
[ 478.961336] ixgbe 0000:01:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 478.973341] ixgbe 0000:01:00.0: PCI Express bandwidth of 8GT/s available
[ 478.980921] ixgbe 0000:01:00.0: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[ 478.988765] ixgbe 0000:01:00.0: This is not sufficient for optimal performance of this card.
[ 478.997873] ixgbe 0000:01:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[ 479.007462] ixgbe 0000:01:00.0: A slot with more lanes and/or higher speed is suggested.
[ 479.016667] ixgbe 0000:01:00.0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[ 479.023628] ixgbe 0000:01:00.0: 9c:37:f4:90:cd:23
[ 481.014736] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[ 481.030916] pci 0000:01:00.1: calling quirk_e100_interrupt+0x0/0x1c0
[ 481.066757] iommu: Adding device 0000:01:00.1 to group 6
[ 481.079297] ixgbe 0000:01:00.1: runtime IRQ mapping not provided by arch
[ 481.095780] ixgbe 0000:01:00.1: enabling bus mastering
[ 481.574664] ixgbe 0000:01:00.1: irq 28 for MSI
[ 481.584598] ixgbe 0000:01:00.1: irq 29 for MSI
[ 481.593121] ixgbe 0000:01:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 481.605182] ixgbe 0000:01:00.1: PCI Express bandwidth of 8GT/s available
[ 481.612810] ixgbe 0000:01:00.1: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[ 481.620683] ixgbe 0000:01:00.1: This is not sufficient for optimal performance of this card.
[ 481.629823] ixgbe 0000:01:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[ 481.639444] ixgbe 0000:01:00.1: A slot with more lanes and/or higher speed is suggested.
[ 481.648550] ixgbe 0000:01:00.1: MAC: 2, PHY: 17, SFP+: 10, PBA No: FFFFFF-0FF
[ 481.656423] ixgbe 0000:01:00.1: 9c:37:f4:90:cd:24
[ 483.786457] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection