Hi all,
I'm trying to use AER inject module and the aer-inject tool to trigger AER handling on a micro TCA based system. The reason is that I'm seeing this particular error appearing every now and then and is reported by the 00:01.0 PCI slot; happens to be root PCIe port on the CPU card. The system also contains a MCH with a PEX 8748 PCIe switch that provides PCIe connectivity to the regular AMC cards inside the crate. There are 5 AMCs connected in the crate , 4x use sis8300drv and one mrf kernel drivers (messages of those as the recovery is happening are seen below).
lspci output:
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 05)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 05)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics P630 (rev 04)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:15.0 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Serial IO I2C Controller #0 (rev 31)
00:15.1 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Serial IO I2C Controller #1 (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)
00:1c.1 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #2 (rev f1)
00:1c.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #3 (rev f1)
00:1c.3 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #4 (rev f1)
00:1f.0 ISA bridge: Intel Corporation CM238 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.3 Audio device: Intel Corporation CM238 HD Audio Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
01:00.1 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
01:00.2 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
01:00.3 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
01:00.4 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
02:01.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
02:02.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
02:08.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
02:09.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
02:0a.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:03.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:08.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:09.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:0a.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:0b.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:11.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
04:12.0 PCI bridge: PLX Technology, Inc. PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (rev ca)
06:00.0 Signal processing controller: Research Centre Juelich Device 0024
08:00.0 Signal processing controller: Research Centre Juelich Device 0024
09:00.0 Signal processing controller: Research Centre Juelich Device 0024
0b:00.0 Signal processing controller: Research Centre Juelich Device 0024
0e:00.0 Signal processing controller: Xilinx Corporation Device 7011
15:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
16:00.0 Ethernet controller: Intel Corporation I210 Gigabit Backplane Connection (rev 03)
17:00.0 Ethernet controller: Intel Corporation I210 Gigabit Backplane Connection (rev 03)
18:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
Note that I had to manually remove following devices to make the recovery report success:
echo 1 > /sys/bus/pci/devices/0000\:02\:02.0/remove
echo 1 > /sys/bus/pci/devices/0000\:02\:08.0/remove
echo 1 > /sys/bus/pci/devices/0000\:02\:09.0/remove
echo 1 > /sys/bus/pci/devices/0000\:02\:0a.0/remove
echo 1 > /sys/bus/pci/devices/0000\:01\:00.1/remove
echo 1 > /sys/bus/pci/devices/0000\:01\:00.2/remove
echo 1 > /sys/bus/pci/devices/0000\:01\:00.3/remove
echo 1 > /sys/bus/pci/devices/0000\:01\:00.4/remove
After that, the PCI device tree looks like this:
00:01.0 root_port, "J6B2", slot 1, device present, speed 8GT/s, width x8
└─01:00.0 upstream_port, PLX Technology, Inc. (10b5), device 8725
└─02:01.0 downstream_port, slot 1, device present, power: Off, speed 8GT/s, width x4
└─03:00.0 upstream_port, PLX Technology, Inc. (10b5) PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (8748)
├─04:00.0 downstream_port, slot 10, power: Off
├─04:01.0 downstream_port, slot 4, device present, power: Off, speed 8GT/s, width x4
│ └─06:00.0 endpoint, Research Centre Juelich (1796), device 0024
├─04:02.0 downstream_port, slot 9, power: Off
├─04:03.0 downstream_port, slot 3, device present, power: Off, speed 8GT/s, width x4
│ └─08:00.0 endpoint, Research Centre Juelich (1796), device 0024
├─04:08.0 downstream_port, slot 5, device present, power: Off, speed 2.5GT/s, width x4
│ └─09:00.0 endpoint, current speed 2.5GT/s target speed 8GT/s, Research Centre Juelich (1796), device 0024
├─04:09.0 downstream_port, slot 11, power: Off
├─04:0a.0 downstream_port, slot 6, device present, power: Off, speed 2.5GT/s, width x4
│ └─0b:00.0 endpoint, current speed 2.5GT/s target speed 8GT/s, Research Centre Juelich (1796), device 0024
├─04:0b.0 downstream_port, slot 12, power: Off
├─04:10.0 downstream_port, slot 8, power: Off
├─04:11.0 downstream_port, slot 2, device present, power: Off, speed 2.5GT/s, width x1
│ └─0e:00.0 endpoint, Xilinx Corporation (10ee), device 7011
└─04:12.0 downstream_port, slot 7, power: Off
00:01.1 root_port, slot 2, device present
00:1c.0 root_port, slot 4, device present, speed 2.5GT/s, width x1
└─15:00.0 endpoint, Intel Corporation (8086) I210 Gigabit Network Connection (1533)
00:1c.1 root_port, slot 5, device present, speed 2.5GT/s, width x1
└─16:00.0 endpoint, Intel Corporation (8086) I210 Gigabit Backplane Connection (1537)
00:1c.2 root_port, slot 6, device present, speed 2.5GT/s, width x1
└─17:00.0 endpoint, Intel Corporation (8086) I210 Gigabit Backplane Connection (1537)
00:1c.3 root_port, "J6B1", slot 7, device present, speed 2.5GT/s, width x1
└─18:00.0 endpoint, Intel Corporation (8086) I210 Gigabit Network Connection (1533)
Finally, here is the result of the AER recovery taking place upon injecting the fatal uncorrectable error into the 00:01.0 slot.
Dec 3 15:25:30 bd-cpu18 kernel: aer 0000:00:01.0:pcie002: aer_inject: Injecting errors 00000000/00004000 into device 0000:00:01.0
Dec 3 15:25:30 bd-cpu18 kernel: pcieport 0000:00:01.0: AER: Uncorrected (Fatal) error received: id=0008
Dec 3 15:25:30 bd-cpu18 kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, id=0008(Requester ID)
Dec 3 15:25:30 bd-cpu18 kernel: pcieport 0000:00:01.0: device [8086:1901] error status/mask=00004000/00000000
Dec 3 15:25:30 bd-cpu18 kernel: pcieport 0000:00:01.0: [14] Completion Timeout
Dec 3 15:25:30 bd-cpu18 kernel: sis8300 0000:06:00.0: [aer_error_detected] called .. state=2
Dec 3 15:25:30 bd-cpu18 kernel: sis8300 0000:08:00.0: [aer_error_detected] called .. state=2
Dec 3 15:25:30 bd-cpu18 kernel: sis8300 0000:09:00.0: [aer_error_detected] called .. state=2
Dec 3 15:25:30 bd-cpu18 kernel: sis8300 0000:0b:00.0: [aer_error_detected] called .. state=2
Dec 3 15:25:30 bd-cpu18 kernel: mrf-pci 0000:0e:00.0: [aer_error_detected] called .. state=2
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:06:00.0: [aer_result_none] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:08:00.0: [aer_result_none] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:09:00.0: [aer_result_none] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:0b:00.0: [aer_result_none] called..
Dec 3 15:25:31 bd-cpu18 kernel: mrf-pci 0000:0e:00.0: [aer_result_none] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:06:00.0: [aer_resume] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:06:00.0: [aer_resume] UC errors cleared!
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:08:00.0: [aer_resume] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:08:00.0: [aer_resume] UC errors cleared!
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:09:00.0: [aer_resume] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:09:00.0: [aer_resume] UC errors cleared!
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:0b:00.0: [aer_resume] called..
Dec 3 15:25:31 bd-cpu18 kernel: sis8300 0000:0b:00.0: [aer_resume] UC errors cleared!
Dec 3 15:25:31 bd-cpu18 kernel: mrf-pci 0000:0e:00.0: [aer_resume] called..
Dec 3 15:25:31 bd-cpu18 kernel: pcieport 0000:00:01.0: AER: Device recovery successful
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:08:00.0: [sis8300_open] available = 1
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:08:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:06:00.0: [sis8300_open] available = 1
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:06:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:09:00.0: [sis8300_open] available = 1
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:09:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:0b:00.0: [sis8300_open] available = 1
Dec 3 15:25:33 bd-cpu18 kernel: sis8300 0000:0b:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:08:00.0: [sis8300_open] available = 1
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:08:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:06:00.0: [sis8300_open] available = 1
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:06:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:09:00.0: [sis8300_open] available = 1
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:09:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:0b:00.0: [sis8300_open] available = 1
Dec 3 15:25:34 bd-cpu18 kernel: sis8300 0000:0b:00.0: [sis8300_release] available = 1, count = 0
Dec 3 15:25:47 bd-cpu18 dbus[1266]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Dec 3 15:25:47 bd-cpu18 dbus[1266]: [system] Successfully activated service 'org.freedesktop.problems'
At this point the PCI space reads for the AMCs returns 0xFFFFFFFF.
Below are messages captured after issuing 'echo 1 > /sys/bus/pci/rescan'. After that the CPU card rebooted by itself.
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:01:00.1: Max Payload Size set to 256 (was 128, max 1024)
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:01:00.2: Max Payload Size set to 256 (was 128, max 1024)
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:01:00.3: Max Payload Size set to 256 (was 128, max 1024)
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:01:00.4: Max Payload Size set to 256 (was 128, max 1024)
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:02:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:02:09.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pci 0000:02:0a.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:03:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:09.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:0a.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:0b.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:10.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:11.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: pcieport 0000:04:12.0: bridge configuration invalid ([bus 00-00]), reconfiguring
Dec 3 15:25:59 bd-cpu18 kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Dec 3 15:25:59 bd-cpu18 kernel: IP: [<ffffffffc0b182f7>] aer_inj_read_config+0x87/0x160 [aer_inject]
Dec 3 15:25:59 bd-cpu18 kernel: PGD 80000001767c4067 PUD 431939067 PMD 0
Dec 3 15:25:59 bd-cpu18 kernel: Oops: 0000 [#1] SMP
Dec 3 15:25:59 bd-cpu18 kernel: Modules linked in: aer_inject sis8300drv(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 devlink tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support mei_wdt intel_wmi_thunderbolt intel_pmc_core snd_hda_codec_hdmi intel_powerclamp coretemp intel_rapl kvm_intel snd_hda_intel snd_hda_codec kvm snd_hda_core irqbypass crc32_pclmul snd_hwdep ghash_clmulni_intel snd_seq snd_seq_device aesni_intel lrw gf128mul snd_pcm glue_helper ablk_helper cryptd pcspkr snd_timer snd i2c_i801 soundcore sg i2c_designware_platform i2c_designware_core mei_me mei ie31200_edac
Dec 3 15:25:59 bd-cpu18 kernel: wmi pinctrl_sunrisepoint pinctrl_intel tpm_crb acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i915 iosf_mbi drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul igb libahci crct10dif_common crc32c_intel drm libata serio_raw ptp pps_core dca i2c_algo_bit drm_panel_orientation_quirks mrf(OE) parport uio i2c_hid video [last unloaded: sis8300drv]
Dec 3 15:25:59 bd-cpu18 kernel: CPU: 4 PID: 8150 Comm: bash Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.6.1.el7.x86_64.debug #1
Dec 3 15:25:59 bd-cpu18 kernel: Hardware name: AMI AM G6x/msd/AM G6x/msd, BIOS 4.08.01 02/19/2019
Dec 3 15:25:59 bd-cpu18 kernel: task: ffff9b22b5f88000 ti: ffff9b22b5bc4000 task.ti: ffff9b22b5bc4000
Dec 3 15:25:59 bd-cpu18 kernel: RIP: 0010:[<ffffffffc0b182f7>] [<ffffffffc0b182f7>] aer_inj_read_config+0x87/0x160 [aer_inject]
Dec 3 15:25:59 bd-cpu18 kernel: RSP: 0018:ffff9b22b5bc7a30 EFLAGS: 00010046
Dec 3 15:25:59 bd-cpu18 kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004
Dec 3 15:25:59 bd-cpu18 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b25718ab000
Dec 3 15:25:59 bd-cpu18 kernel: RBP: ffff9b22b5bc7a68 R08: ffff9b22b5bc7a84 R09: ffffffffc0b1a0b0
Dec 3 15:25:59 bd-cpu18 kernel: R10: 0000000000000001 R11: 69c1fefdd26da26d R12: 0000000000000000
Dec 3 15:25:59 bd-cpu18 kernel: R13: ffffffffc0b1a050 R14: ffff9b22b5bc7a84 R15: ffff9b25718ab000
Dec 3 15:25:59 bd-cpu18 kernel: FS: 00007f3d00e8b740(0000) GS:ffff9b259ce00000(0000) knlGS:0000000000000000
Dec 3 15:25:59 bd-cpu18 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 3 15:25:59 bd-cpu18 kernel: CR2: 0000000000000000 CR3: 0000000175796000 CR4: 00000000003607e0
Dec 3 15:25:59 bd-cpu18 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 3 15:25:59 bd-cpu18 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 3 15:25:59 bd-cpu18 kernel: Call Trace:
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c61a9c>] pci_bus_read_config_dword+0x8c/0xb0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c63a68>] pci_bus_read_dev_vendor_id+0x28/0xe0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65eba>] pci_scan_slot+0x3a/0x140
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c6711d>] pci_scan_child_bus_extend+0x4d/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c62bcb>] ? pci_bus_write_config_dword+0x6b/0x80
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f4b>] pci_scan_bridge_extend+0x47b/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] ? pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c67286>] pci_scan_child_bus_extend+0x1b6/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f4b>] pci_scan_bridge_extend+0x47b/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c67286>] pci_scan_child_bus_extend+0x1b6/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c673d0>] pci_scan_child_bus+0x10/0x20
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f01>] pci_scan_bridge_extend+0x431/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] ? pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c671e7>] pci_scan_child_bus_extend+0x117/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c674b6>] pci_rescan_bus+0x16/0x40
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c70d78>] bus_rescan_store+0x78/0xa0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2d56909>] bus_attr_store+0x29/0x30
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2b5272a>] sysfs_kf_write+0x4a/0x60
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2b51ff6>] kernfs_fop_write+0x106/0x190
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2aaf3fc>] vfs_write+0xdc/0x240
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2ad5984>] ? fget_light+0x3c4/0x550
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2ab029a>] SyS_write+0x8a/0x100
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3098b12>] system_call_fastpath+0x25/0x2a
Dec 3 15:25:59 bd-cpu18 kernel: Code: 81 f9 b0 a0 b1 c0 74 5c 4d 3b 79 10 75 ee 49 8b 41 18 4d 8b af b8 00 00 00 89 de 49 89 87 b8 00 00 00 4d 89 f0 44 89 e2 4c 89 ff <48> 8b 00 e8 41 f3 10 e2 48 8b 75 d0 89 c3 4d 89 af b8 00 00 00
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.711+0000: 2132: info : libvirt version: 4.5.0, package: 23.el7_7.6 (CentOS BuildSystem <http://bugs.centos.org>, 2020-03-17-23:39:10, x86-01.bsys.centos.org)
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.711+0000: 2132: info : hostname: bd-cpu18.cslab.esss.lu.se
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.711+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:01:00.1 not found: could not access /sys/bus/pci/devices/0000:01:00.1/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.713+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:01:00.2 not found: could not access /sys/bus/pci/devices/0000:01:00.2/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.714+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:01:00.3 not found: could not access /sys/bus/pci/devices/0000:01:00.3/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.714+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:01:00.4 not found: could not access /sys/bus/pci/devices/0000:01:00.4/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.715+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:02:02.0 not found: could not access /sys/bus/pci/devices/0000:02:02.0/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.715+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:02:09.0 not found: could not access /sys/bus/pci/devices/0000:02:09.0/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.715+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:02:08.0 not found: could not access /sys/bus/pci/devices/0000:02:08.0/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 libvirtd: 2020-12-03 14:25:59.716+0000: 2132: error : virPCIDeviceNew:1784 : Device 0000:02:0a.0 not found: could not access /sys/bus/pci/devices/0000:02:0a.0/config: No such file or directory
Dec 3 15:25:59 bd-cpu18 kernel: RIP [<ffffffffc0b182f7>] aer_inj_read_config+0x87/0x160 [aer_inject]
Dec 3 15:25:59 bd-cpu18 kernel: RSP <ffff9b22b5bc7a30>
Dec 3 15:25:59 bd-cpu18 kernel: CR2: 0000000000000000
Dec 3 15:25:59 bd-cpu18 kernel: ---[ end trace 452aceb2952fc499 ]---
Dec 3 15:25:59 bd-cpu18 kernel: BUG: sleeping function called from invalid context at kernel/rwsem.c:21
Dec 3 15:25:59 bd-cpu18 kernel: in_atomic(): 1, irqs_disabled(): 1, pid: 8150, name: bash
Dec 3 15:25:59 bd-cpu18 kernel: INFO: lockdep is turned off.
Dec 3 15:25:59 bd-cpu18 kernel: irq event stamp: 87980
Dec 3 15:25:59 bd-cpu18 kernel: hardirqs last enabled at (87979): [<ffffffffa308bcd6>] _raw_spin_unlock_irqrestore+0x36/0x70
Dec 3 15:25:59 bd-cpu18 kernel: hardirqs last disabled at (87980): [<ffffffffa308c55b>] _raw_spin_lock_irqsave+0x2b/0xa0
Dec 3 15:25:59 bd-cpu18 kernel: softirqs last enabled at (87300): [<ffffffffa28ba134>] __do_softirq+0x1a4/0x470
Dec 3 15:25:59 bd-cpu18 kernel: softirqs last disabled at (87281): [<ffffffffa309c2bc>] call_softirq+0x1c/0x30
Dec 3 15:25:59 bd-cpu18 kernel: CPU: 4 PID: 8150 Comm: bash Kdump: loaded Tainted: G D OE ------------ 3.10.0-1160.6.1.el7.x86_64.debug #1
Dec 3 15:25:59 bd-cpu18 kernel: Hardware name: AMI AM G6x/msd/AM G6x/msd, BIOS 4.08.01 02/19/2019
Dec 3 15:25:59 bd-cpu18 kernel: Call Trace:
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3080c37>] dump_stack+0x19/0x1b
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28f500b>] __might_sleep+0x17b/0x240
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3087aba>] down_read+0x2a/0xb0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28ccad3>] exit_signals+0x33/0x150
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28b5f6c>] do_exit+0xcc/0xb80
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28b2390>] ? kmsg_dump+0x130/0x1d0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28b2294>] ? kmsg_dump+0x34/0x1d0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa308f910>] oops_end+0xa0/0xe0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28836a4>] no_context+0x144/0x310
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa293eac3>] ? __bfs+0x103/0x280
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa288398a>] __bad_area_nosemaphore+0x11a/0x230
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2883ab4>] bad_area_nosemaphore+0x14/0x20
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3092ce8>] __do_page_fault+0x358/0x5a0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2944229>] ? __lock_acquire+0xa49/0x1630
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3092f65>] do_page_fault+0x35/0x90
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa308e818>] page_fault+0x28/0x30
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffc0b182f7>] ? aer_inj_read_config+0x87/0x160 [aer_inject]
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c61a9c>] pci_bus_read_config_dword+0x8c/0xb0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c63a68>] pci_bus_read_dev_vendor_id+0x28/0xe0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65eba>] pci_scan_slot+0x3a/0x140
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c6711d>] pci_scan_child_bus_extend+0x4d/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c62bcb>] ? pci_bus_write_config_dword+0x6b/0x80
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f4b>] pci_scan_bridge_extend+0x47b/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] ? pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c67286>] pci_scan_child_bus_extend+0x1b6/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f4b>] pci_scan_bridge_extend+0x47b/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c67286>] pci_scan_child_bus_extend+0x1b6/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c673d0>] pci_scan_child_bus+0x10/0x20
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f01>] pci_scan_bridge_extend+0x431/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] ? pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c671e7>] pci_scan_child_bus_extend+0x117/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c674b6>] pci_rescan_bus+0x16/0x40
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c70d78>] bus_rescan_store+0x78/0xa0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2d56909>] bus_attr_store+0x29/0x30
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2b5272a>] sysfs_kf_write+0x4a/0x60
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2b51ff6>] kernfs_fop_write+0x106/0x190
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2aaf3fc>] vfs_write+0xdc/0x240
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2ad5984>] ? fget_light+0x3c4/0x550
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2ab029a>] SyS_write+0x8a/0x100
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3098b12>] system_call_fastpath+0x25/0x2a
Dec 3 15:25:59 bd-cpu18 kernel: BUG: scheduling while atomic: bash/8150/0x10000003
Dec 3 15:25:59 bd-cpu18 kernel: INFO: lockdep is turned off.
Dec 3 15:25:59 bd-cpu18 kernel: Modules linked in: aer_inject sis8300drv(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 devlink tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support mei_wdt intel_wmi_thunderbolt intel_pmc_core snd_hda_codec_hdmi intel_powerclamp coretemp intel_rapl kvm_intel snd_hda_intel snd_hda_codec kvm snd_hda_core irqbypass crc32_pclmul snd_hwdep ghash_clmulni_intel snd_seq snd_seq_device aesni_intel lrw gf128mul snd_pcm glue_helper ablk_helper cryptd pcspkr snd_timer snd i2c_i801 soundcore sg i2c_designware_platform i2c_designware_core mei_me mei ie31200_edac
Dec 3 15:25:59 bd-cpu18 kernel: wmi pinctrl_sunrisepoint pinctrl_intel tpm_crb acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i915 iosf_mbi drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul igb libahci crct10dif_common crc32c_intel drm libata serio_raw ptp pps_core dca i2c_algo_bit drm_panel_orientation_quirks mrf(OE) parport uio i2c_hid video [last unloaded: sis8300drv]
Dec 3 15:25:59 bd-cpu18 kernel: irq event stamp: 87980
Dec 3 15:25:59 bd-cpu18 kernel: hardirqs last enabled at (87979): [<ffffffffa308bcd6>] _raw_spin_unlock_irqrestore+0x36/0x70
Dec 3 15:25:59 bd-cpu18 kernel: hardirqs last disabled at (87980): [<ffffffffa308c55b>] _raw_spin_lock_irqsave+0x2b/0xa0
Dec 3 15:25:59 bd-cpu18 kernel: softirqs last enabled at (87300): [<ffffffffa28ba134>] __do_softirq+0x1a4/0x470
Dec 3 15:25:59 bd-cpu18 kernel: softirqs last disabled at (87281): [<ffffffffa309c2bc>] call_softirq+0x1c/0x30
Dec 3 15:25:59 bd-cpu18 kernel: CPU: 4 PID: 8150 Comm: bash Kdump: loaded Tainted: G D OE ------------ 3.10.0-1160.6.1.el7.x86_64.debug #1
Dec 3 15:25:59 bd-cpu18 kernel: Hardware name: AMI AM G6x/msd/AM G6x/msd, BIOS 4.08.01 02/19/2019
Dec 3 15:25:59 bd-cpu18 kernel: Call Trace:
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3080c37>] dump_stack+0x19/0x1b
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3078933>] __schedule_bug+0x7d/0x8c
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa30887ec>] __schedule+0x7cc/0x810
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28f98e6>] __cond_resched+0x26/0x30
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3088b7a>] _cond_resched+0x3a/0x50
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3087abf>] down_read+0x2f/0xb0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28ccad3>] exit_signals+0x33/0x150
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28b5f6c>] do_exit+0xcc/0xb80
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28b2390>] ? kmsg_dump+0x130/0x1d0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28b2294>] ? kmsg_dump+0x34/0x1d0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa308f910>] oops_end+0xa0/0xe0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa28836a4>] no_context+0x144/0x310
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa293eac3>] ? __bfs+0x103/0x280
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa288398a>] __bad_area_nosemaphore+0x11a/0x230
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2883ab4>] bad_area_nosemaphore+0x14/0x20
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3092ce8>] __do_page_fault+0x358/0x5a0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2944229>] ? __lock_acquire+0xa49/0x1630
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3092f65>] do_page_fault+0x35/0x90
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa308e818>] page_fault+0x28/0x30
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffc0b182f7>] ? aer_inj_read_config+0x87/0x160 [aer_inject]
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c61a9c>] pci_bus_read_config_dword+0x8c/0xb0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c63a68>] pci_bus_read_dev_vendor_id+0x28/0xe0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65eba>] pci_scan_slot+0x3a/0x140
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c6711d>] pci_scan_child_bus_extend+0x4d/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c62bcb>] ? pci_bus_write_config_dword+0x6b/0x80
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f4b>] pci_scan_bridge_extend+0x47b/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] ? pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c67286>] pci_scan_child_bus_extend+0x1b6/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f4b>] pci_scan_bridge_extend+0x47b/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c67286>] pci_scan_child_bus_extend+0x1b6/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c673d0>] pci_scan_child_bus+0x10/0x20
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c66f01>] pci_scan_bridge_extend+0x431/0x5e0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c65e04>] ? pci_scan_single_device+0x74/0xf0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c671e7>] pci_scan_child_bus_extend+0x117/0x2f0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c674b6>] pci_rescan_bus+0x16/0x40
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2c70d78>] bus_rescan_store+0x78/0xa0
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2d56909>] bus_attr_store+0x29/0x30
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2b5272a>] sysfs_kf_write+0x4a/0x60
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2b51ff6>] kernfs_fop_write+0x106/0x190
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2aaf3fc>] vfs_write+0xdc/0x240
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2ad5984>] ? fget_light+0x3c4/0x550
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa2ab029a>] SyS_write+0x8a/0x100
Dec 3 15:25:59 bd-cpu18 kernel: [<ffffffffa3098b12>] system_call_fastpath+0x25/0x2a
Dec 3 15:25:59 bd-cpu18 kernel: note: bash[8150] exited with preempt_count 2
Dec 3 15:26:00 bd-cpu18 sh: abrt-dump-oops: Found oopses: 2
Dec 3 15:26:00 bd-cpu18 sh: abrt-dump-oops: Creating problem directories
Dec 3 15:26:00 bd-cpu18 sh: abrt-dump-oops: Not going to make dump directories world readable because PrivateReports is on
Dec 3 15:26:01 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:01 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:02 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:02 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:02 bd-cpu18 abrt-server: Looking for kernel package
Dec 3 15:26:02 bd-cpu18 abrt-dump-oops: Reported 2 kernel oopses to Abrt
Dec 3 15:26:02 bd-cpu18 abrt-server: Kernel package kernel-debug-3.10.0-1160.6.1.el7.x86_64 found
Dec 3 15:26:04 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:04 bd-cpu18 abrt-server: Email address of sender was not specified. Would you like to do so now? If not, 'user@localhost' is to be used [y/N]
Dec 3 15:26:04 bd-cpu18 abrt-server: Email address of receiver was not specified. Would you like to do so now? If not, 'root@localhost' is to be used [y/N]
Dec 3 15:26:04 bd-cpu18 abrt-server: Sending an email...
Dec 3 15:26:04 bd-cpu18 abrt-server: Sending a notification email to: root@localhost
Dec 3 15:26:04 bd-cpu18 abrt-server: Email was sent to: root@localhost
Dec 3 15:26:05 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:05 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:05 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:06 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:06 bd-cpu18 abrt-server: Looking for kernel package
Dec 3 15:26:06 bd-cpu18 abrt-server: Kernel package kernel-debug-3.10.0-1160.6.1.el7.x86_64 found
Dec 3 15:26:07 bd-cpu18 rtkit-daemon[1338]: The canary thread is apparently starving. Taking action.
Dec 3 15:26:07 bd-cpu18 rtkit-daemon[1338]: Demoting known real-time threads.
Dec 3 15:26:07 bd-cpu18 rtkit-daemon[1338]: Successfully demoted thread 2594 of process 2591 (/usr/bin/pulseaudio).
Dec 3 15:26:07 bd-cpu18 rtkit-daemon[1338]: Successfully demoted thread 2591 of process 2591 (/usr/bin/pulseaudio).
Dec 3 15:26:07 bd-cpu18 rtkit-daemon[1338]: Demoted 2 threads.
Dec 3 15:26:08 bd-cpu18 abrt-server: '/var/spool/abrt/oops-2020-12-03-14:42:51-4266-0' is not a problem directory
Dec 3 15:26:08 bd-cpu18 abrt-server: Email address of sender was not specified. Would you like to do so now? If not, 'user@localhost' is to be used [y/N]
Dec 3 15:26:08 bd-cpu18 abrt-server: Email address of receiver was not specified. Would you like to do so now? If not, 'root@localhost' is to be used [y/N]
Dec 3 15:26:08 bd-cpu18 abrt-server: Sending an email...
Dec 3 15:26:08 bd-cpu18 abrt-server: Sending a notification email to: root@localhost
I guess this is the root cause of oops:
Dec 3 15:25:59 bd-cpu18 kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Dec 3 15:25:59 bd-cpu18 kernel: IP: [<ffffffffc0b182f7>] aer_inj_read_config+0x87/0x160 [aer_inject]
Looking at the actual source code of the running kernel this is the source of the offending aer_inj_read_config():
static int aer_inj_read_config(struct pci_bus *bus, unsigned int devfn,
int where, int size, u32 *val)
{
u32 *sim;
struct aer_error *err;
unsigned long flags;
struct pci_ops *ops;
struct pci_ops *my_ops;
int domain;
int rv;
spin_lock_irqsave(&inject_lock, flags);
if (size != sizeof(u32))
goto out;
domain = pci_domain_nr(bus);
if (domain < 0)
goto out;
err = __find_aer_error(domain, bus->number, devfn);
if (!err)
goto out;
sim = find_pci_config_dword(err, where, NULL);
if (sim) {
*val = *sim;
spin_unlock_irqrestore(&inject_lock, flags);
return 0;
}
out:
ops = __find_pci_bus_ops(bus);
/*
* pci_lock must already be held, so we can directly
* manipulate bus->ops. Many config access functions,
* including pci_generic_config_read() require the original
* bus->ops be installed to function, so temporarily put them
* back.
*/
my_ops = bus->ops;
bus->ops = ops;
rv = ops->read(bus, devfn, where, size, val);
bus->ops = my_ops;
spin_unlock_irqrestore(&inject_lock, flags);
return rv;
}
Here is 'objdump -D aer_inject.ko':
0000000000000270 <aer_inj_read_config>:
270: e8 00 00 00 00 callq 275 <aer_inj_read_config+0x5>
275: 55 push %rbp
276: 48 89 e5 mov %rsp,%rbp
279: 41 57 push %r15
27b: 49 89 ff mov %rdi,%r15
27e: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
285: 41 56 push %r14
287: 4d 89 c6 mov %r8,%r14
28a: 41 55 push %r13
28c: 41 54 push %r12
28e: 41 89 d4 mov %edx,%r12d
291: 53 push %rbx
292: 89 f3 mov %esi,%ebx
294: 48 83 ec 10 sub $0x10,%rsp
298: 89 4d cc mov %ecx,-0x34(%rbp)
29b: e8 00 00 00 00 callq 2a0 <aer_inj_read_config+0x30>
2a0: 8b 4d cc mov -0x34(%rbp),%ecx
2a3: 48 89 45 d0 mov %rax,-0x30(%rbp)
2a7: 83 f9 04 cmp $0x4,%ecx
2aa: 0f 84 88 00 00 00 je 338 <aer_inj_read_config+0xc8>
2b0: 4c 8b 0d 00 00 00 00 mov 0x0(%rip),%r9 # 2b7 <aer_inj_read_config+0x47>
2b7: 49 81 f9 00 00 00 00 cmp $0x0,%r9
2be: 75 14 jne 2d4 <aer_inj_read_config+0x64>
2c0: eb 6e jmp 330 <aer_inj_read_config+0xc0>
2c2: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
2c8: 4d 8b 09 mov (%r9),%r9
2cb: 49 81 f9 00 00 00 00 cmp $0x0,%r9
2d2: 74 5c je 330 <aer_inj_read_config+0xc0>
2d4: 4d 3b 79 10 cmp 0x10(%r9),%r15
2d8: 75 ee jne 2c8 <aer_inj_read_config+0x58>
2da: 49 8b 41 18 mov 0x18(%r9),%rax
2de: 4d 8b af b8 00 00 00 mov 0xb8(%r15),%r13
2e5: 89 de mov %ebx,%esi
2e7: 49 89 87 b8 00 00 00 mov %rax,0xb8(%r15)
2ee: 4d 89 f0 mov %r14,%r8
2f1: 44 89 e2 mov %r12d,%edx
2f4: 4c 89 ff mov %r15,%rdi
2f7: 48 8b 00 mov (%rax),%rax
2fa: e8 00 00 00 00 callq 2ff <aer_inj_read_config+0x8f>
2ff: 48 8b 75 d0 mov -0x30(%rbp),%rsi
303: 89 c3 mov %eax,%ebx
305: 4d 89 af b8 00 00 00 mov %r13,0xb8(%r15)
30c: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
313: e8 00 00 00 00 callq 318 <aer_inj_read_config+0xa8>
318: 89 d8 mov %ebx,%eax
31a: 48 83 c4 10 add $0x10,%rsp
31e: 5b pop %rbx
31f: 41 5c pop %r12
321: 41 5d pop %r13
323: 41 5e pop %r14
325: 41 5f pop %r15
327: 5d pop %rbp
328: c3 retq
329: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
330: 31 c0 xor %eax,%eax
332: eb aa jmp 2de <aer_inj_read_config+0x6e>
334: 0f 1f 40 00 nopl 0x0(%rax)
338: 49 8b 87 c8 00 00 00 mov 0xc8(%r15),%rax
33f: 8b 00 mov (%rax),%eax
341: 85 c0 test %eax,%eax
343: 0f 88 67 ff ff ff js 2b0 <aer_inj_read_config+0x40>
349: 48 8b 3d 00 00 00 00 mov 0x0(%rip),%rdi # 350 <aer_inj_read_config+0xe0>
350: 41 0f b6 97 d8 00 00 movzbl 0xd8(%r15),%edx
357: 00
358: 48 81 ff 00 00 00 00 cmp $0x0,%rdi
35f: 0f 84 4b ff ff ff je 2b0 <aer_inj_read_config+0x40>
365: 3b 47 10 cmp 0x10(%rdi),%eax
368: 74 1b je 385 <aer_inj_read_config+0x115>
36a: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
370: 48 8b 3f mov (%rdi),%rdi
373: 48 81 ff 00 00 00 00 cmp $0x0,%rdi
37a: 0f 84 30 ff ff ff je 2b0 <aer_inj_read_config+0x40>
380: 3b 47 10 cmp 0x10(%rdi),%eax
383: 75 eb jne 370 <aer_inj_read_config+0x100>
385: 3b 57 14 cmp 0x14(%rdi),%edx
388: 75 e6 jne 370 <aer_inj_read_config+0x100>
38a: 3b 5f 18 cmp 0x18(%rdi),%ebx
38d: 75 e1 jne 370 <aer_inj_read_config+0x100>
38f: 48 85 ff test %rdi,%rdi
392: 0f 84 18 ff ff ff je 2b0 <aer_inj_read_config+0x40>
398: 31 d2 xor %edx,%edx
39a: 44 89 e6 mov %r12d,%esi
39d: 89 4d cc mov %ecx,-0x34(%rbp)
3a0: e8 5b fc ff ff callq 0 <find_pci_config_dword>
3a5: 48 85 c0 test %rax,%rax
3a8: 8b 4d cc mov -0x34(%rbp),%ecx
3ab: 0f 84 ff fe ff ff je 2b0 <aer_inj_read_config+0x40>
3b1: 8b 00 mov (%rax),%eax
3b3: 48 8b 75 d0 mov -0x30(%rbp),%rsi
3b7: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
3be: 41 89 06 mov %eax,(%r14)
3c1: e8 00 00 00 00 callq 3c6 <aer_inj_read_config+0x156>
3c6: 31 c0 xor %eax,%eax
3c8: e9 4d ff ff ff jmpq 31a <aer_inj_read_config+0xaa>
3cd: 0f 1f 00 nopl (%rax)
Thank you!
