On Sat, Apr 28, 2018 at 02:37:20PM +1000, Michael Neuling wrote:
> commit f0295e047fcf52ccb42561fb7de6942f5201b676 upstream.
>
> The current EEH callbacks can race with a driver unbind. This can
> result in a backtraces like this:
>
> EEH: Frozen PHB#0-PE#1fc detected
> EEH: PE location: S000009, PHB location: N/A
> CPU: 2 PID: 2312 Comm: kworker/u258:3 Not tainted 4.15.6-openpower1 #2
> Workqueue: nvme-wq nvme_reset_work [nvme]
> Call Trace:
> dump_stack+0x9c/0xd0 (unreliable)
> eeh_dev_check_failure+0x420/0x470
> eeh_check_failure+0xa0/0xa4
> nvme_reset_work+0x138/0x1414 [nvme]
> process_one_work+0x1ec/0x328
> worker_thread+0x2e4/0x3a8
> kthread+0x14c/0x154
> ret_from_kernel_thread+0x5c/0xc8
> nvme nvme1: Removing after probe failure status: -19
> <snip>
> cpu 0x23: Vector: 300 (Data Access) at [c000000ff50f3800]
> pc: c0080000089a0eb0: nvme_error_detected+0x4c/0x90 [nvme]
> lr: c000000000026564: eeh_report_error+0xe0/0x110
> sp: c000000ff50f3a80
> msr: 9000000000009033
> dar: 400
> dsisr: 40000000
> current = 0xc000000ff507c000
> paca = 0xc00000000fdc9d80 softe: 0 irq_happened: 0x01
> pid = 782, comm = eehd
> Linux version 4.15.6-openpower1 (smc@smc-desktop) (gcc version 6.4.0 (Buildroot 2017.11.2-00008-g4b6188e)) #2 SM P Tue Feb 27 12:33:27 PST 2018
> enter ? for help
> eeh_report_error+0xe0/0x110
> eeh_pe_dev_traverse+0xc0/0xdc
> eeh_handle_normal_event+0x184/0x4c4
> eeh_handle_event+0x30/0x288
> eeh_event_handler+0x124/0x170
> kthread+0x14c/0x154
> ret_from_kernel_thread+0x5c/0xc8
>
> The first part is an EEH (on boot), the second half is the resulting
> crash. nvme probe starts the nvme_reset_work() worker thread. This
> worker thread starts touching the device which see a device error
> (EEH) and hence queues up an event in the powerpc EEH worker
> thread. nvme_reset_work() then continues and runs
> nvme_remove_dead_ctrl_work() which results in unbinding the driver
> from the device and hence releases all resources. At the same time,
> the EEH worker thread starts doing the EEH .error_detected() driver
> callback, which no longer works since the resources have been freed.
>
> This fixes the problem in the same way the generic PCIe AER code (in
> drivers/pci/pcie/aer/aerdrv_core.c) does. It makes the EEH code hold
> the device_lock() while performing the driver EEH callbacks and
> associated code. This ensures either the callbacks are no longer
> register, or if they are registered the driver will not be removed
> from underneath us.
>
> This has been broken forever. The EEH call backs were first introduced
> in 2005 (in 77bd7415610) but it's not clear if a lock was needed back
> then.
>
> Fixes: 77bd74156101 ("[PATCH] powerpc: PCI Error Recovery: PPC64 core recovery routines")
> Cc: stable@xxxxxxxxxxxxxxx # v4.9, v4.14
> Signed-off-by: Michael Neuling <mikey@xxxxxxxxxxx>
> Reviewed-by: Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx>
> Signed-off-by: Michael Ellerman <mpe@xxxxxxxxxxxxxx>
>
> ---
> Greg,
>
> > And it breaks the build on the 4.14.y tree:
> > arch/powerpc/kernel/eeh_driver.c: In function 'eeh_report_resume':
> > arch/powerpc/kernel/eeh_driver.c:395:13: error: 'struct eeh_ops' has no member named 'notify_resume'
>
> This fixes the build issue from the last try.
>
> Sorry again,
not a problem, thanks for the fixup, I'll queue this up once these
latest stable kernels get released in a few days.
greg k-h