On Mon, 1 Nov 2021 17:35:04 +1300 Matthew Ruffell <matthew.ruffell@xxxxxxxxxxxxx> wrote: > Hi Alex, > > Nathan has been running a workload on the 5.14 kernel + the test patch, and has > ran into some interesting softlockups and hardlockups. > > The first, happened on a secondary server running a Windows VM, with 7 (of 10) > 1080TI GPUs passed through. > > Full dmesg: > https://paste.ubuntu.com/p/Wx5hCBBXKb/ > > There isn't any "irq x: nobody cared" messages, and the crashkernel gets stuck > in the usual copying IR tables from dmar, which suggests an ongoing interrupt > storm. > > Nathan disabled "kernel.hardlockup_panic = 1" sysctl, and managed to reproduce > the issue again, suggesting that we get stuck in kernel space for too long > without the ability for interrupts to be serviced. > > It starts with the NIC hitting a tx queue timeout, and then does a NMI to unwind > the stack of each CPU, although the stacks don't appear to indicate where things > are stuck. The server then remains softlocked, and keeps unwinding stacks every > 26 seconds or so, until it eventually hardlockups. Google finds numerous complaints about transmit queue time outs on igb devices, bad NICs, bad cabling, bad drivers(?). I also see some hearsay related specifically to supermicro compatibility. I'd also suspect that a dual 1GbE NIC is sub-par for anything involving 7+ GPUs. Time for an upgrade? It's not clear to me how this would be related to the GPU assignment perhaps other than the elevated workload on the host. > The next interesting thing to report is when Nathan started the same Windows VM > on the primary host we have been debugging on, with the 8x 2080TI GPUs. Nathan > experienced a stuck VM, with the host responding just fine. When Nathan reset > the VM, he got 4x "irq xx: nobody cared" messages on IRQs 25, 27, 29 and 31, > which at the time corresponded to the PEX 8747 upstream PCI switches. > > Interestingly, Nathan also observed 2x GPU Audio devices sharing the same IRQ > line as the upstream PCI switch, although Nathan mentioned this only occured > very briefly, and the GPU audio devices were re-assigned different IRQs shortly > afterward. IME, the legacy interrupt support on NVIDIA GPU audio devices is marginal for assignment. We don't claim to support assignment of the audio function, even for Quadro cards on RHEL due to this. I can't remember the details off the top of my head, but even with the hacky safeguards added in the test patch, we still rely on hardware to both honor the INTx disable bit in the command register and accurately report if the device is asserting INTx is the status register. It seems like one of these was a bit dicey in this controller. Now that I think about it more, I recall that the issue was predominantly with Linux guests, where the snd_intel_hda driver includes: /* quirks for Nvidia */ #define AZX_DCAPS_PRESET_NVIDIA \ (AZX_DCAPS_NO_MSI | AZX_DCAPS_CORBRP_SELF_CLEAR |\ AZX_DCAPS_SNOOP_TYPE(NVIDIA)) And the device table includes: { PCI_DEVICE(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID), .class = PCI_CLASS_MULTIMEDIA_HD_AUDIO << 8, .class_mask = 0xffffff, .driver_data = AZX_DRIVER_NVIDIA | AZX_DCAPS_PRESET_NVIDIA }, That NO_MSI quirk forces the sound driver to use legacy interrupts for all NVIDIA HD audio devices. I think this made audio function assignment to Linux guests essentially unusable without using the snd_hda_intel.enable_msi=1 driver option to re-enable MSI. Windows uses MSI for these devices, so it works better by default, but when we're resetting the VM we're still transitioning through this mode where I don't have a good opinion that the hardware behaves in a manageable way. My PCIe switch configuration with NVIDIA GPUs only has Tesla cards, so I don't have a way to reproduce this specific shared INTx issue, but it may be time to revisit examining the register behavior while running in INTx mode. Thanks, Alex
