On 18.8.2024 23.56, FPS wrote:
Hi! I hope this mail is not off-topic for this list :) I have recently acquired a mini PC equipped with an N97 CPU for use in my electric guitar setup (I have good experiences with my previous Intel Celeron J4125-based mini PC), but sadly I can't get it to work reliably with ALSA/Jack using a USB audio class 2.0 sound card. The system is running a decently setup rt-kernel: Linux ogfx97 6.6.44-rt39 #1-NixOS SMP PREEMPT_RT Sat Aug 3 06:54:42 UTC 2024 x86_64 GNU/Linux Cyclictest gives sufficiently nice results that give me hope that it should be possible to use this system for realtime-audio. This is just a short example run but it's quite representative: [fps@ogfx97:~]$ sudo cyclictest -m -p 90 -S # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 2.07 1.36 0.74 2/644 9634 T: 0 ( 9615) P:90 I:1000 C: 8715 Min: 1 Act: 1 Avg: 2 Max: 14 T: 1 ( 9616) P:90 I:1500 C: 5808 Min: 1 Act: 5 Avg: 2 Max: 28 T: 2 ( 9617) P:90 I:2000 C: 4355 Min: 1 Act: 1 Avg: 2 Max: 15 T: 3 ( 9618) P:90 I:2500 C: 3483 Min: 1 Act: 1 Avg: 3 Max: 24 As usual I have elevated the xhci and ahci IRQ kernel threads to SCHED_FIFO priority 90 and done some other shenanigans like idle=poll, disabling C-states in the UEFI firmware, etc.. The xhci in this system is: 00:14.0 USB controller: Intel Corporation Alder Lake-N PCH USB 3.2 xHCI Host Controller (prog-if 30 [XHCI]) DeviceName: Onboard - Other Subsystem: Intel Corporation Device 7270 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 124 Region 0: Memory at 6001100000 (64-bit, non-prefetchable) [size=64K] Capabilities: <access denied> Kernel driver in use: xhci_hcd Kernel modules: xhci_pci Sadly there's some built-in junk on the USB besides my keyboard, mouse and audio interface: [fps@ogfx97:~]$ lsusb -tv /: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/12p, 480M ID 1d6b:0002 Linux Foundation 2.0 root hub |__ Port 001: Dev 004, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M ID 046d:c31c Logitech, Inc. Keyboard K120 |__ Port 001: Dev 004, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M ID 046d:c31c Logitech, Inc. Keyboard K120 |__ Port 002: Dev 005, If 0, Class=Human Interface Device, Driver=usbhid, 12M ID 046d:c07d Logitech, Inc. G502 Mouse |__ Port 002: Dev 005, If 1, Class=Human Interface Device, Driver=usbhid, 12M ID 046d:c07d Logitech, Inc. G502 Mouse |__ Port 004: Dev 014, If 0, Class=Audio, Driver=snd-usb-audio, 480M ID 1235:8202 Focusrite-Novation Focusrite Scarlett 2i2 2nd Gen |__ Port 004: Dev 014, If 1, Class=Audio, Driver=snd-usb-audio, 480M ID 1235:8202 Focusrite-Novation Focusrite Scarlett 2i2 2nd Gen |__ Port 004: Dev 014, If 2, Class=Audio, Driver=snd-usb-audio, 480M ID 1235:8202 Focusrite-Novation Focusrite Scarlett 2i2 2nd Gen |__ Port 004: Dev 014, If 3, Class=Vendor Specific Class, Driver=[none], 480M ID 1235:8202 Focusrite-Novation Focusrite Scarlett 2i2 2nd Gen |__ Port 005: Dev 002, If 0, Class=Wireless, Driver=[none], 12M ID 0bda:c821 Realtek Semiconductor Corp. |__ Port 005: Dev 002, If 1, Class=Wireless, Driver=[none], 12M ID 0bda:c821 Realtek Semiconductor Corp. |__ Port 006: Dev 003, If 0, Class=Audio, Driver=snd-usb-audio, 12M ID 0573:1573 Zoran Co. Personal Media Division (Nogatech) |__ Port 006: Dev 003, If 1, Class=Audio, Driver=snd-usb-audio, 12M ID 0573:1573 Zoran Co. Personal Media Division (Nogatech) |__ Port 006: Dev 003, If 2, Class=Audio, Driver=snd-usb-audio, 12M ID 0573:1573 Zoran Co. Personal Media Division (Nogatech) |__ Port 006: Dev 003, If 3, Class=Human Interface Device, Driver=[none], 12M ID 0573:1573 Zoran Co. Personal Media Division (Nogatech) /: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M ID 1d6b:0003 Linux Foundation 3.0 root hub OK, with that out of the way, what's the problem? Well, with an USB audio class 2.0 device I would expect a maximum jitter of a microframe (125 us) regardless of the buffer size and I have used this audio interface successfully on different xHCI HCDs with buffer sizes as low as 24 samples at 48000 Hz sampling rate (0.5 ms period duration). But in this mini PC I cannot run such low period durations/buffer sizes at all. Sporadically I see delays of up to 3 or 4 ms - which makes it absolutely unusable for real-time audio stuff. It can't really be NMIs from what I can tell - they would show up in the cyclictest results. So I suspect the xHCI is doing something funky. Here's another observation: When I run the audio interface with 2 periods of 48 frames at 48000 Hz sampling rate I get constant Xruns from jack of about 0.004-0.008 ms. It seems that the xHCI is not willing to deliver all microframes in a timely manner but somehow wants a minimum time interval of 1 ms between them (I read the term interrupt moderation somewhere on the web - but I don't know if the linux xhci driver does something like that at all.)
The default interrupt moderation is 1ms for xHC hardware, but the xhci PCI driver should set it to 40 microseconds. Interupt moderation value can be checked from debugfs: # mount -t debugfs none /sys/kernel/debug/ # cat /sys/kernel/debug/usb/xhci/0000:00:14.0/reg-runtime MFINDEX = 0x00002570 IR0_IMAN = 0x00000002 IR0_IMOD = 0x000000a0 IR0_ERSTSZ = 0x00000002 IR0_ERSTBA_LOW = 0x05069000 IR0_ERSTBA_HIGH = 0x00000001 IR0_ERDP_LOW = 0x05067b80 IR0_ERDP_HIGH = 0x00000001 IR0_IMOD shows the interrupt moderation value in 250 nanosecond steps. i.e. 0xa0 is 160 * 250ns => 40000ns
How could I debug this further? All help appreciated :)
Was looking at a similar issue which turned out to be wifi driver debug blocking softirqs for a long time. This caused the the URBs that were given back to audio class driver in a tasklet (softirq) context to be delayed. This changed recently a bit. USB core now uses workqueues instead of tasklets, but issue could be similar. I'm not familiar with cyclictest. I was debugging using ftrace myslelf. maybe ftrace preemptoff could show if something is blocking for a long time? echo preemptoff > /sys/kernel/debug/tracing/current_tracer xhci dynamic debug could show if xHC controller has any issues processing TRBs on ring buffer in time. This debug should be limited to handle_tx_event() as dynamic debug causes some delays itself. If there are any issues on xHC side it should print underrun/overrun, or miss service interval errors. Thanks Mathias
