Hello Oliver and others, two consecutive daily runs of our CAN latency system https://canbus.pages.fel.cvut.cz/#can-bus-channels-mutual-latency-testing shows extreme increase in latency of the kernel CAN gateway under the load. The first run with increased latency (run-DATE-TIME-KERNEL-OPTIONS) run-231002-045216-hist+6.6.0-rc3-rt5-ge31516c1e553+flood-kern-prio-fd-load.jsonn previous one consistent with daily runs form May run-231001-045220-hist+6.6.0-rc3-rt5-ge31516c1e553+flood-kern-prio-fd-load.json The history of the monitoring for kernel gateway under the load for latest RT kernels, branch run on "linux-rt-devel/for-kbuild-bot/current-stable" branch https://canbus.pages.fel.cvut.cz/can-latester/inspect.html?kernel=rt&load=1&flood=1&fd=1&prio=1&kern=1 Monitoring of latency when userspace application is used to forward data from one to another CAN interface does not show similar excess https://canbus.pages.fel.cvut.cz/can-latester/inspect.html?kernel=rt&load=1&flood=1&fd=1&prio=1&kern=0 It is interesting that when priority of CAN controller interrupt service routines are not boosted then problem does not appear. Priority 90 is set for each irq/[0-9]+-can[0-9] thread by chrt -f --pid 90 $pid The device under the test as well as messages generation and monitoring system are MZ_APO boards (AMD/XlinX Zynq XC7Z010) with CTU CAN FD IP core CAN controller configured for 10 ns frames timestamping. The problem can be in configuration of our system, CTU CAN FD IP core driver or specific to Zynq ARM platform. But it is generally suspicious because after initial tuning of the test system there has not been modifications for long time. Monitoring system is running 6.2.0-rt3-00007-ge3a16816f987 kernel for all time and no problem with some Rx buffers overflow on the tester side is reported for time covering all tests in the question. Please, report if you have some idea which change between reported versions from 2023-10-01 and 2023-10-02 could be reason for the change. I plan to keep eye on results till end of the week and if the problem continues then I start to investigate more by beginning of the next week when I should find a little more time. I am quite busy by preparation for conference and teaching this week so I do not expect to find much time. Best wishes, Pavel -- Pavel Pisa phone: +420 603531357 e-mail: pisa@xxxxxxxxxxxxxxxx Department of Control Engineering FEE CVUT Karlovo namesti 13, 121 35, Prague 2 university: http://control.fel.cvut.cz/ personal: http://cmp.felk.cvut.cz/~pisa social: https://social.kernel.org/ppisa projects: https://www.openhub.net/accounts/ppisa CAN related:http://canbus.pages.fel.cvut.cz/ RISC-V education: https://comparch.edu.cvut.cz/ Open Technologies Research Education and Exchange Services https://gitlab.fel.cvut.cz/otrees/org/-/wikis/home
