On 2023-04-28 20:46:17 [+1000], Rod Webster wrote: > Sebastian Hi Rod, please keep list in Cc: Avoiding top-posting is a quite nice. > Thanks, the Real TIme PREEMPT_RT kernel is a prerequisite for > Linuxcnc. Note in this discussion, I am only referring to X86 machines Okay. Then it could be related to softirq rework which started in v5.0.19-rt11 and updated in v5.9-rc6-rt9. Are you able to recompile a kernel? > although we also install our software on ARM and others. > linuxcnc-uspace is included in the debian Bookworm repositories and > linux-image-rt is one of its dependencies.. > There are really two issues we have experienced. These have only > become a factor since the 5.X kernels. Only one seems to be relevant > to the RT project. > 1. Excessive real time latency/jitter in the Debian RT images. I have > escalated this to them on their Bug 1034550. Ref > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034550 As mentioned, the images are provided for convince. Depending one the problem, the bugs are forwarded sometimes here. However as per you Debian bug report v4.19-RT is fine, v5.10-RT isn't and v6.1-RT is better or is it fine? > 2. Excessive real time network latency/jitter when using the > PREEMPT_RT kernel when communicating with Intel ethernet hardware in > an RT environment. Where is the Intel ethernet coming from? You were saying it is the realtek nic. > It is the second issue I wish to bring to your attention. eg. > Excessive network latency. This appears to be more prevalent on > Realtek hardware. > Note that we document setting hardware-irq-coalesce-rx-usecs 0 on > Intel NIC's is required. This setting has no effect on other makes of > NIC's. > > In relation to Realtek NIC's, I can only speak for Debian but by > default they use the R8169 Kernel module. (CONFIG_R8169?) They also > offer the R8168-dkms and R8125-dkms drivers. We have many data points > from our users that the R8169 driver offers very poor RT performance. > But just installing a dkms driver is not enough on Debian to resolve > the issue. Is the vendor driver better than in the tree? This is not obvious from what you are writing. > We need to also build a RT kernel from the upstream sources and apply > your RT patches. > > So aside from the awful high latency/jitter implementation of RT by > Debian since the 5.10 kernel (Bookworm), the current strategy to use > the R8169 as a generic driver is flawed. Debian says to report any use > of the R8168-dkms driver and they will fix it. > > A lot of small industrial PC's use Realtek NIC and with the pending > release of Bookworm now Linuxcnc is in its repos, there will be a wide > ranging impact. > > Our FGPA ethernet hardware is capable of reporting maximum ethernet > read and write times in timer ticks and is used in a point to point > connection (no routers). > > I hope you can investigate this matter. I am surprised to hear the vendor driver is doing fine while the in-tree driver is bad. Two things you could do: - enabling tracing to see what is causing the delays/ latency spike. You will need a trigger to notice the latency and then stop the trace at this point. If so, I could provide additional steps unless you can do it yourself. - You could try to isolate the realtek driver on one CPU and moving everything else to another. > end > -Rod Webster > > > > Rod Webster > > VMN® > > www.vmn.com.au Sebastian
