On Mon, May 22, 2023 at 01:37:19PM -0700, Peter Wallace wrote: > On Tue, 23 May 2023, Rod Webster wrote: > > > Date: Tue, 23 May 2023 06:02:13 +1000 > > From: Rod Webster <rod@xxxxxxxxxx> > > To: Marcelo Tosatti <mtosatti@xxxxxxxxxx> > > Cc: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>, > > linux-rt-users@xxxxxxxxxxxxxxx > > Subject: Re: Excessive network latency when using Realtek R8168/R8111 et al > > NIC > > > > This stuff is hard! I just realised that rtapi_app is a red herring! > > rtapi_app is Linuxcnc and there is nothing wrong with it. Its thread > > is on a 1000us cycle so it seems it gets all its jobs done in 200us > > and then sleeps for 800us which makes perfect sense! > > > > The issue we have is deeper than that. I think we should be looking at > > the NIC interrupt (but don't trust the novice!). > > The network communication is consuming more than the 800us slack from > > time to time. When that happens, our hardware sees the timing overrun > > and increments an internal packet error count. If too many of these > > happen in succession, the hardware decides the RT environment can't be > > relied on, disables further communication and returns an "error > > finishing read" to Linuxcnc to say it's given up. > > > > Marcelo, we didn't resort to C. We were able to use a bash script and > > use a linuxcnc tool called halcmd to query the hardware as shown here. > > #!/usr/bin/bash > > stat=0 > > while (($stat < 1)) > > do > > stat=`halcmd getp hm2_7i96s.0.packet-error-total` > > done > > trace-cmd stop > > > > I think we need to increase the stat threshold so we get more samples > > in our trace before stopping it. The current trace will only have one > > instance. > > Thanks for letting me see the issue more clearly. > > > > > > Rod Webster > > > > > I should note that at least for Intel MACs, the 6.3.1-rt13 and 6.4.0-rc2-rt1 > kernels seem to solve the issue. Not sure what changed but maximum read time > is now in the 200.. 250 usec peak region (about 100 usec more than average) > This is the peak read latency after about 3 days of videos, compiling and > local network activity. > > Sadly 6.4.0-rc3-rt2 has regressed slightly in network latency on my test > systems > > My test systems were all Intel CPUs with 4 cores, isolcpus=3 and the Ethernet > IRQ pinned to CPU3 > > > Peter Wallace Are you guys using the realtime profiles from Tuned? Edit /etc/tuned/realtime-virtual-host-variables.conf, Then run tuned-adm profile realtime-virtual-host Note this will perform steps to isolate the configured CPU's, including unpinning all IRQs from the isolated CPUs, (which you can fix after applying the profile). enabling nohz_full=, rcu_nocbs=, etc (can check /usr/lib/tuned/realtime-virtual-host/tuned.conf and script.sh to see what what it (and its parent profiles) do).
