On 10/10/24 9:03 AM, David Ahern wrote: > On 10/10/24 8:21 AM, Jens Axboe wrote: >>> which adds zc send. I ran a quick test, and it does reduce cpu >>> utilization on the sender from 100% to 95%. I'll keep poking... >> >> Update on this - did more testing and the 100 -> 95 was a bit of a >> fluke, it's still maxed. So I added io_uring send and sendzc support to >> kperf, and I still saw the sendzc being maxed out sending at 100G rates >> with 100% cpu usage. >> >> Poked a bit, and the reason is that it's all memcpy() off >> skb_orphan_frags_rx() -> skb_copy_ubufs(). At this point I asked Pavel >> as that made no sense to me, and turns out the kernel thinks there's a >> tap on the device. Maybe there is, haven't looked at that yet, but I >> just killed the orphaning and tested again. >> >> This looks better, now I can get 100G line rate from a single thread >> using io_uring sendzc using only 30% of the single cpu/thread (including >> irq time). That is good news, as it unlocks being able to test > 100G as >> the sender is no longer the bottleneck. >> >> Tap side still a mystery, but it unblocked testing. I'll figure that >> part out separately. >> > > Thanks for the update. 30% cpu is more inline with my testing. > > For the "tap" you need to make sure no packet socket applications are > running -- e.g., lldpd is a typical open I have a seen in tests. Check > /proc/net/packet Here's what I see: sk RefCnt Type Proto Iface R Rmem User Inode 0000000078c66cbc 3 3 0003 2 1 0 0 112645 00000000558db352 3 3 0003 2 1 0 0 109578 00000000486837f4 3 3 0003 4 1 0 0 109580 00000000f7c6edd6 3 3 0003 4 1 0 0 22563 000000006ec0363c 3 3 0003 2 1 0 0 22565 0000000095e63bff 3 3 0003 5 1 0 0 22567 was just now poking at what could be causing this. This is a server box, nothing really is running on it... The nic in question is ifindex 2. -- Jens Axboe
