On 16/12/2024 9:38, Song Yoong Siang wrote:
The igc_close()/igc_open() functions are too drastic for installing a new XDP prog because they cause undesirable link down event and device reset. To avoid delays in Ethernet traffic, improve the XDP_SETUP_PROG process by using the same sequence as igc_xdp_setup_pool(), which performs only the necessary steps, as follows: 1. stop the traffic and clean buffer 2. stop NAPI 3. install the XDP program 4. resume NAPI 5. allocate buffer and resume the traffic This patch has been tested using the 'ip link set xdpdrv' command to attach a simple XDP prog that always returns XDP_PASS. Before this patch, attaching xdp program will cause ptp4l to lose sync for few seconds, as shown in ptp4l log below: ptp4l[198.082]: rms 4 max 8 freq +906 +/- 2 delay 12 +/- 0 ptp4l[199.082]: rms 3 max 4 freq +906 +/- 3 delay 12 +/- 0 ptp4l[199.536]: port 1 (enp2s0): link down ptp4l[199.536]: port 1 (enp2s0): SLAVE to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED) ptp4l[199.600]: selected local clock 22abbc.fffe.bb1234 as best master ptp4l[199.600]: port 1 (enp2s0): assuming the grand master role ptp4l[199.600]: port 1 (enp2s0): master state recommended in slave only mode ptp4l[199.600]: port 1 (enp2s0): defaultDS.priority1 probably misconfigured ptp4l[202.266]: port 1 (enp2s0): link up ptp4l[202.300]: port 1 (enp2s0): FAULTY to LISTENING on INIT_COMPLETE ptp4l[205.558]: port 1 (enp2s0): new foreign master 44abbc.fffe.bb2144-1 ptp4l[207.558]: selected best master clock 44abbc.fffe.bb2144 ptp4l[207.559]: port 1 (enp2s0): LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[208.308]: port 1 (enp2s0): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[208.933]: rms 742 max 1303 freq -195 +/- 682 delay 12 +/- 0 ptp4l[209.933]: rms 178 max 274 freq +387 +/- 243 delay 12 +/- 0 After this patch, attaching xdp program no longer cause ptp4l to lose sync, as shown in ptp4l log below: ptp4l[201.183]: rms 1 max 3 freq +959 +/- 1 delay 8 +/- 0 ptp4l[202.183]: rms 1 max 3 freq +961 +/- 2 delay 8 +/- 0 ptp4l[203.183]: rms 2 max 3 freq +958 +/- 2 delay 8 +/- 0 ptp4l[204.183]: rms 3 max 5 freq +961 +/- 3 delay 8 +/- 0 ptp4l[205.183]: rms 2 max 4 freq +964 +/- 3 delay 8 +/- 0 Besides, before this patch, attaching xdp program will causes flood ping to lose 10 packets, as shown in ping statistics below: --- 169.254.1.2 ping statistics --- 100000 packets transmitted, 99990 received, +6 errors, 0.01% packet loss, time 34001ms rtt min/avg/max/mdev = 0.028/0.301/3104.360/13.838 ms, pipe 10, ipg/ewma 0.340/0.243 ms After this patch, attaching xdp program no longer cause flood ping to loss any packets, as shown in ping statistics below: --- 169.254.1.2 ping statistics --- 100000 packets transmitted, 100000 received, 0% packet loss, time 32326ms rtt min/avg/max/mdev = 0.027/0.231/19.589/0.155 ms, pipe 2, ipg/ewma 0.323/0.322 ms On the other hand, this patch has been tested with tools/testing/selftests/ bpf/xdp_hw_metadata app to make sure AF_XDP zero-copy is working fine with XDP Tx and Rx metadata. Below is the result of last packet after received 10000 UDP packets with interval 1 ms: poll: 1 (0) skip=0 fail=0 redir=10000 xsk_ring_cons__peek: 1 0x55881c7ef7a8: rx_desc[9999]->addr=8f110 addr=8f110 comp_addr=8f110 EoP rx_hash: 0xFB9BB6A3 with RSS type:0x1 HW RX-time: 1733923136269470866 (sec:1733923136.2695) delta to User RX-time sec:0.0000 (43.280 usec) XDP RX-time: 1733923136269482482 (sec:1733923136.2695) delta to User RX-time sec:0.0000 (31.664 usec) No rx_vlan_tci or rx_vlan_proto, err=-95 0x55881c7ef7a8: ping-pong with csum=ab19 (want 315b) csum_start=34 csum_offset=6 0x55881c7ef7a8: complete tx idx=9999 addr=f010 HW TX-complete-time: 1733923136269591637 (sec:1733923136.2696) delta to User TX-complete-time sec:0.0001 (108.571 usec) XDP RX-time: 1733923136269482482 (sec:1733923136.2695) delta to User TX-complete-time sec:0.0002 (217.726 usec) HW RX-time: 1733923136269470866 (sec:1733923136.2695) delta to HW TX-complete-time sec:0.0001 (120.771 usec) 0x55881c7ef7a8: complete rx idx=10127 addr=8f110 Signed-off-by: Song Yoong Siang <yoong.siang.song@xxxxxxxxx> --- V2: https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20241211134532.3489335-1-yoong.siang.song@xxxxxxxxx/ V1: https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20241204120233.3148482-1-yoong.siang.song@xxxxxxxxx/ V3 changelog: - Change commit title to be more specific. (Paul) - Use unsigned int, instead of int, for array elements. (Paul) - Fix grammar mistakes on commit message. (Paul) - Explain why igc_close()/igc_open() are not needed in commit message. (Paul) V2 changelog: - shows some examples of problem in commit message. (Vinicius) - igc_close()/igc_open() are too big a hammer for installing a new XDP program. Only do we we really need. (Vinicius) --- drivers/net/ethernet/intel/igc/igc_xdp.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) Tested-by: Avigail Dahan <avigailx.dahan@xxxxxxxxx>
