On 17/03/22 15:59, Jay Vosburgh wrote:
Federico Parola <federico.parola@xxxxxxxxx> wrote:Hello everybody, I'm experiencing a strange problem when running an AF_XDP application with busy poll enabled on a Intel XL710 NIC (i40e driver). The problem can be replicated running the xdpsock kernel sample in rx or l2fwd mode. The first packet I send to the machine is correctly received by the application. After this, packets are only received in batches of 8. If I send 7 packets the application sees nothing, while the 8th one triggers the reception of all 8 packets. Disabling the busy poll mode everything works fine and packets are immediately received as they are sent. I tried changing kernel (5.12, 5.14 and 5.16) but all present the same problem. I also tried using another NIC, an Intel X540 with ixgbe driver and the problem isn't there, so I guess is NIC/driver related. I tried monitoring ethtool statistics. When sending packets between 1 and 7 these counters are increased: stat: 64 ( 64) <= port.rx_bytes /sec stat: 1 ( 1) <= port.rx_size_64 /sec stat: 1 ( 1) <= port.rx_unicast /sec stat: 1 ( 1) <= rx_unicast /sec While the 8th one triggers this updates: stat: 64 ( 64) <= port.rx_bytes /sec stat: 1 ( 1) <= port.rx_size_64 /sec stat: 1 ( 1) <= port.rx_unicast /sec stat: 477 ( 477) <= rx-0.bytes /sec stat: 8 ( 8) <= rx-0.packets /sec stat: 477 ( 477) <= rx_bytes /sec stat: 8 ( 8) <= rx_packets /sec stat: 1 ( 1) <= rx_unicast /sec As far as I understand the first set of counters are hardware counters, so it makes me think that packets are kept in the NIC and not even sent to memory. Does anyone have any suggestion on what could be causing this problem? Does enabling busy poll set some flag on the NIC?We observed similar "batching" behavior on i40e devices late last year in ordinary use (not XDP, but using SR-IOV VFs). We instrumented the drivers at the send and receive sides, and determined that it appeared to be a behavior of the receiving device itself, i.e., packets 1 - 7 would be held indefinitely (as I recall, no interrupt or update of the RX ring pointers) until packet 8 arrives, at which point all 8 were delivered simultaneously. The issue was evidently in the firmware, and was resolved after a firmware upgrade.
Hi Jay,I just updated the firmware to the latest version (v8.50 from v8.30) but unfortunately the problem is still there. However I'm experiencing the problem only when using AF_XDP in busy poll mode, all other modes (standard AF_XDP and normal packet reception) work just fine.
Maybe the two problems are correlated in some way.
-JBest regards, Federico Parola--- -Jay Vosburgh, jay.vosburgh@xxxxxxxxxxxxx
|