> -----Original Message----- > From: Andrew Lunn <andrew@xxxxxxx> > Sent: 2023年7月5日 8:25 > To: Wei Fang <wei.fang@xxxxxxx> > Cc: davem@xxxxxxxxxxxxx; edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx; > pabeni@xxxxxxxxxx; ast@xxxxxxxxxx; daniel@xxxxxxxxxxxxx; > hawk@xxxxxxxxxx; john.fastabend@xxxxxxxxx; Shenwei Wang > <shenwei.wang@xxxxxxx>; Clark Wang <xiaoning.wang@xxxxxxx>; > netdev@xxxxxxxxxxxxxxx; dl-linux-imx <linux-imx@xxxxxxx>; > linux-kernel@xxxxxxxxxxxxxxx; bpf@xxxxxxxxxxxxxxx > Subject: Re: [PATCH net 3/3] net: fec: increase the size of tx ring and update > thresholds of tx ring > > > After trying various methods, I think that increase the size of tx BD > > ring is simple and effective. Maybe the best resolution is that > > allocate NAPI for each queue to improve the efficiency of the NAPI > > callback, but this change is a bit big and I didn't try this method. > > Perheps this method will be implemented in a future patch. > > How does this affect platforms like Vybrid with its fast Ethernet? Sorry, I don't have the Vybrid platform, but I think I don't think it has much impact, at most it just takes up some more memory. > Does the burst latency go up? No, for fec, when a packet is attached to the BDs, the software will immediately trigger the hardware to send the packet. In addition, I think it may improve the latency, because the size of the tx ring becomes larger, and more packets can be attached to the BD ring for burst traffic. > > > In addtion, this patch also updates the tx_stop_threshold and the > > tx_wake_threshold of the tx ring. In previous logic, the value of > > tx_stop_threshold is 217, however, the value of tx_wake_threshold is > > 147, it does not make sense that tx_wake_threshold is less than > > tx_stop_threshold. > > What do these actually mean? I could imagine that as the ring fills you don't > want to stop until it is 217/512 full. There is then some hysteresis, such that it > has to drop below 147/512 before more can be added? > You must have misunderstood, let me explain more clearly, the queue will be stopped when the available BDs are less than tx_stop_threshold (217 BDs). And the queue will be waked when the available BDs are greater than tx_wake_threshold (147 BDs). So in most cases, the available BDs are greater than tx_wake_threshold when the queue is stopped, the only effect is to delay packet sending. In my opinion, tx_wake_threshold should be greater than tx_stop_threshold, we should stop queue when the available BDs are not enough for a skb to be attached. And wake the queue when the available BDs are sufficient for a skb. > > Besides, both XDP path and 'slow path' share the tx BD rings. So if > > tx_stop_threshold is 217, in the case of heavy XDP traffic, the slow > > path is easily to be stopped, this will have a serious impact on the > > slow path. > > Please post your iperf results for various platforms, so we can see the effects of > this. We generally don't accept tuning patches without benchmarks which > prove the improvements, and also show there is no regression. And given the > wide variety of SoCs using the FEC, i expect testing on a number of SoCs, but > Fast and 1G. > Below are the results on i.MX6UL/8MM/8MP/8ULP/93 platforms, i.MX6UL and 8ULP only support Fast ethernet. Others support 1G. 1.1 i.MX6UL with tx ring size 512 root@imx6ul7d:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 47148 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.1 MBytes 84.9 Mbits/sec 0 103 KBytes [ 5] 1.00-2.00 sec 9.88 MBytes 82.6 Mbits/sec 0 103 KBytes [ 5] 2.00-3.00 sec 9.82 MBytes 82.7 Mbits/sec 0 103 KBytes [ 5] 3.00-4.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes [ 5] 4.00-5.00 sec 9.88 MBytes 82.9 Mbits/sec 0 103 KBytes [ 5] 5.00-6.00 sec 9.94 MBytes 83.4 Mbits/sec 0 103 KBytes [ 5] 6.00-7.00 sec 10.1 MBytes 84.3 Mbits/sec 0 103 KBytes [ 5] 7.00-8.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes [ 5] 8.00-9.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes [ 5] 9.00-10.00 sec 9.88 MBytes 82.9 Mbits/sec 0 103 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 99.0 MBytes 83.1 Mbits/sec 0 sender [ 5] 0.00-10.01 sec 98.8 MBytes 82.8 Mbits/sec receiver 1.2 i.MX6UL with tx ring size 1024 root@imx6ul7d:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 55670 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.2 MBytes 85.4 Mbits/sec 0 236 KBytes [ 5] 1.00-2.00 sec 10.1 MBytes 84.6 Mbits/sec 0 236 KBytes [ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 249 KBytes [ 5] 3.00-4.00 sec 10.1 MBytes 85.1 Mbits/sec 0 249 KBytes [ 5] 4.00-5.00 sec 10.1 MBytes 84.7 Mbits/sec 0 249 KBytes [ 5] 5.00-6.00 sec 10.0 MBytes 84.1 Mbits/sec 0 249 KBytes [ 5] 6.00-7.00 sec 10.1 MBytes 85.1 Mbits/sec 0 249 KBytes [ 5] 7.00-8.00 sec 10.1 MBytes 84.9 Mbits/sec 0 249 KBytes [ 5] 8.00-9.00 sec 10.3 MBytes 85.9 Mbits/sec 0 249 KBytes [ 5] 9.00-10.00 sec 10.2 MBytes 85.6 Mbits/sec 0 249 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 101 MBytes 85.1 Mbits/sec 0 sender [ 5] 0.00-10.01 sec 101 MBytes 84.5 Mbits/sec receiver 2.1 i.MX8ULP with tx ring size 512 root@imx8ulpevk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 54342 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.8 MBytes 90.3 Mbits/sec 0 99.0 KBytes [ 5] 1.00-2.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes [ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes [ 5] 3.00-4.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes [ 5] 4.00-5.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes [ 5] 5.00-6.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes [ 5] 6.00-7.00 sec 9.69 MBytes 81.3 Mbits/sec 0 99.0 KBytes [ 5] 7.00-8.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes [ 5] 8.00-9.00 sec 9.69 MBytes 81.3 Mbits/sec 0 99.0 KBytes [ 5] 9.00-10.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 100 MBytes 84.3 Mbits/sec 0 sender [ 5] 0.00-9.90 sec 100 MBytes 84.7 Mbits/sec receiver 2.1 i.MX8ULP with tx ring size 1024 root@imx8ulpevk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 44770 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.7 MBytes 90.1 Mbits/sec 0 93.3 KBytes [ 5] 1.00-2.00 sec 9.94 MBytes 83.4 Mbits/sec 0 93.3 KBytes [ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 93.3 KBytes [ 5] 3.00-4.00 sec 10.1 MBytes 85.0 Mbits/sec 0 93.3 KBytes [ 5] 4.00-5.00 sec 9.94 MBytes 83.4 Mbits/sec 0 100 KBytes [ 5] 5.00-6.00 sec 10.2 MBytes 85.5 Mbits/sec 0 100 KBytes [ 5] 6.00-7.00 sec 9.69 MBytes 81.3 Mbits/sec 0 100 KBytes [ 5] 7.00-8.00 sec 9.94 MBytes 83.4 Mbits/sec 0 100 KBytes [ 5] 8.00-9.00 sec 10.2 MBytes 85.5 Mbits/sec 0 100 KBytes [ 5] 9.00-10.00 sec 9.69 MBytes 81.3 Mbits/sec 0 100 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 101 MBytes 84.4 Mbits/sec 0 sender [ 5] 0.00-9.92 sec 100 MBytes 84.8 Mbits/sec receiver 3.1 i.MX8MM with tx ring size 512 root@imx8mmevk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 55734 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 111 MBytes 934 Mbits/sec 0 577 KBytes [ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 577 KBytes [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 609 KBytes [ 5] 3.00-4.00 sec 113 MBytes 945 Mbits/sec 0 638 KBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 638 KBytes [ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes [ 5] 7.00-8.00 sec 112 MBytes 943 Mbits/sec 0 638 KBytes [ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 638 KBytes [ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver 3.2 i.MX8MM with tx ring size 1024 root@imx8mmevk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 53350 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 0 585 KBytes [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 585 KBytes [ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec 0 585 KBytes [ 5] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 0 648 KBytes [ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes [ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 648 KBytes [ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes [ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver 4.1 i.MX8MP with tx ring size 512 root@imx8mpevk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 51892 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 594 KBytes [ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes [ 5] 2.00-3.00 sec 113 MBytes 946 Mbits/sec 0 626 KBytes [ 5] 3.00-4.00 sec 112 MBytes 937 Mbits/sec 0 626 KBytes [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes [ 5] 5.00-6.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes [ 5] 6.00-7.00 sec 113 MBytes 946 Mbits/sec 0 626 KBytes [ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 626 KBytes [ 5] 8.00-9.00 sec 111 MBytes 935 Mbits/sec 0 626 KBytes [ 5] 9.00-10.00 sec 112 MBytes 943 Mbits/sec 0 626 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver 4.2 i.MX8MP with tx ring size 1024 root@imx8mpevk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 37922 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 951 Mbits/sec 0 608 KBytes [ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 608 KBytes [ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec 0 608 KBytes [ 5] 3.00-4.00 sec 111 MBytes 934 Mbits/sec 0 608 KBytes [ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes [ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec 0 608 KBytes [ 5] 6.00-7.00 sec 113 MBytes 949 Mbits/sec 0 608 KBytes [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes [ 5] 8.00-9.00 sec 112 MBytes 936 Mbits/sec 0 608 KBytes [ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver 5.1 i.MX93 with tx ring size 512 root@imx93evk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 44216 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 115 MBytes 965 Mbits/sec 0 656 KBytes [ 5] 1.00-2.00 sec 111 MBytes 934 Mbits/sec 0 656 KBytes [ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes [ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes [ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 656 KBytes [ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes [ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes [ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver 5.2 i.MX93 with tx ring size 1024 root@imx93evk:~# iperf3 -c 192.168.3.6 Connecting to host 192.168.3.6, port 5201 [ 5] local 192.168.3.9 port 51058 connected to 192.168.3.6 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 588 KBytes [ 5] 1.00-2.00 sec 112 MBytes 935 Mbits/sec 0 649 KBytes [ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes [ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes [ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes [ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 649 KBytes [ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes [ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 940 Mbits/sec receiver
