Hi Vlad Just tested that now, as per your suggestion (max_burst=0 on 3.14.0, all other parameters same as before). The behaviour is the same - that is, SCTP throughput still does not scale, whereas TCP throughput and UDP throughput do. Results are shown below for 1, 2, 3 and 4 parallel associations all transmitting DATA at their maximum rate. Note that I used the stock 3.14.0 kernel, but with Daniel Borkmann's suggestion (from April 11 - see other mailing list thread that I initiated called " Is SCTP throughput really this low compared to TCP? ") to revert out some code, namely: git revert ef2820a735f74ea60335f8ba3801b844f0cb184d (Without this modification, the 3.14.0 throughput is very poor, as originally reported.) Here are the results (SCTP only) using your suggestion, based on 10-minute tests for each configuration. In each case the CPU usage is very small - the throughput is limited entirely by the network latency, not the CPU and/or network bandwidths. SCTP, 1 association: 1 x 102 Mbps = 102 Mbps total throughput SCTP, 2 associations: 2 x 59.2 Mbps = 118 Mbps total throughput SCTP, 3 associations: 3 x 39.8 Mbps = 119 Mbps total throughput SCTP, 4 associations: 4 x 30.7 Mbps = 123 Mbps total throughput For quick reference, here are the results for the same setup, but on the 3.4.2 kernel (and max_burst having its default value of 4). SCTP, 1 association: 1 x 122 Mbps = 122 Mbps total throughput SCTP, 2 associations: 2 x 61.4 Mbps = 123 Mbps total throughput SCTP, 3 associations: 3 x 41.4 Mbps = 124 Mbps total throughput SCTP, 4 associations: 4 x 25.3 Mbps = 127 Mbps total throughput -----Original Message----- From: Vlad Yasevich [mailto:vyasevich@xxxxxxxxx] Sent: May-01-14 6:52 PM To: Butler, Peter; linux-sctp@xxxxxxxxxxxxxxx Subject: Re: SCTP throughput does not scale On 05/01/2014 01:55 PM, Butler, Peter wrote: > It would appear that, in a latency-limited* network, the overall SCTP throughput for a given system does not scale. That is, no matter how many associations are instantiated on a given system the overall throughput remains capped at the same limit. This is in contrast to TCP and UDP where one can, for example, obtain twice the throughput by instantiating 2 TCP/UDP connections as opposed to only 1. > > *The above observation applies to a network where the throughput is limited due to network latency as opposed to CPU and/or network bandwidth. (In a low-latency system where throughput is instead limited by CPU and/or network bandwidth we would not expect to be able to obtain better throughput with more associations as the system/network is already maxed out). > > Quantitative summary and detailed results are shown below for TCP and SCTP (UDP not shown, for brevity, but behavior is similar to TCP: 2 UDP 'connections' yield twice the throughput as 1 UDP 'connection', etc.). > > Testing performed on both the 3.4.2 kernel and 3.14.0 kernel, using iperf3, with a network RTT of 50 ms (manually implemented via tc-netem), using 1000-byte messages, with both send and receive socket kernel buffer sizes of 2 MB, over a 10 Gbps backplane (although for this particular testing (i.e. with 50 ms RTT latency) the high-speed backplane doesn't really factor in). > Hi Peter on the 3.14 kernel, could you retest with /proc/sys/net/sctp/max_burst = 0 Thanks -vlad > Summary (rounding errors included in numbers below): > > TCP, 1 connection: 1 x 144 Mbps = 144 Mbps total throughput > TCP, 2 connections: 2 x 144 Mbps = 287 Mbps total throughput > TCP, 3 connections: 3 x 145 Mbps = 434 Mbps total throughput > > SCTP, 1 association: 1 x 122 Mbps = 122 Mbps total throughput > SCTP, 2 associations: 2 x 61.4 Mbps = 123 Mbps total throughput > SCTP, 3 associations: 3 x 41.4 Mbps = 124 Mbps total throughput > > TCP (and UDP, not shown) scales, SCTP does not. > > > > Actual iperf3 output below. > > > (1a) TCP, one connection > > iperf version 3.0.1 (10 January 2014) > Linux Lab200slot2 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC > 2012 x86_64 > Time: Sat, 26 Apr 2014 08:17:43 GMT > Connecting to host 192.168.240.3, port 5201 > Cookie: Lab200slot2.1398500263.347708.23b404 > TCP MSS: 1448 (default) > [ 4] local 192.168.240.2 port 40776 connected to 192.168.240.3 port > 5201 Starting Test: protocol: TCP, 1 streams, 1000 byte blocks, > omitting 0 seconds, 600 second test . > . <snip> > . > Test Complete. Summary Results: > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-600.05 sec 10.1 GBytes 144 Mbits/sec 0 sender > [ 4] 0.00-600.05 sec 10.1 GBytes 144 Mbits/sec receiver > CPU Utilization: local/sender 1.2% (0.1%u/1.1%s), remote/receiver 3.2% > (0.3%u/2.9%s) > > > > (1b) TCP, two connections > > iperf version 3.0.1 (10 January 2014) > Linux Lab200slot2 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC > 2012 x86_64 > Time: Sat, 26 Apr 2014 08:28:44 GMT > Connecting to host 192.168.240.3, port 5201 > Cookie: Lab200slot2.1398500924.026363.19e3a5 > TCP MSS: 1448 (default) > [ 4] local 192.168.240.2 port 40780 connected to 192.168.240.3 port > 5201 [ 6] local 192.168.240.2 port 40781 connected to 192.168.240.3 > port 5201 Starting Test: protocol: TCP, 2 streams, 1000 byte blocks, > omitting 0 seconds, 600 second test . > . <snip> > . > Test Complete. Summary Results: > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-600.05 sec 10.0 GBytes 144 Mbits/sec 137 sender > [ 4] 0.00-600.05 sec 10.0 GBytes 144 Mbits/sec receiver > [ 6] 0.00-600.05 sec 10.0 GBytes 144 Mbits/sec 2 sender > [ 6] 0.00-600.05 sec 10.0 GBytes 144 Mbits/sec receiver > [SUM] 0.00-600.05 sec 20.1 GBytes 287 Mbits/sec 139 sender > [SUM] 0.00-600.05 sec 20.0 GBytes 287 Mbits/sec receiver > CPU Utilization: local/sender 2.5% (0.2%u/2.3%s), remote/receiver 4.4% > (0.4%u/4.1%s) > > > > (1c) TCP, three connections > > iperf version 3.0.1 (10 January 2014) > Linux Lab200slot2 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC > 2012 x86_64 > Time: Sat, 26 Apr 2014 08:39:44 GMT > Connecting to host 192.168.240.3, port 5201 > Cookie: Lab200slot2.1398501584.755085.3c10fa > TCP MSS: 1448 (default) > [ 4] local 192.168.240.2 port 40785 connected to 192.168.240.3 port > 5201 [ 6] local 192.168.240.2 port 40786 connected to 192.168.240.3 > port 5201 [ 8] local 192.168.240.2 port 40787 connected to > 192.168.240.3 port 5201 Starting Test: protocol: TCP, 3 streams, 1000 > byte blocks, omitting 0 seconds, 600 second test . > . <snip> > . > Test Complete. Summary Results: > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-600.05 sec 10.1 GBytes 145 Mbits/sec 0 sender > [ 4] 0.00-600.05 sec 10.1 GBytes 145 Mbits/sec receiver > [ 6] 0.00-600.05 sec 10.1 GBytes 145 Mbits/sec 7 sender > [ 6] 0.00-600.05 sec 10.1 GBytes 145 Mbits/sec receiver > [ 8] 0.00-600.05 sec 10.1 GBytes 145 Mbits/sec 4 sender > [ 8] 0.00-600.05 sec 10.1 GBytes 145 Mbits/sec receiver > [SUM] 0.00-600.05 sec 30.3 GBytes 434 Mbits/sec 11 sender > [SUM] 0.00-600.05 sec 30.3 GBytes 434 Mbits/sec receiver > CPU Utilization: local/sender 3.7% (0.3%u/3.4%s), remote/receiver 5.7% > (0.5%u/5.3%s) > > > > (2a) SCTP, one association > > iperf version 3.0.1 (10 January 2014) > Linux Lab200slot2 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC > 2012 x86_64 > Time: Sat, 26 Apr 2014 05:30:34 GMT > Connecting to host 192.168.240.3, port 5201 > Cookie: Lab200slot2.1398490234.326620.36ee2f > [ 4] local 192.168.240.2 port 46631 connected to 192.168.240.3 port > 5201 Starting Test: protocol: SCTP, 1 streams, 1000 byte blocks, > omitting 0 seconds, 600 second test . > . <snip> > . > Test Complete. Summary Results: > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-600.07 sec 8.52 GBytes 122 Mbits/sec sender > [ 4] 0.00-600.07 sec 8.52 GBytes 122 Mbits/sec receiver > CPU Utilization: local/sender 4.1% (0.2%u/3.9%s), remote/receiver 2.0% > (0.2%u/1.8%s) > > > > (2b) SCTP, two associations > > iperf version 3.0.1 (10 January 2014) > Linux Lab200slot2 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC > 2012 x86_64 > Time: Sat, 26 Apr 2014 05:41:35 GMT > Connecting to host 192.168.240.3, port 5201 > Cookie: Lab200slot2.1398490895.079359.28c5c1 > [ 4] local 192.168.240.2 port 34175 connected to 192.168.240.3 port > 5201 [ 6] local 192.168.240.2 port 41448 connected to 192.168.240.3 > port 5201 Starting Test: protocol: SCTP, 2 streams, 1000 byte blocks, > omitting 0 seconds, 600 second test . > . <snip> > . > Test Complete. Summary Results: > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-600.09 sec 4.29 GBytes 61.4 Mbits/sec sender > [ 4] 0.00-600.09 sec 4.29 GBytes 61.4 Mbits/sec receiver > [ 6] 0.00-600.09 sec 4.29 GBytes 61.4 Mbits/sec sender > [ 6] 0.00-600.09 sec 4.29 GBytes 61.4 Mbits/sec receiver > [SUM] 0.00-600.09 sec 8.58 GBytes 123 Mbits/sec sender > [SUM] 0.00-600.09 sec 8.58 GBytes 123 Mbits/sec receiver > CPU Utilization: local/sender 2.5% (0.1%u/2.4%s), remote/receiver 1.7% > (0.2%u/1.5%s) > > > > (2c) SCTP, three associations > > iperf version 3.0.1 (10 January 2014) > Linux Lab200slot2 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC > 2012 x86_64 > Time: Sat, 26 Apr 2014 05:52:36 GMT > Connecting to host 192.168.240.3, port 5201 > Cookie: Lab200slot2.1398491555.947776.10993d > [ 4] local 192.168.240.2 port 45551 connected to 192.168.240.3 port > 5201 [ 6] local 192.168.240.2 port 35528 connected to 192.168.240.3 > port 5201 [ 8] local 192.168.240.2 port 44540 connected to > 192.168.240.3 port 5201 Starting Test: protocol: SCTP, 3 streams, 1000 > byte blocks, omitting 0 seconds, 600 second test . > . <snip> > . > Test Complete. Summary Results: > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-600.08 sec 2.90 GBytes 41.4 Mbits/sec sender > [ 4] 0.00-600.08 sec 2.90 GBytes 41.4 Mbits/sec receiver > [ 6] 0.00-600.08 sec 2.90 GBytes 41.4 Mbits/sec sender > [ 6] 0.00-600.08 sec 2.90 GBytes 41.4 Mbits/sec receiver > [ 8] 0.00-600.08 sec 2.90 GBytes 41.4 Mbits/sec sender > [ 8] 0.00-600.08 sec 2.90 GBytes 41.4 Mbits/sec receiver > [SUM] 0.00-600.08 sec 8.69 GBytes 124 Mbits/sec sender > [SUM] 0.00-600.08 sec 8.69 GBytes 124 Mbits/sec receiver > CPU Utilization: local/sender 2.6% (0.1%u/2.5%s), remote/receiver 1.6% > (0.1%u/1.5%s) > > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-sctp" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
