Hello, On 04/14/2014 06:45 PM, ext Daniel Borkmann wrote: > Ok, thanks! I'll send out a revert for now today as this is otherwise > catastrophic ... when that is done, we/the developer from NSN can still > think about his patch and how to solve that differently. Thanks from my side as well, I will for sure look into it. Congestion control seems to be completely broken. Seems stack doesnt like the new way how rwnd is calculated. Dependency to ipv4 is also interesting. I also wasnt able to reproduce with my laptop/desktop, it seems some higher processing power is also needed to hit the case. > On 04/14/2014 06:43 PM, Butler, Peter wrote: >> With the git revert command applied, the SCTP+IPv4 performance on 3.14.0 is back to normal! Same throughput as 3.4.2 and steady and smooth: >> >> [root@Lab200slot2 ~]# iperf3 --sctp -4 -c 192.168.240.3 -V -l 1452 -t 60 >> iperf version 3.0.1 (10 January 2014) >> Linux Lab200slot2 3.14.0+ #1 SMP Mon Apr 14 12:06:40 EDT 2014 x86_64 >> Time: Mon, 14 Apr 2014 16:40:48 GMT >> Connecting to host 192.168.240.3, port 5201 >> Cookie: Lab200slot2.1397493648.413274.65e131 >> [ 4] local 192.168.240.2 port 50548 connected to 192.168.240.3 port 5201 >> Starting Test: protocol: SCTP, 1 streams, 1452 byte blocks, omitting 0 seconds, 60 second test >> [ ID] Interval Transfer Bandwidth >> [ 4] 0.00-1.00 sec 240 MBytes 2.02 Gbits/sec >> [ 4] 1.00-2.00 sec 239 MBytes 2.01 Gbits/sec >> [ 4] 2.00-3.00 sec 240 MBytes 2.01 Gbits/sec >> [ 4] 3.00-4.00 sec 239 MBytes 2.00 Gbits/sec >> [ 4] 4.00-5.00 sec 245 MBytes 2.05 Gbits/sec >> [ 4] 5.00-6.00 sec 240 MBytes 2.01 Gbits/sec >> [ 4] 6.00-7.00 sec 240 MBytes 2.02 Gbits/sec >> [ 4] 7.00-8.00 sec 239 MBytes 2.01 Gbits/sec >> >> >> >> >> >> >> -----Original Message----- >> From: Daniel Borkmann [mailto:dborkman@xxxxxxxxxx] >> Sent: April-11-14 7:58 PM >> To: Butler, Peter >> Cc: Vlad Yasevich; linux-sctp@xxxxxxxxxxxxxxx >> Subject: Re: Is SCTP throughput really this low compared to TCP? >> >> On 04/11/2014 10:18 PM, Butler, Peter wrote: >>> It may take a little time to do the bisection. Can you provide me >>> with a >> > guesstimate as to which kernel(s) may have introduced this behaviour? >> >> Just out of curiosity, could you do a ... >> >> git revert ef2820a735f74ea60335f8ba3801b844f0cb184d >> >> ... on your tree and try to see what you get with that? >> >>> -----Original Message----- >>> From: Daniel Borkmann [mailto:dborkman@xxxxxxxxxx] >>> Sent: April-11-14 2:40 PM >>> To: Butler, Peter >>> Cc: Vlad Yasevich; linux-sctp@xxxxxxxxxxxxxxx >>> Subject: Re: Is SCTP throughput really this low compared to TCP? >>> >>> On 04/11/2014 08:22 PM, Butler, Peter wrote: >>>> The difference between 3.14 and 3.4.2 is staggering. An order of >>>> magnitude or so. For example, >>> > using the precisely same setup as before, whereas I get about 2.1 Gbps throughput with 3.4 2, I > can only manage between 70-150 Mbps with 3.14 - a staggering difference. >>>> >>>> Moreover, the SCTP throughput seems to 'choke' itself with 3.14, such >>>> that it is always trying to >>> > recover. For example, with 3.4.2 the 2.1 Gbps throughput is quite consistent from one second to > the next (as you would expect): >>>> >>>> [root@Lab200slot2 ~]# iperf3 --sctp -4 -c 192.168.241.3 -V -l 1452 -t >>>> 60 iperf version 3.0.1 (10 January 2014) Linux Lab200slot2 >>>> 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC 2012 x86_64 >>>> Time: Fri, 11 Apr 2014 18:19:15 GMT >>>> Connecting to host 192.168.241.3, port 5201 >>>> Cookie: Lab200slot2.1397240355.069035.0d5b0f >>>> [ 4] local 192.168.241.2 port 56030 connected to 192.168.241.3 port >>>> 5201 Starting Test: protocol: SCTP, 1 streams, 1452 byte blocks, omitting 0 seconds, 60 second test >>>> [ ID] Interval Transfer Bandwidth >>>> [ 4] 0.00-1.00 sec 255 MBytes 2.14 Gbits/sec >>>> [ 4] 1.00-2.00 sec 253 MBytes 2.12 Gbits/sec >>>> [ 4] 2.00-3.00 sec 255 MBytes 2.14 Gbits/sec >>>> [ 4] 3.00-4.00 sec 255 MBytes 2.14 Gbits/sec >>>> [ 4] 4.00-5.00 sec 255 MBytes 2.14 Gbits/sec >>>> [ 4] 5.00-6.00 sec 257 MBytes 2.15 Gbits/sec >>>> [ 4] 6.00-7.00 sec 253 MBytes 2.13 Gbits/sec >>>> [ 4] 7.00-8.00 sec 254 MBytes 2.13 Gbits/sec >>>> [ 4] 8.00-9.00 sec 255 MBytes 2.14 Gbits/sec >>>> [ 4] 9.00-10.00 sec 252 MBytes 2.12 Gbits/sec >>>> (etc) >>>> >>>> but with 3.14 the numbers as all over the place: >>>> >>>> [root@Lab200slot2 ~]# iperf3 --sctp -4 -c 192.168.241.3 -V -l 1452 -t >>>> 60 iperf version 3.0.1 (10 January 2014) Linux Lab200slot2 3.14.0 #1 >>>> SMP Thu Apr 3 23:18:29 EDT 2014 x86_64 >>>> Time: Fri, 11 Apr 2014 17:56:21 GMT >>>> Connecting to host 192.168.241.3, port 5201 >>>> Cookie: Lab200slot2.1397238981.812898.548918 >>>> [ 4] local 192.168.241.2 port 38616 connected to 192.168.241.3 port >>>> 5201 Starting Test: protocol: SCTP, 1 streams, 1452 byte blocks, omitting 0 seconds, 60 second test >>>> [ ID] Interval Transfer Bandwidth >>>> [ 4] 0.00-1.09 sec 20.8 MBytes 161 Mbits/sec >>>> [ 4] 1.09-2.13 sec 10.8 MBytes 86.8 Mbits/sec >>>> [ 4] 2.13-3.15 sec 3.57 MBytes 29.5 Mbits/sec >>>> [ 4] 3.15-4.16 sec 4.33 MBytes 35.7 Mbits/sec >>>> [ 4] 4.16-6.21 sec 10.4 MBytes 42.7 Mbits/sec >>>> [ 4] 6.21-6.21 sec 0.00 Bytes 0.00 bits/sec >>>> [ 4] 6.21-7.35 sec 34.6 MBytes 253 Mbits/sec >>>> [ 4] 7.35-11.45 sec 22.0 MBytes 45.0 Mbits/sec >>>> [ 4] 11.45-11.45 sec 0.00 Bytes 0.00 bits/sec [ 4] 11.45-11.45 >>>> sec 0.00 Bytes 0.00 bits/sec [ 4] 11.45-11.45 sec 0.00 Bytes >>>> 0.00 bits/sec >>>> [ 4] 11.45-12.51 sec 16.0 MBytes 126 Mbits/sec >>>> [ 4] 12.51-13.59 sec 20.3 MBytes 158 Mbits/sec >>>> [ 4] 13.59-14.65 sec 13.4 MBytes 107 Mbits/sec >>>> [ 4] 14.65-16.79 sec 33.3 MBytes 130 Mbits/sec >>>> [ 4] 16.79-16.79 sec 0.00 Bytes 0.00 bits/sec [ 4] 16.79-17.82 >>>> sec 5.94 MBytes 48.7 Mbits/sec >>>> (etc) >>>> >>>> Note: the difference appears to be SCTP-specific, as I get exactly >>>> the same TCP >>> > throughput in both kernels. >>> >>> Hmm, okay. :/ Could you further bisect on your side to narrow down from which kernel onwards this behaviour can be seen? >>> >>> Thanks, >>> >>> Daniel >>> >>>> -----Original Message----- >>>> From: Daniel Borkmann [mailto:dborkman@xxxxxxxxxx] >>>> Sent: April-11-14 11:22 AM >>>> To: Butler, Peter >>>> Cc: Vlad Yasevich; linux-sctp@xxxxxxxxxxxxxxx >>>> Subject: Re: Is SCTP throughput really this low compared to TCP? >>>> >>>> On 04/11/2014 05:07 PM, Butler, Peter wrote: >>>>> Yes indeed this is ixgbe and I do have SCTP checksum offloading >>>>> enabled. (For what >>>> > it's worth, the checksum offload gives about a 20% throughput >>>> gain >>>> - but this is, > of course, already included in the numbers I posted >>>> to this thread as I've been using > the CRC offload all along.) >>>> >>>> Ok, understood. >>>> >>>>> I re-did all the tests with TSO/GSO/LRO/GRO disabled (on both sides >>>>> of the association - >>>> > i.e. on both endpoint nodes), and using 1452-byte messages instead of 1000-byte > messages. With this new setup, the TCP performance drops significantly, as expected, > while the SCTP performance is boosted, and the playing field is somewhat more 'level'. >>>> > (Note that I could not use 1464-byte messages as suggested by >>>> Vlad, as anything above > 1452 cut the SCTP performance in half - >>>> must have hit the segmentation limit at this > slightly lower >>>> message size. MTU is 1500.) >>>>> >>>>> So comparing "apples to apples" now, TCP only out-performs SCTP by >>>>> approximately 40-70% >>>> > over the various range of network latencies I tested with (RTTs of 0.2 ms, 10 ms, 20 ms, > and 50 ms). 40-70% is still significant, but nowhere near the 200% better (i.e. 3 > times the throughput) I was getting before. >>>>> >>>>> Does this value (i.e. 40-70%) sound reasonable? Is this the >>>>> more-or-less accepted >>>> > performance difference with the current LKSCTP implementation? >>>> >>>> Yes, that sounds reasonable to me. There are still a lot of open todos in terms of performance that we need to tackle over time, e.g. the way chunks are handled, imho, and copies involved in fast path, also that we're heavily using atomic reference counting, and other open issues. >>>> >>>>> Also, for what it's worth, I get better SCTP throughput numbers with >>>>> the older kernel >>>> > (3.4.2) than with the newer kernel (3.14)... >>>> >>>> Interesting, a lot of things happened in between; were you able to bisect/identify a possible commit that causes this? How big is the difference? >>>> > -- > To unsubscribe from this list: send the line "unsubscribe linux-sctp" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
