OK.... let's do an experiment.
Two c5.9xlarge instances.
$ ./netperf -H 10.0.161.101 -t UDP_STREAM -- -m 8970
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.161.101 () port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
212992 8970 10.00 1114099 0 7994.75
212992 10.00 1114099 7994.75
OK, it gets less impressive if I limit myself to 1400-byte packets:
$ ./netperf -H 10.0.161.101 -t UDP_STREAM -- -m 1400
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.161.101 () port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
212992 1400 10.00 5077116 0 5686.36
212992 10.00 5074446 5683.37
So... kernel ESP between the two. Here's a script which sets it up:
cat > espsetup.sh <<EOF
#!/bin/sh
KEY=0x1234567890123456789012345678901234567890123456789012345678901234
SPI1=0x12345678
SPI2=0x87654321
SRC=10.0.161.101
DST=10.0.186.131
LOCAL=172.16.0.1
REMOTE=172.16.0.2
if ip addr list dev eth0 | grep -q $DST; then
xDST=$SRC
SRC=$DST
DST=$xDST
xLOCAL=$REMOTE
REMOTE=$LOCAL
LOCAL=$xLOCAL
xSPI1=$SPI2
SPI2=$SPI1
SPI1=$xSPI1
fi
echo $SPI1 $SPI2
ip xfrm state del src $SRC dst $DST proto esp spi $SPI1
ip xfrm state del src $DST dst $SRC proto esp spi $SPI2
ip xfrm state add src $SRC dst $DST proto esp spi $SPI1 reqid $SPI1 mode tunnel auth sha1 $KEY enc aes $KEY encap espinudp 8443 8443 $SRC
ip xfrm state add src $DST dst $SRC proto esp spi $SPI2 reqid $SPI2 mode tunnel auth sha1 $KEY enc aes $KEY encap espinudp 8443 8443 $SRC
ip xfrm policy add src $LOCAL dst $REMOTE dir out tmpl src $SRC dst $DST proto esp reqid $SPI1 mode tunnel
ip xfrm policy add src $REMOTE dst $LOCAL dir in tmpl src $DST dst $SRC proto esp reqid $SPI2 mode tunnel
ip addr add $LOCAL dev lo
ip route add $REMOTE dev eth0 src $LOCAL
EOF
And if you want the kernel to actually receive and process the UDP frames, you have to run something that binds the socket...
(http://techblog.newsnow.co.uk/2011/11/simple-udp-esp-encapsulation-nat-t-for.html)
cat > esplisten.pl <<EOF
#!/usr/bin/perl -w
use strict;
use Socket qw( IPPROTO_IP IPPROTO_UDP AF_INET SOCK_DGRAM SOL_SOCKET SO_REUSEADDR INADDR_ANY sockaddr_in );
my $UDP_ENCAP = 100;
# UDP encapsulation types
my $UDP_ENCAP_ESPINUDP_NON_IKE = 1; # /* draft-ietf-ipsec-nat-t-ike-00/01 */
my $UDP_ENCAP_ESPINUDP = 2; # /* draft-ietf-ipsec-udp-encaps-06 */
my $UDP_ENCAP_L2TPINUDP = 3; # /* rfc2661 */
my $Sock;
socket( $Sock, AF_INET, SOCK_DGRAM, IPPROTO_UDP ) || die "::socket: $!";
setsockopt( $Sock, SOL_SOCKET, SO_REUSEADDR, pack( "l", 1 ) );
# struct sadb_x_policy {
# uint16_t sadb_x_policy_len;
# uint16_t sadb_x_policy_exttype;
# uint16_t sadb_x_policy_type;
# uint8_t sadb_x_policy_dir;
# uint8_t sadb_x_policy_reserved;
# uint32_t sadb_x_policy_id;
# uint32_t sadb_x_policy_priority;
# } __attribute__((packed));
# /* sizeof(struct sadb_x_policy) == 16 */
my $SADB_X_EXT_POLICY = 18;
my $IP_IPSEC_POLICY = 16;
my $IPSEC_POLICY_BYPASS = 4;
my $IPSEC_DIR_INBOUND = 1;
my $IPSEC_DIR_OUTBOUND = 2;
# policy.sadb_x_policy_len = sizeof(policy) / sizeof(u_int64_t);
# policy.sadb_x_policy_exttype = SADB_X_EXT_POLICY;
# policy.sadb_x_policy_type = IPSEC_POLICY_BYPASS;
# policy.sadb_x_policy_dir = IPSEC_DIR_OUTBOUND;
my $policy1 = pack("SSSCCLL", 2, $SADB_X_EXT_POLICY, $IPSEC_POLICY_BYPASS, $IPSEC_DIR_OUTBOUND, 0, 0, 0);
my $policy2 = pack("SSSCCLL", 2, $SADB_X_EXT_POLICY, $IPSEC_POLICY_BYPASS, $IPSEC_DIR_INBOUND, 0, 0, 0);
# See http://strongswan.sourcearchive.com/documentation/4.1.4/socket_8c-source.html
if( defined setsockopt( $Sock, IPPROTO_IP, $IP_IPSEC_POLICY, $policy1 ) ) {
print "setsockopt:: policy OK\n"; }
else { print "setsockopt:: policy FAIL\n"; }
if( defined setsockopt( $Sock, IPPROTO_IP, $IP_IPSEC_POLICY, $policy2 ) ) {
print "setsockopt:: policy OK\n"; }
else { print "setsockopt:: policy FAIL\n"; }
if( defined setsockopt( $Sock, IPPROTO_UDP, $UDP_ENCAP, $UDP_ENCAP_ESPINUDP) ) {
print "setsockopt:: UDP_ENCAP OK\n"; }
else { print "setsockopt:: UDP_ENCAP FAIL\n"; }
bind( $Sock, sockaddr_in( 8443, INADDR_ANY ) ) || die "::bind: $!";
sleep;
1;
EOF
Ok, how does that perform?
$ sudo perf record -a ./netperf -H 172.16.0.1 -t UDP_STREAM -- -m 1400
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.0.1 () port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
212992 1400 10.00 1198093 0 1341.86
212992 10.00 1198044 1341.80
At this point netperf is taking 100% of CPU, doing this:
Samples: 49K of event 'cycles', Event count (approx.): 32648308583
Overhead Command Shared Object Symbol
31.59% netperf [kernel.vmlinux] [k] sha_transform
17.49% netperf [kernel.vmlinux] [k] _aesni_enc1
2.88% netperf [kernel.vmlinux] [k] _raw_spin_lock
2.22% netperf [ena] [k] ena_start_xmit
1.26% netperf [kernel.vmlinux] [k] aesni_cbc_enc
1.24% swapper [kernel.vmlinux] [k] intel_idle
1.23% netperf [ena] [k] ena_com_prepare_tx
1.13% netperf [kernel.vmlinux] [k] fib_table_lookup
1.02% netperf [kernel.vmlinux] [k] entry_SYSCALL_64
0.99% netperf [kernel.vmlinux] [k] csum_partial_copy_generic
Next step... tricking OpenConnect on one end into talking to the kernel ESP on the other.
With openssl s_server and a handful of canned responses, let's see how far I can get...
HTTP/1.1 200 OK
Content-Length: 207
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<response>
<ip-address>172.16.0.1</ip-address>
<netmask>255.255.255.0</netmask>
<mtu>1200</mtu>
<gw-address>172.16.0.2</gw-address>
</response>
HTTP/1.1 200 OK
Content-Length: 122
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<response>
<hip-report-needed>no</hip-report-needed>
</response>
START_TUNNEL
I think I just need to add the ESP config into that initial config
response, then hack openconnect to bind to local UDP port 8443...
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ openconnect-devel mailing list openconnect-devel@xxxxxxxxxxxxxxxxxxx http://lists.infradead.org/mailman/listinfo/openconnect-devel
