On 10/14/23 12:04 AM, Kuniyuki Iwashima wrote:
Under SYN Flood, the TCP stack generates SYN Cookie to remain stateless
for the connection request until a valid ACK is responded to the SYN+ACK.
The cookie contains two kinds of host-specific bits, a timestamp and
secrets, so only can it be validated by the generator. It means SYN
Cookie consumes network resources between the client and the server;
intermediate nodes must remember which nodes to route ACK for the cookie.
SYN Proxy reduces such unwanted resource allocation by handling 3WHS at
the edge network. After SYN Proxy completes 3WHS, it forwards SYN to the
backend server and completes another 3WHS. However, since the server's
ISN differs from the cookie, the proxy must manage the ISN mappings and
fix up SEQ/ACK numbers in every packet for each connection. If a proxy
node is down, all the connections through it are also down. Keeping a
state at proxy is painful from that perspective.
At AWS, we use a dirty hack to build truly stateless SYN Proxy at scale.
Our SYN Proxy consists of the front proxy layer and the backend kernel
module. (See slides of netconf [0], p6 - p15)
The cookie that SYN Proxy generates differs from the kernel's cookie in
that it contains a secret (called rolling salt) (i) shared by all the proxy
nodes so that any node can validate ACK and (ii) updated periodically so
that old cookies cannot be validated. Also, ISN contains WScale, SACK, and
ECN, not in TS val. This is not to sacrifice any connection quality, where
some customers turn off the timestamp option due to retro CVE.
After 3WHS, the proxy restores SYN and forwards it and ACK to the backend
server. Our kernel module works at Netfilter input/output hooks and first
feeds SYN to the TCP stack to initiate 3WHS. When the module is triggered
for SYN+ACK, it looks up the corresponding request socket and overwrites
tcp_rsk(req)->snt_isn with the proxy's cookie. Then, the module can
complete 3WHS with the original ACK as is.
This way, our SYN Proxy does not manage the ISN mappings and can stay
stateless. It's working very well for high-bandwidth services like
multiple Tbps, but we are looking for a way to drop the dirty hack and
further optimise the sequences.
If we could validate an arbitrary SYN Cookie on the backend server with
BPF, the proxy would need not restore SYN nor pass it. After validating
ACK, the proxy node just needs to forward it, and then the server can do
the lightweight validation (e.g. check if ACK came from proxy nodes, etc)
and create a connection from the ACK.
This series adds two SOCK_OPS hooks to generate and validate arbitrary
SYN Cookie. Each hook is invoked if BPF_SOCK_OPS_SYNCOOKIE_CB_FLAG is
set to the listening socket in advance by bpf_sock_ops_cb_flags_set().
The user interface looks like this:
BPF_SOCK_OPS_GEN_SYNCOOKIE_CB
input
|- bpf_sock_ops.sk : 4-tuple
|- bpf_sock_ops.skb : TCP header
|- bpf_sock_ops.args[0] : MSS
`- bpf_sock_ops.args[1] : BPF_SYNCOOKIE_XXX flags
output
|- bpf_sock_ops.replylong[0] : ISN (SYN Cookie) ------.
`- bpf_sock_ops.replylong[1] : TS value -----------. |
| |
BPF_SOCK_OPS_CHECK_SYNCOOKIE_CB | |
| |
input | |
|- bpf_sock_ops.sk : 4-tuple | |
|- bpf_sock_ops.skb : TCP header | |
|- bpf_sock_ops.args[0] : ISN (SYN Cookie) <-----'
`- bpf_sock_ops.args[1] : TS value <----------'
output
|- bpf_sock_ops.replylong[0] : MSS
`- bpf_sock_ops.replylong[1] : BPF_SYNCOOKIE_XXX flags
To establish a connection from SYN Cookie, BPF_SOCK_OPS_CHECK_SYNCOOKIE_CB
hook must set a valid MSS to bpf_sock_ops.replylong[0], meaning that
BPF_SOCK_OPS_GEN_SYNCOOKIE_CB hook must encode MSS to ISN or TS val to be
restored in the validation hook.
If WScale, SACK, and ECN are detected to be available in SYN packet, the
corresponding flags are passed to args[0] of BPF_SOCK_OPS_GEN_SYNCOOKIE_CB
so that bpf prog need not parse the TCP header. The same flags can be set
to replylong[0] of BPF_SOCK_OPS_CHECK_SYNCOOKIE_CB to enable each feature
on the connection.
For details, please see each patch. Here's an overview:
patch 1 - 4 : Misc cleanup
patch 5, 6 : Add SOCK_OPS hook (only ISN is available here)
patch 7, 8 : Make TS val available as the second cookie storage
patch 9, 10 : Make WScale, SACK, and ECN configurable from ACK
patch 11 : selftest, need some help from BPF experts...
[0]: https://netdev.bots.linux.dev/netconf/2023/kuniyuki.pdf
Fyi, just as quick feedback, this fails BPF CI selftests :
https://github.com/kernel-patches/bpf/actions/runs/6513838231/job/17694669376
Notice: Success: 427/3396, Skipped: 24, Failed: 1
Error: #274 tcpbpf_user
Error: #274 tcpbpf_user
test_tcpbpf_user:PASS:open and load skel 0 nsec
test_tcpbpf_user:PASS:test__join_cgroup(/tcpbpf-user-test) 0 nsec
test_tcpbpf_user:PASS:attach_cgroup(bpf_testcb) 0 nsec
run_test:PASS:start_server 0 nsec
run_test:PASS:connect_to_fd(listen_fd) 0 nsec
run_test:PASS:accept(listen_fd) 0 nsec
run_test:PASS:send(cli_fd) 0 nsec
run_test:PASS:recv(accept_fd) 0 nsec
run_test:PASS:send(accept_fd) 0 nsec
run_test:PASS:recv(cli_fd) 0 nsec
run_test:PASS:recv(cli_fd) for fin 0 nsec
run_test:PASS:recv(accept_fd) for fin 0 nsec
verify_result:PASS:event_map 0 nsec
verify_result:PASS:bytes_received 0 nsec
verify_result:PASS:bytes_acked 0 nsec
verify_result:PASS:data_segs_in 0 nsec
verify_result:PASS:data_segs_out 0 nsec
verify_result:FAIL:bad_cb_test_rv unexpected bad_cb_test_rv: actual 0 != expected 128
verify_result:PASS:good_cb_test_rv 0 nsec
verify_result:PASS:num_listen 0 nsec
verify_result:PASS:num_close_events 0 nsec
verify_result:PASS:tcp_save_syn 0 nsec
verify_result:PASS:tcp_saved_syn 0 nsec
verify_result:PASS:window_clamp_client 0 nsec
verify_result:PASS:window_clamp_server 0 nsec
