Under SYN Flood, the TCP stack generates SYN Cookie to remain stateless
for the connection request until a valid ACK is responded to the SYN+ACK.
The cookie contains two kinds of host-specific bits, a timestamp and
secrets, so only can it be validated by the generator. It means SYN
Cookie consumes network resources between the client and the server;
intermediate nodes must remember which nodes to route ACK for the cookie.
SYN Proxy reduces such unwanted resource allocation by handling 3WHS at
the edge network. After SYN Proxy completes 3WHS, it forwards SYN to the
backend server and completes another 3WHS. However, since the server's
ISN differs from the cookie, the proxy must manage the ISN mappings and
fix up SEQ/ACK numbers in every packet for each connection. If a proxy
node is down, all the connections through it are also down. Keeping a
state at proxy is painful from that perspective.
At AWS, we use a dirty hack to build truly stateless SYN Proxy at scale.
Our SYN Proxy consists of the front proxy layer and the backend kernel
module. (See slides of LPC2023 [0], p37 - p48)
The cookie that SYN Proxy generates differs from the kernel's cookie in
that it contains a secret (called rolling salt) (i) shared by all the proxy
nodes so that any node can validate ACK and (ii) updated periodically so
that old cookies cannot be validated. Also, ISN contains WScale, SACK, and
ECN, not in TS val. This is not to sacrifice any connection quality, where
some customers turn off the timestamp option due to retro CVE.
After 3WHS, the proxy restores SYN and forwards it and ACK to the backend
server. Our kernel module works at Netfilter input/output hooks and first
feeds SYN to the TCP stack to initiate 3WHS. When the module is triggered
for SYN+ACK, it looks up the corresponding request socket and overwrites
tcp_rsk(req)->snt_isn with the proxy's cookie. Then, the module can
complete 3WHS with the original ACK as is.
This way, our SYN Proxy does not manage the ISN mappings and can remain
stateless. It's working very well for high-bandwidth services like
multiple Tbps, but we are looking for a way to drop the dirty hack and
further optimise the sequences.
If we could validate an arbitrary SYN Cookie on the backend server with
BPF, the proxy would need not restore SYN nor pass it. After validating
ACK, the proxy node just needs to forward it, and then the server can do
the lightweight validation (e.g. check if ACK came from proxy nodes, etc)
and create a connection from the ACK.
This series adds a new kfunc available on TC to create a reqsk and
configure it based on the argument populated from SYN Cookie.
Usage:
struct tcp_cookie_attributes attr = {
.tcp_opt = {
.mss_clamp = mss,
.wscale_ok = wscale_ok,
.snd_scale = send_scale, /* < 15 */
.tstamp_ok = tstamp_ok,
.sack_ok = sack_ok,
},
.ecn_ok = ecn_ok,
.usec_ts_ok = usec_ts_ok,
};
skc = bpf_skc_lookup_tcp(...);
sk = (struct sock *)bpf_skc_to_tcp_sock(skc);
bpf_sk_assign_tcp_reqsk(skb, sk, attr, sizeof(attr));
bpf_sk_release(skc);
For details, please see each patch. Here's an overview:
Patch 1 - 6 : Misc cleanup
Patch 7, 8 : Factorise non-BPF SYN Cookie handling
Patch 9, 10 : Support arbitrary SYN Cookie with BPF
Patch 11 : Selftest
[0]: https://lpc.events/event/17/contributions/1645/attachments/1350/2701/SYN_Proxy_at_Scale_with_BPF.pdf
Changes:
v3:
Patch 10:
* Guard kfunc and req->syncookie part in inet6?_steal_sock() with
CONFIG_SYN_COOKIE (kernel test robot)
v2: https://lore.kernel.org/netdev/20231120222341.54776-1-kuniyu@xxxxxxxxxx/
* Drop SOCK_OPS and move SYN Cookie validation logic to TC with kfunc.
* Add cleanup patches to reduce discrepancy between cookie_v[46]_check()
v1: https://lore.kernel.org/bpf/20231013220433.70792-1-kuniyu@xxxxxxxxxx/
Kuniyuki Iwashima (11):
tcp: Clean up reverse xmas tree in cookie_v[46]_check().
tcp: Cache sock_net(sk) in cookie_v[46]_check().
tcp: Clean up goto labels in cookie_v[46]_check().
tcp: Don't pass cookie to __cookie_v[46]_check().
tcp: Don't initialise tp->tsoffset in tcp_get_cookie_sock().
tcp: Move TCP-AO bits from cookie_v[46]_check() to tcp_ao_syncookie().
tcp: Factorise cookie req initialisation.
tcp: Factorise non-BPF SYN Cookie handling.
bpf: tcp: Handle BPF SYN Cookie in cookie_v[46]_check().
bpf: tcp: Support arbitrary SYN Cookie.
selftest: bpf: Test bpf_sk_assign_tcp_reqsk().
include/linux/netfilter_ipv6.h | 8 +-
include/net/inet6_hashtables.h | 16 +-
include/net/inet_hashtables.h | 16 +-
include/net/tcp.h | 49 +-
include/net/tcp_ao.h | 6 +-
net/core/filter.c | 113 +++-
net/core/sock.c | 14 +-
net/ipv4/syncookies.c | 273 +++++----
net/ipv4/tcp_ao.c | 16 +-
net/ipv6/syncookies.c | 112 ++--
net/netfilter/nf_synproxy_core.c | 4 +-
tools/testing/selftests/bpf/bpf_kfuncs.h | 10 +
.../bpf/prog_tests/tcp_custom_syncookie.c | 163 +++++
.../selftests/bpf/progs/test_siphash.h | 64 ++
.../bpf/progs/test_tcp_custom_syncookie.c | 570 ++++++++++++++++++
.../bpf/progs/test_tcp_custom_syncookie.h | 161 +++++
16 files changed, 1393 insertions(+), 202 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/tcp_custom_syncookie.c
create mode 100644 tools/testing/selftests/bpf/progs/test_siphash.h
create mode 100644 tools/testing/selftests/bpf/progs/test_tcp_custom_syncookie.c
create mode 100644 tools/testing/selftests/bpf/progs/test_tcp_custom_syncookie.h
--
2.30.2
