On Sun, Apr 21, 2024 at 3:27 PM Stefan Metzmacher <metze@xxxxxxxxx> wrote: > > Am 20.04.24 um 21:32 schrieb Xin Long: > > On Fri, Apr 19, 2024 at 3:19 PM Xin Long <lucien.xin@xxxxxxxxx> wrote: > >> > >> On Fri, Apr 19, 2024 at 2:51 PM Stefan Metzmacher <metze@xxxxxxxxx> wrote: > >>> > >>> Hi Xin Long, > >>> > >>>>> But I think its unavoidable for the ALPN and SNI fields on > >>>>> the server side. As every service tries to use udp port 443 > >>>>> and somehow that needs to be shared if multiple services want to > >>>>> use it. > >>>>> > >>>>> I guess on the acceptor side we would need to somehow detach low level > >>>>> udp struct sock from the logical listen struct sock. > >>>>> > >>>>> And quic_do_listen_rcv() would need to find the correct logical listening > >>>>> socket and call quic_request_sock_enqueue() on the logical socket > >>>>> not the lowlevel udo socket. The same for all stuff happening after > >>>>> quic_request_sock_enqueue() at the end of quic_do_listen_rcv. > >>>>> > >>>> The implementation allows one low level UDP sock to serve for multiple > >>>> QUIC socks. > >>>> > >>>> Currently, if your 3 quic applications listen to the same address:port > >>>> with SO_REUSEPORT socket option set, the incoming connection will choose > >>>> one of your applications randomly with hash(client_addr+port) vi > >>>> reuseport_select_sock() in quic_sock_lookup(). > >>>> > >>>> It should be easy to do a further match with ALPN between these 3 quic > >>>> socks that listens to the same address:port to get the right quic sock, > >>>> instead of that randomly choosing. > >>> > >>> Ah, that sounds good. > >>> > >>>> The problem is to parse the TLS Client_Hello message to get the ALPN in > >>>> quic_sock_lookup(), which is not a proper thing to do in kernel, and > >>>> might be rejected by networking maintainers, I need to check with them. > >>> > >>> Is the reassembling of CRYPTO frames done in the kernel or > >>> userspace? Can you point me to the place in the code? > >> In quic_inq_handshake_tail() in kernel, for Client Initial packet > >> is processed when calling accept(), this is the path: > >> > >> quic_accept()-> quic_accept_sock_init() -> quic_packet_process() -> > >> quic_packet_handshake_process() -> quic_frame_process() -> > >> quic_frame_crypto_process() -> quic_inq_handshake_tail(). > >> > >> Note that it's with the accept sock, not the listen sock. > >> > >>> > >>> If it's really impossible to do in C code maybe > >>> registering a bpf function in order to allow a listener > >>> to check the intial quic packet and decide if it wants to serve > >>> that connection would be possible as last resort? > >> That's a smart idea! man. > >> I think the bpf hook in reuseport_select_sock() is meant to do such > >> selection. > >> > >> For the Client initial packet (the only packet you need to handle), > >> I double you will need to do the reassembling, as Client Hello TLS message > >> is always less than 400 byte in my env. > >> > >> But I think you need to do the decryption for the Client initial packet > >> before decoding it then parsing the TLS message from its crypto frame. > > I created this patch: > > > > https://github.com/lxin/quic/commit/aee0b7c77df3f39941f98bb901c73fdc560befb8 > > > > to do this decryption in quic_sock_look() before calling > > reuseport_select_sock(), so that it provides the bpf selector with > > a plain-text QUIC initial packet: > > > > https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.2 > > > > If it's complex for you to do the decryption for the initial packet in > > the bpf selector, I will apply this patch. Please let me know. > > I guess in addition to quic_server_handshake(), which is called > after accept(), there should be quic_server_prepare_listen() > (and something similar for in kernel servers) that setup the reuseport > magic for the socket, so that it's not needed in every application. > > It seems there is only a single ebpf program possible per > reuseport group, so there has to be just a single one. > > But is it possible for in kernel servers to also register an epbf program? > Just confirmed from other ebpf experts, there are no in-kernel interfaces for loading and interacting with BPF maps/programs(other than from BPF itself). It seems that we have to do this match in QUIC stack. In the latest QUIC code, I added quic_packet_get_alpn(), a 59-line function, to parse ALPNs and then it will search for the listen sock with these ALPNs in quic_sock_lookup(). I introduced 'alpn_match' module param, and it can be enabled when loading the module QUIC by: # modprobe quic alpn_match=1 You can test it by tests/sample_test in the latest code: Start 3 servers: # ./sample_test server 0.0.0.0 1234 \ ./keys/server-key.pem ./keys/server-cert.pem smbd # ./sample_test server 0.0.0.0 1234 \ ./keys/server-key.pem ./keys/server-cert.pem h3 # ./sample_test server 0.0.0.0 1234 \ ./keys/server-key.pem ./keys/server-cert.pem ksmbd Try to connect on clients with: # ./sample_test client 127.0.0.1 1234 ksmbd # ./sample_test client 127.0.0.1 1234 smbd # ./sample_test client 127.0.0.1 1234 h3 to see if the corresponding server responds. There might be some concerns but it's also a useful feature that can not be implemented in userland QUICs. The commit is here: https://github.com/lxin/quic/commit/de82f8135f4e9196b503b4ab5b359d88f2b2097f Please check if this is enough for SMB applications. Note as a listen socket is now identified by [address + port + ALPN] when alpn_match=1, this feature does NOT require SO_REUSEPORT socket option to be set, unless one wants multiple sockets to listen to the same [address + port + ALPN]. Thanks.
