Am 13.03.24 um 20:39 schrieb Xin Long:
On Wed, Mar 13, 2024 at 1:28 PM Stefan Metzmacher <metze@xxxxxxxxx> wrote:
Am 13.03.24 um 17:03 schrieb Xin Long:
On Wed, Mar 13, 2024 at 4:56 AM Stefan Metzmacher <metze@xxxxxxxxx> wrote:
Hi Xin Long,
first many thanks for working on this topic!
Hi, Stefan
Thanks for the comment!
Usage
=====
This implementation supports a mapping of QUIC into sockets APIs. Similar
to TCP and SCTP, a typical Server and Client use the following system call
sequence to communicate:
Client Server
------------------------------------------------------------------
sockfd = socket(IPPROTO_QUIC) listenfd = socket(IPPROTO_QUIC)
bind(sockfd) bind(listenfd)
listen(listenfd)
connect(sockfd)
quic_client_handshake(sockfd)
sockfd = accecpt(listenfd)
quic_server_handshake(sockfd, cert)
sendmsg(sockfd) recvmsg(sockfd)
close(sockfd) close(sockfd)
close(listenfd)
Please note that quic_client_handshake() and quic_server_handshake() functions
are currently sourced from libquic in the github lxin/quic repository, and might
be integrated into ktls-utils in the future. These functions are responsible for
receiving and processing the raw TLS handshake messages until the completion of
the handshake process.
I see a problem with this design for the server, as one reason to
have SMB over QUIC is to use udp port 443 in order to get through
firewalls. As QUIC has the concept of ALPN it should be possible
let a conumer only listen on a specif ALPN, so that the smb server
and web server on "h3" could both accept connections.
We do provide a sockopt to set ALPN before bind or handshaking:
https://github.com/lxin/quic/wiki/man#quic_sockopt_alpn
But it's used more like to verify if the ALPN set on the server
matches the one received from the client, instead of to find
the correct server.
Ah, ok.
Just note that, with a bit change in the current libquic, it still
allows users to use ALPN to find the correct function or thread in
the *same* process, usage be like:
listenfd = socket(IPPROTO_QUIC);
/* match all during handshake with wildcard ALPN */
setsockopt(listenfd, QUIC_SOCKOPT_ALPN, "*");
bind(listenfd)
listen(listenfd)
while (1) {
sockfd = accept(listenfd);
/* the alpn from client will be set to sockfd during handshake */
quic_server_handshake(sockfd, cert);
getsockopt(sockfd, QUIC_SOCKOPT_ALPN, alpn);
Would quic_server_handshake() call setsockopt()?
switch (alpn) {
case "smbd": smbd_thread(sockfd);
case "h3": h3_thread(sockfd);
case "ksmbd": ksmbd_thread(sockfd);
}
}
Ok, but that would mean all application need to be aware of each other,
but it would be possible and socket fds could be passed to other
processes.
So you expect (k)smbd server and web server both to listen on UDP
port 443 on the same host, and which APP server accepts the request
from a client depends on ALPN, right?
yes.
Got you. This can be done by also moving TLS 1.3 message exchange to
kernel where we can get the ALPN before looking up the listening socket.
However, In-kernel TLS 1.3 Handshake had been NACKed by both kernel
netdev maintainers and userland ssl lib developers with good reasons.
Currently, in Kernel, this implementation doesn't process any raw TLS
MSG/EXTs but deliver them to userspace after decryption, and the accept
socket is created before processing handshake.
I'm actually curious how userland QUIC handles this, considering
that the UDP sockets('listening' on the same IP:PORT) are used in
two different servers' processes. I think socket lookup with ALPN
has to be done in Kernel Space. Do you know any userland QUIC
implementation for this?
I don't now, but I guess QUIC is only used for http so
far and maybe dns, but that seems to use port 853.
So there's no strict need for it and the web server
would handle all relevant ALPNs.
Honestly, I don't think any userland QUIC can use ALPN to lookup for
different sockets used by different servers/processes. As such thing
can be only done in Kernel Space.
So the server application should have a way to specify the desired
ALPN before or during the bind() call. I'm not sure if the
ALPN is available in cleartext before any crypto is needed,
so if the ALPN is encrypted it might be needed to also register
a server certificate and key together with the ALPN.
Because multiple application may not want to share the same key.
On send side, ALPN extension is in raw TLS messages created in userspace
and passed into the kernel and encoded into QUIC crypto frame and then
*encrypted* before sending out.
Ok.
On recv side, after decryption, the raw TLS messages are decoded from
the QUIC crypto frame and then delivered to userspace, so in userspace
it processes certificate validation and also see cleartext ALPN.
Let me know if I don't make it clear.
But the first "new" QUIC pdu from will trigger the accept() to
return and userspace (or the kernel helper function) will to
all crypto? Or does the first decryption happen in kernel (before accept returns)?
Good question!
The first "new" QUIC pdu will cause to create a 'request sock' (contains
4-tuple and connection IDs only) and queue up to reqsk list of the listen
sock (if validate_peer_address param is not set), and this pdu is enqueued
in the inq->backlog_list of the listen sock.
When accept() is called, in Kernel, it dequeues the "request sock" from the
reqsk list of the listen sock, and creates the accept socket based on this
reqsk. Then it processes the pdu for this new accept socket from the
inq->backlog_list of the listen sock, including *decrypting* QUIC packet
and decoding CRYPTO frame, then deliver the raw/cleartext TLS message to
the Userspace libquic.
Ok, when the kernel already decrypts it could already
look find the ALPN. It doesn't mean it should do the full
handshake, but parse enough to find the ALPN.
But I don't yet understand how the kernel gets the key to
do the initlal decryption, I'd assume some call before listen()
need to tell the kernel about the keys.
Then in Userspace libquic, it handles the received TLS message and creates
a new raw/cleartext TLS message for response via libgnutls, and delivers to
kernel. In kernel, it will encode this message to a CRYPTO frame in a QUIC
packet and then *encrypt* this QUIC packet and send it out.
So as you can see, there's no en/decryption happening in Userspace. In
Userspace libquic, it only does raw/cleartext TLS message exchange. ALL
en/decryption happens in Kernel Space, as these en/decryption are done
against QUIC packets, not directly against the TLS messages.
Maybe it would be possible to optionally have socket option to
register ALPNs with certificates so that tls_server_hello_x509()
could be called automatically before accept returns (even for
userspace consumers).
It may mean the tlshd protocol needs to be extended...
so that userspace consumers don't need quic_client/server_handshake(), and
accept() returns a socket that already has the handshake done, right?
We didn't do that, as:
1. It's not a good idea for Userspace consumers' applications to reply on
a daemon like tlshd, not convenient for users, also a bit weird for
userspace app to ask another userspace app to help do the handshake.
2. It's too complex to implement, especially if we also want to call
tls_client_hello_x509() before connect() returns on client side.
3. For Kernel usage, I prefer leaving this to the kernel consumers for
more flexibility for handshake requests.
As for the ALPNs with certificates, not sure if I understand correctly.
But if you want the server to select certificates according to the ALPN
received from the client during handshake. I think it could be done in
userspace libquic. But yes, tlshd service may also need to extend.
I was just brainstorming for ideas...
metze
