Fwd: Proper API usage with DTLS over custom net transport

[Павел Балашов <balpvl@xxxxxxxxx>]

Hello, 

I would be really grateful if someone can point me in the right direction with proper OpenSSL API usage in the following scenario.

I have a custom network transport - ICE (essentially UDP socket, as a part of typical WebRTC stack) and I need to implement a DTLS connection over it.

So given some hints from the internet I assume that using BIO_s_mem is a tool of choice in this situation.

Suppose SSL_CTX, SSL, BIOs are created and ready.
Overall my current understanding of API usage is somewhere like this (very broad strokes, kind of pseudocode):

if (client)  SSL_set_connect_state();

if (server) SSL_set_accept_state();
while (true)
{

    // On initial iteration this would make OpenSSL put some data for us to read from out_BIO to send it over

    // if we are client, and don't put anything if we are server

    // On following iterations pretty much the same behaviour remains.
    SSL_do_handshake();

    if (out_BIO_has_data) send_data_from_out_BIO_to_net();

    // So at some loop iteration on line above we've sent everything OpenSSL wanted us to send and 

    // OpenSSL tells us handshake is done, so we quit the loop

    if (SSL_is_init_finished()) break;

    // We've sent some flight of DTLS handshake, but there could be need to retransmit

    // we ask for retransmit timeout

    timeout = DTLSv1_get_timeout();

    // "select"-like machinery

    wait_for_timeout_or_socket_data();

    if (select_resulted_in_timeout)

    {

        // I assume this is going to generate DTLS-protocol-wise correct retransmission flight data in out_BIO

        DTLSv1_handle_timeout();

        continue;
    }

    // socket_data that is going to be put in in_BIO is about to be processed by OpenSSL internals in next SSL_do_handshake() on next loop iter above

    if (select_resulted_in_socket_data) put_socket_data_to_in_BIO();

}

    
// So here I assume as client we've got last flight from server and ready to send data

// And if we are a server we've sent out last flight, but we don't know whether it has got through

// So if everything more or less correct up to this point we can expect multiplexed traffic on ICE "socket": SRTP, SRTCP, and DTLS

// DeMultiplexing is not a problem (some RFCs cover this)
while (true)
{

    sock_data = read_socket_data();

    if (is_rtp(sock_data)) ...;

    if (is_rtcp(sock_data)) ...;

    if (is_dtls(sock_data)) ????
}

So now the questions:

(1) If we receive some dtls data at the line above with '????' what should we do in terms of OpenSSL API calls ?  I assume this dtls data could be a client's retransmission due to server's last flight was lost or this could be client receiving server's last flight duplicated (theoretically could happen as long as lower layer protocol is UDP) or this could be DTLS-encrypted real-app data or this could be DTLS-renegotiation, this also could be a DTLS shutdown alert and anything else DTSL-related. What is the supposed way of inferring and reacting to those different events with API ?

(2) Is the whole usage of OpenSSL even right for this scenario - maybe the structure and sequence of API calls should be rearranged somehow ?

(3) There is an option to pass custom info_callback with SSL_CTX_set_info_callback(). Would there be a proper usage of this kind of callback in this scenario ?

Any other input, links to any kind of relevant supplemental material is really appreciated.

Thanks a lot for reading, very special thanks to authors and maintainers for all the hard work on this project.

---

Regards,
Pavel.