From: openssl-users [mailto:openssl-users-bounces@xxxxxxxxxxx] On Behalf Of Eran Borovik Sent: Monday, January 27, 2020 07:07 > When do I stop? what is the best way to actually determine there can > be no more forward progress both on the send and the receive side, and > epoll must be used? ... > I am afraid that if the send buffer is full and there is nothing to receive, > I will keep getting WANT_READ for receive, and WANT_WRITE for send until > actual data arrives or can be sent which defeats the purpose of epoll. The following is untested and off the top of my head. Think of it this way. You have these possible states: 1. You don't want to do anything with the conversation. It can be closed. 2. You want to receive. There are three possible causes: - You'd like to receive data (or close/error indication) from the peer. - SSL_send returned WANT_READ. - Both of the above are true. 3. You want to send. There are three possible causes: - You have buffered outbound application data (or shutdown/close) which you haven't been able to send yet. - SSL_receive returned WANT_WRITE. - Both of the above are true. 4. You want to receive and to send. So, keep track of the above using a per-conversation pair of want_to_send and want_to_receive variables and use the following algorithm: /* any time we buffer outbound data, set want_to_send = 1 */ want_to_receive = 1 /* at start of conversation, we want data from peer */ while want_to_send or want_to_receive { /* Wait until we can do something we want to do */ do_recv = false, do_send = false if want_to_send and want_to_receive { epoll until socket is readable or writable do_recv = readable do_send = writable } else if want_to_send { epoll until socket is writable do_send = true } else if want_to_receive { epoll until socket is readable do_recv = true } /* Now perform I/O */ if do_recv { SSL_read(...) if WANT_WRITE want_to_send = true else if WANT_READ want_to_receive = true /* couldn't send enough to clear WANT_WRITE */ else if error ... else want_to_receive = false /* maybe, depending on application */ process_received_data(...) } if do_send { SSL_write(...) if WANT_READ want_to_receive = true else if WANT_WRITE want_to_send = true /* couldn't send enough to clear WANT_WRITE */ else if error ... else { update send buffer info if all buffered data sent want_to_send = false /* any pending WANT_WRITE should now be clear */ } } } close conversation Now, in practice, you probably always want to poll on readability even if you're not expecting data from the peer, because you'll want to be notified of peer close. And you may want to be able to break out of an epoll for readability only in order to send data, depending on your application design and requirements. You can do that using a timeout and polling the want_to_send state variable, or using an internal channel such as a pipe or socket which the polling thread adds to the epoll readable-descriptor set, and the sending thread writes a wakeup message to. It's an open design question if you want to set want_to_receive to false in the do_recv body, as I have it above, and then set it back to true if you decide you're not done with the conversation in process_received_data; or leave it set to true and set it to false only when you've received a close indication from the peer; or possibly some other behavior that's appropriate for your application. In brief: at the top of the loop, figure out if you want to try to send, receive, or both. The types of I/O you want to do is the union of the application state and the OpenSSL WANT_* state. Then poll for all the types of I/O you want, and when the socket is available for any of them, attempt that type. Deal with any successful operation, and loop. In the "problem case" you described above, this code will discover that you want to receive and to send, so it will first poll for either type of availability on the socket. If the socket ends up readable, you'll do a receive, which may succeed or may return WANT_WRITE or may indicate a peer close or may indicate an error. It could also return WANT_READ if it's not able to receive enough TLS data to satisfy a pending WANT_READ. In that case we keep want_to_receive set and go around the loop again. And if the socket is writable (regardless of whether it's also readable), it will try to send some of your application data; that will also try to handle any pending WANT_WRITE condition. The result might be sending all the application data, sending some or none of the application data, satisfying a WANT_WRITE condition, failing to send enough to satisfy the WANT_WRITE, getting an error, etc. If we do send some application data, we can assume any pending WANT_WRITE from the previous receive operation was satisfied. -- Michael Wojcik Distinguished Engineer, Micro Focus
