Hi Kadir, I think this is something that I once wanted to implement during my GSoC project: On-the-wire encryption, but I am not capable of finishing that, so I am really curious to see your code, can you share a branch? Regards, Zhao On Wed, Mar 7, 2018 at 10:12 AM, Kadir Ozdemir <kozdemir@xxxxxxxxxxxxxx> wrote: > Hi All, > > > We cannot deploy Ceph in our data centers without encrypting its > network traffic. We have decided to implement SSL/TLS within the > Messenger layer using the OpenSSL library on our jewel branch. > Although our implementation has not been completed, we do not see any > obstacles to complete and deploy it. I would like to give some > overview, sketch the design in the rest of this email, and get your > feedback. If the community is interested in merging this effort to the > upstream, we will be very happy to collaborate and contribute. > > Two main structures of the OpenSSL library are SSL Context and SSL. An > SSL Context object holds certificates, a private key, and options > regarding the TLS protocol and algorithms. An SSL Context object is > used to provide the context to create SSL objects which represent SSL > sessions. SSL objects are responsible for encryption, and session > handshake and renegotiation among other things. > > Ceph Messengers have implemented somewhat sophisticated control logic > over the socket layer, and require a different programing model than > the simple way of using OpenSSL. OpenSSL has a rich set of APIs at > different abstraction levels. In order to decouple the TLS operations > from the underlying I/O layer (that is, in our case the socket layer), > OpenSSL provides an abstraction for handling input and output at the > encrypted side of TLS, which is referred as BIO (Basic Input Output). > We will be using a pair of (i.e., input and output) memory BIOs for > buffering encrypted data to be received from or sent to the socket > layer. BIOs allow us to preserve the implementation of the existing > socket handling logic within the Ceph networking layer. > > When TLS is enabled on top of a TCP connection, TLS inserts handshake > and renegotiation messages to the TCP stream and encapsulates the > plaintext stream of bytes transferred by an application into TLS > records. An application in our context is a Ceph Messenger. The TLS > handshake happens at the beginning of a TLS session as part of > establishing the session. Before any application data can be sent or > received over a TLS session, the TLS handshake has to be completed. > During handshake, the end points of the session are verified using PKI > certificates, and encryption key is established among other things. > > Assuming that TLS handshake is completed (after TCP connection is > established), now we can trace the send path. The send path starts > with calling SSL_write() which takes the handle for the SSL object, > the address of a data buffer, and the length of the buffer. The SSL > object encrypts the data into one or more TLS records using its > internal buffer and then writes the TLS records to the output BIO. > When SSL_write() succeeds, we know that the data is encrypted and > copied to the internal buffer of the output BIO. The next step is to > read the encrypted data from the output BIO, and write it to the > socket. In order to do this operation, a separate buffer needs to be > used to copy data from the output BIO to the socket. > > TLS can be in the middle of renegotiating while the application > attempts to receive or send data. Please note that the SSL object can > only write to or read from a BIO in the context of the caller (i.e., > the calling thread). This is one of the reasons, it fails SSL_write or > SSL_read calls to inform the caller that it wants to read or write. In > this case, the (SSL_write or SSL_read) operation needs to be repeated > after the necessary action is taken. If the SSL object needs to read, > then the application needs to read more encrypted data from the > socket, and write it to the input BIO. If the SSL object needs to > write, then the data from the output BIO needs to be read and written > to the socket. > > The read path is a bit more complicated as the SSL object may have > some remaining plaintext from the earlier SSL_read operation. So, the > application needs to attempt to read from the SSL object first using > SSL_read(). If there is no leftover data, then SSL_read() will return > zero bytes. Then, the application needs to check why the read failed > using SSL_get_error() as in the case of SSL_write() failure. If the > SSL layer wants to read as expected when there is no data in the SSL > object, then the application needs to receive more encrypted data from > the socket, push this data to the input BIO using BIO_write(), and > attempt to read the plaintext using the SSL_read() operation on the > SSL object. > > There are two main objectives that shape the design to be described > here. The first one is to change the existing Ceph code and behavior > minimally. The second one is to support Simple and Async Messengers. > The main ideas behind the design are as follows: > > - Introduce a class called Socket to replace the socket descriptor in > the Pipe and AsyncConnection classes. Socket will wrap the socket > operations used currently in both Messengers. The socket descriptor > will be an attribute of Socket. Essentially, the system calls for > socket operations such as send and recv will be replaced by the > corresponding member functions in this class. This is to achieve the > minimal code change objective. Socket implements the plain TCP socket. > > - Introduce a class called TlsSocket to implement TLS specific > behavior common to both Simple and Async Messengers, such as > retrieving SSL Context, initiating TLS handshake, and reading from and > writing to the SSL object. It maintains two sets each of which > contains a lock, buffer, and BIO object; one set for the receive path > and the other for the send path. The locks serialize the access to > SSL, BIO, buffer and socket on the receive and send paths. The buffer > size is set to 16KB. TlsSocket inherits from the Socket class, that > is, it as a type of Socket. The existing Ceph code does not need to > distinguish if the socket instance is a plain TCP socket or TLS > enabled socket. TlsSocket allows us to separate the TLS specific code > from the rest. > > - Introduce SimpleTlsSocket and AsyncTlsSocket classes to implement > the behavior specific to Simple and Async Messengers. These classes > inherit from TlsSocket. Simple Messenger uses blocking sockets while > Async Messenger uses non-blocking sockets. The differences between > them will be implemented within these classes. These classes are > responsible for interacting with the BIO objects and socket layer for > sending and receiving encrypted data. > > - Introduce a class called SslContext to be a wrapper for the SSL > Context object of the OpenSSL library. Each Messenger object can have > up to two SslContext objects, one for the client role and the other > for the server role. Ceph clients allocate only the client SslContext > objects since they only initiate connections. Ceph servers both > initiate and accept connections, and thus, allocate both client and > server SslContext objects. > > - Introduce a class called TLS to represent OpenSSL library and be > responsible for initializing the library. > > New configuration parameters are defined to enable TLS. These are > “tls”, “tls_client_cert_file”, “tls_sever_cert_file”, > “tls_client_key_file”, “tls_server_key_file”,and “tls_ca_cert_file”. > The tls parameter can take one the three values : “none”, “desired”, > “required”. “none” means TLS is not enabled. The default value for > this parameter is “none”. For the older Ceph versions, it is > considered that “tls” is “none”. > > “desired” means if both ends of a TCP connection are configured with > “desired”, or “required”, the session between them must be a TLS > session. Otherwise, the session would be a plain TCP session. The > "desired" value is used temporarily during rolling upgrade from plain > TCP sessions to TLS sessions. If one side is configured with > “required” and the other side is “none”, then Ceph connection attempts > between them will fail. > > The rolling upgrade from plain TCP sessions to TLS sessions can be > done as follows. After a Ceph client or server is upgraded to a TLS > supported version, the “tls” parameter is set to “desired”. For Ceph > clients, this parameter is read from the ceph.config fie, and for the > servers, the parameter is dynamically set without restarting the > server using the injectargs capability. When all the clients and > servers are configured with “tls = desired” then the servers and > clients can be configured with “tls = required”. When the “tls” value > is changed dynamically to “required”, existing connections initiated > or terminated from a server are dropped (and new connections where tls > is required are established). Client config files are updated with > “tls = required”, and clients can be restarted. > > We have considered two options for rolling upgrades. The first one > requires changing the Ceph protocol to advertise TLS configuration > during Ceph handshake. This allows accepting TLS sessions over the > existing Ceph ports that are used for plain TCP connections. In this > case, the connections are upgraded to the TLS sessions by starting TLS > handshake during or immediately after Ceph handshake. > > The second option is to use a separate set of port numbers for TLS. > This does not require changing the existing protocol since Ceph > handshake (i.e., the banner and the rest of) messages will be > exchanged only after TLS sessions are established. The clients > configured with the desired mode attempts to connect servers over the > TLS ports first. If it is not successful, then they attempt over the > plain TCP ports. The clients and servers configured with the required > mode just use the TLS ports. We appreciate the feedback on these > options. Our security team prefers the second option. > > The other parameters, the parameters for certificate and key files, > hold the locations (i.e., paths) of the corresponding files. > > The existing Ceph authentication protocol works as before over TLS > since this design is compatible with it. The design does NOT support > the kernel rbd module (krbd). We are planning to use librbd (via the > user space rbd-nbd client) for the block use cases. The overhead of > rbd-nbd is mostly around 10 to 15% based on the fio runs on Centos 7. > Our initial performance runs show that TLS overhead should also be > about 10 to 15% based on the rados bench throughput tests. More > performance characterization needs to be done to get more reliable > results. > > Thanks, > Kadir > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
