This is very exciting! With the way you've described it slotting in, I don't think there should be any problems getting the code merged upstream. But you will want to be aware of the work on "msgr2". I believe it's principally been Ricardo Dias working on that from a template we've discussed in the past. One of its goals is adding encryption, but that sits inside of the Ceph messaging protocol rather than on top of it and they should be able to coexist happily. Other comments inline. On Tue, Mar 6, 2018 at 6:12 PM, Kadir Ozdemir <kozdemir@xxxxxxxxxxxxxx> wrote: > Hi All, > > > <snip> > > There are two main objectives that shape the design to be described > here. The first one is to change the existing Ceph code and behavior > minimally. The second one is to support Simple and Async Messengers. > The main ideas behind the design are as follows: > > - Introduce a class called Socket to replace the socket descriptor in > the Pipe and AsyncConnection classes. Socket will wrap the socket > operations used currently in both Messengers. The socket descriptor > will be an attribute of Socket. Essentially, the system calls for > socket operations such as send and recv will be replaced by the > corresponding member functions in this class. This is to achieve the > minimal code change objective. Socket implements the plain TCP socket. > > - Introduce a class called TlsSocket to implement TLS specific > behavior common to both Simple and Async Messengers, such as > retrieving SSL Context, initiating TLS handshake, and reading from and > writing to the SSL object. It maintains two sets each of which > contains a lock, buffer, and BIO object; one set for the receive path > and the other for the send path. The locks serialize the access to > SSL, BIO, buffer and socket on the receive and send paths. The buffer > size is set to 16KB. TlsSocket inherits from the Socket class, that > is, it as a type of Socket. The existing Ceph code does not need to > distinguish if the socket instance is a plain TCP socket or TLS > enabled socket. TlsSocket allows us to separate the TLS specific code > from the rest. I'm not familiar with the SSL interface, but its general outline and this 16KB buffer size make me wonder: can we send small packets out easily, or does it require some minimum size? I'm thinking of the per-message protocol acknowledgements and the protocol heartbeat, for instance, which are only a few tens of bytes and need to be delivered at some reasonable latency. > - Introduce SimpleTlsSocket and AsyncTlsSocket classes to implement > the behavior specific to Simple and Async Messengers. These classes > inherit from TlsSocket. Simple Messenger uses blocking sockets while > Async Messenger uses non-blocking sockets. The differences between > them will be implemented within these classes. These classes are > responsible for interacting with the BIO objects and socket layer for > sending and receiving encrypted data. > > - Introduce a class called SslContext to be a wrapper for the SSL > Context object of the OpenSSL library. Each Messenger object can have > up to two SslContext objects, one for the client role and the other > for the server role. Ceph clients allocate only the client SslContext > objects since they only initiate connections. Ceph servers both > initiate and accept connections, and thus, allocate both client and > server SslContext objects. I'm not clear on why a server would want two SslContexts instead of one, but I trust the code will make this all clearer. > > - Introduce a class called TLS to represent OpenSSL library and be > responsible for initializing the library. > > New configuration parameters are defined to enable TLS. These are > “tls”, “tls_client_cert_file”, “tls_sever_cert_file”, > “tls_client_key_file”, “tls_server_key_file”,and “tls_ca_cert_file”. > The tls parameter can take one the three values : “none”, “desired”, > “required”. “none” means TLS is not enabled. The default value for > this parameter is “none”. For the older Ceph versions, it is > considered that “tls” is “none”. > > “desired” means if both ends of a TCP connection are configured with > “desired”, or “required”, the session between them must be a TLS > session. Otherwise, the session would be a plain TCP session. The > "desired" value is used temporarily during rolling upgrade from plain > TCP sessions to TLS sessions. If one side is configured with > “required” and the other side is “none”, then Ceph connection attempts > between them will fail. > > The rolling upgrade from plain TCP sessions to TLS sessions can be > done as follows. After a Ceph client or server is upgraded to a TLS > supported version, the “tls” parameter is set to “desired”. For Ceph > clients, this parameter is read from the ceph.config fie, and for the > servers, the parameter is dynamically set without restarting the > server using the injectargs capability. When all the clients and > servers are configured with “tls = desired” then the servers and > clients can be configured with “tls = required”. When the “tls” value > is changed dynamically to “required”, existing connections initiated > or terminated from a server are dropped (and new connections where tls > is required are established). Client config files are updated with > “tls = required”, and clients can be restarted. Once the servers switch to "required", if the clients already have "desired" set, this should all be transparent to users, right? > We have considered two options for rolling upgrades. The first one > requires changing the Ceph protocol to advertise TLS configuration > during Ceph handshake. This allows accepting TLS sessions over the > existing Ceph ports that are used for plain TCP connections. In this > case, the connections are upgraded to the TLS sessions by starting TLS > handshake during or immediately after Ceph handshake. > > The second option is to use a separate set of port numbers for TLS. > This does not require changing the existing protocol since Ceph > handshake (i.e., the banner and the rest of) messages will be > exchanged only after TLS sessions are established. The clients > configured with the desired mode attempts to connect servers over the > TLS ports first. If it is not successful, then they attempt over the > plain TCP ports. The clients and servers configured with the required > mode just use the TLS ports. We appreciate the feedback on these > options. Our security team prefers the second option. Can you talk a little more about why your security team prefers that? Just because then you have a blanket promise that all data is encrypted? Or is there something specific about startup or envelope data they're worried about? I actually might be inclined to just force a cluster to run all-TLS if they want to set it up, rather than having the multiple messengers of enabling both. It *would* require more client-side configuration than having the TLS be part of the Ceph session setup, though, in addition to preventing access by the kernel client. Hrmmm... -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
