I agree with Lars's concerns: the main problems with the current dm-crypt approach are that there isn't any key management integration yet and the root volume and swap aren't encrypted. Those are easy to solve (and I'm hoping we'll be able to address them in time for Jewel). On the other hand, implementing encryption within RADOS will be complex, and I don't see what the benefits are over whole-disk encryption. Can someone summarize what per-pool encryption keys and the ability to rotate keys gives us? If the threat is an attacker who is on the storage network and has compromised an OSD the game is pretty much up... At a high level, I think almost anything beyond at-rest encryption (that is aimed at throwing out disks or physically walking a server out of the data center) turns into a key management and threat mitigation design nightmare (with few, if any, compelling solutions) until you give up and have clients encrypt their data and don't trust the cluster with the keys at all... sage On Tue, 15 Dec 2015, Lars Marowsky-Bree wrote: > On 2015-12-14T14:17:08, Radoslaw Zarzynski <rzarzynski@xxxxxxxxxxxx> wrote: > > Hi all, > > great to see this revived. > > However, I have come to see some concerns with handling the encryption > within Ceph itself. > > The key part to any such approach is formulating the threat scenario. > For the use cases we have seen, the data-at-rest encryption matters so > they can confidently throw away disks without leaking data. It's not > meant as a defense against an online attacker. There usually is no > problem with "a few" disks being privileged, or one or two nodes that > need an admin intervention for booting (to enter some master encryption > key somehow, somewhere). > > However, that requires *all* data on the OSDs to be encrypted. > > Crucially, that includes not just the file system meta data (so not just > the data), but also the root and especially the swap partition. Those > potentially include swapped out data, coredumps, logs, etc. > > (As an optional feature, it'd be cool if an OSD could be moved to a > different chassis and continue operating there, to speed up recovery. > Another optional feature would be to eventually be able, for those > customers that trust them ;-), supply the key to the on-disk encryption > (OPAL et al).) > > The proposal that Joshua posted a while ago essentially remained based > on dm-crypt, but put in simple hooks to retrieve the keys from some > "secured" server via sftp/ftps instead of loading them from the root fs. > Similar to deo, that ties the key to being on the network and knowing > the OSD UUID. > > This would then also be somewhat easily extensible to utilize the same > key management server via initrd/dracut. > > Yes, this means that each OSD disk is separately encrypted, but given > modern CPUs, this is less of a problem. It does have the benefit of > being completely transparent to Ceph, and actually covering the whole > node. > > Of course, one of the key issues is always the key server. > Putting/retrieving/deleting keys is reasonably simple, but the question > of how to ensure HA for it is a bit tricky. But doable; people have been > building HA ftp/http servers for a while ;-) Also, a single key server > setup could theoretically serve multiple Ceph clusters. > > It's not yet perfect, but I think the approach is superior to being > implemented in Ceph natively. If there's any encryption that should be > implemented in Ceph, I believe it'd be the on-the-wire encryption to > protect against evasedroppers. > > Other scenarios would require client-side encryption. > > > Current data at rest encryption is achieved through dm-crypt placed > > under OSD?s filestore. This solution is a generic one and cannot > > leverage Ceph-specific characteristics. The best example is that > > encryption is done multiple times - one time for each replica. Another > > issue is lack of granularity - either OSD encrypts nothing, or OSD > > encrypts everything (with dm-crypt on). > > True. But for the threat scenario, a holistic approach to encryption > seems actually required. > > > Cryptographic keys are stored on filesystem of storage node that hosts > > OSDs. Changing them require redeploying the OSDs. > > This is solvable by storing the key on an external key server. > > Changing the key is only necessary if the key has been exposed. And with > dm-crypt, that's still possible - it's not the actual encryption key > that's stored, but the secret that is needed to unlock it, and that can > be re-encrypted quite fast. (In theory; it's not implemented yet for > the Ceph OSDs.) > > > > Data incoming from Ceph clients would be encrypted by primary OSD. It > > would replicate ciphertext to non-primary members of an acting set. > > This still exposes data in coredumps or on swap on the primary OSD, and > metadata on the secondaries. > > > Regards, > Lars > > -- > Architect Storage/HA > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) > "Experience is the name everyone gives to their mistakes." -- Oscar Wilde > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > >
