On Wed, 2020-08-05 at 11:22 +0000, Joao Eduardo Luis wrote: > On 20/08/04 01:16PM, Jeff Layton wrote: > > On Tue, 2020-08-04 at 09:04 -0400, Jason Dillaman wrote: > > > On Mon, Aug 3, 2020 at 4:48 PM Joao Eduardo Luis <joao@xxxxxxx> wrote: > > > > Even though we currently have at-rest encryption, ensuring data security on the > > > > physical device, this is currently on an OSD-basis, and it is too coarse-grained > > > > to allow different entities/clients/tenants to have their data encrypted with > > > > different keys. > > > > Is it really valuable to allow different entities to use different keys, when the OSDs have access to all of them? If the OSD's master store of keys is compromised then all of the tenant data is compromised. This wouldn't necessarily be the case with a scheme where the encryption is done on the clients. > > > > The intent here is to allow different tenants to have their data encrypted at > > > > rest, independently, and without necessarily relying on full osd encryption. > > > > This way one could have anywhere between a handful to dozens or hundreds of > > > > tenants with their data encrypted on disk, while not having to maintain full > > > > at-rest encryption should the administrator consider it too cumbersome or > > > > unnecessary. > > > > > > I would be interested to hear the tenant use-case where they trust the > > > backing storage system (Ceph) with all things encryption and don't > > > have any effective control over the keys / ciphers / rotation policies > > > / etc. If you have a vulnerability that exposes the current OSD > > > dm-crypt keys, I would think it would be possible to get the > > > per-namespace keys though a similar vector if they are stored > > > effectively side-by-side? > > > > > > > Agreed. If I were a cloud tenant, I'd not be thrilled at a scheme that > > required me to trust the OSD to encrypt my cleartext data for me. > > > > You might want to take a step back and consider that there are really > > two problems you have to deal with: > > > > 1/ encryption: where and how do I perform the encryption of the data? > > > > I think doing this as close to the edges as possible would be best, as > > it means fewer trusted parties. Conceptually, that makes this part > > fairly straightforward. Just run the crypto over the appropriate buffers > > before you send and just after you receive. You will need to do things > > like ensure that someone with a bad key can't corrupt data in an > > encrypted namespace. > > That is a very fair point, and I don't disagree. I do think that having the > client handling their own encryption, and being in control of their secrets, > is the safest approach. > > However, the use cases brought to us have been somewhat like what we have with > at-rest encryption with dmcrypt: the provider controls the keys, and retiring > a tenant is simply a matter of destroying their key. > > Mind you, this is something we could achieve with pools, if the granularity > wasn't so coarse IMO. > I guess I'm not 100% clear on the use-case for this. Are there really folks just interested in knowing that the OSD will store the data encrypted and don't really care that it may deal with unencrypted data in its memory? Is this being driven by some sort of regulatory requirement? > > > 2/ key management: how do I get keys to feed into the crypto engine? > > > > This is the hard part, IMO. While their scheme isn't perfect, you may > > want to look closely at how Linux fscrypt works. Basically, each fs has > > a master key and then you derive keys for the individual inodes from > > that. Every inode has a nonce generated, that's used to ensure that > > (e.g.) two identical files don't have identical encrypted contents. > > Definitely agree. In this whole proposal, it's the encryption portion (a small > detail, I know) that scares me the most, and is something that needs proper > ironing before this could even be phantomed to be implemented. > > > > The master keys are stored encrypted themselves, and the filesystem has > > a number of ways that you can set up to unlock them -- passwords, hard > > tokens, etc. Note that this part requires special tools to set up the > > keys. > > > > You'll probably want to aim for some sort of similar hierarchy of keys > > here, I think. You may need a "special" per-object xattr or something to > > store nonces, and you'll need to think about where they get generated. > > > > One question : are you planning to encrypt object names too? Ideally, > > you want to allow any client-generated data to be encrypted. In a > > filesystem, that's basically filenames and contents. An object store is > > probably pretty similar in that regard. > > That would be the plan. The intent would be to encrypt pretty much anything > that is encryptable, data and metadata. From the object store, the only thing > the client needs to know is the object the data is stored in. AFAIU, this is > currently achieved by hashing the namespace and the object name. From the > storage backend's perspective, accessing an object in an encrypted namespace > means encrypting or decrypting whatever the client wants. > > I'm not entirely sure how feasible it will be to encrypt all on-disk metadata, > but after speaking with Igor about this I came out under the impression that > it would be feasible to do so. > > > > > > While there are very good arguments for ensuring this encryption is performed > > > > client-side, such that each client actively controls their own secrets, a > > > > server-side approach has several other benefits that may outweigh a client-side > > > > approach. > > > > > > > > On the one hand, > > > > > > > > * encrypting server side means encrypting N times, depending on replication > > > > size and scheme; > > > > * the secrets keyring will be centralized, likely in the monitor, much like > > > > what we do for dmcrypt; even though encrypted. > > > > * on-the-wire data will still need to rely on msgr2 encryption; even though > > > > one could argue that this will likely happen regardless of whether a client- > > > > or server-side approach is being used. > > > > > > > > But on the other, > > > > > > > > 1. encryption becomes transparent for the client, avoiding the effort of > > > > implementing such schemes in client libraries and kernel drivers; > > > > > > Just an FYI: krbd supports client-side via dm-crypt, kernel CephFS is > > > actively looking to incorporate fscrypt, librbd can utilize > > > QEMU-layered LUKS for many use-cases and work is in-progress on > > > built-in librbd client-side encryption. RGW has had client-side > > > encryption for a while. > > > > > > > 2. tighter control over the unit of data being encrypted, reducing the load of > > > > encrypting a whole object versus a disk block in bluestore. > > > > > > RBD client-side encryption doesn't rely on the underlying object size > > > (512 bytes for dm-crypt I think and looking at 4KiB blocks for the > > > librbd built-in encryption). I can't speak for CephFS+fscrypt, but I > > > suspect it wouldn't require re-encrypting the full file or backing > > > object (probably 4KiB page). > > > > > > > For fscrypt, the client basically just encrypts/decrypts file data on a > > per-block basis. For ceph, that means we'll just operate on it a page at > > a time. > > > > fwiw, the data path looks reasonably simple to deal with. The hard part > > is dealing with encrypted filenames. > > I'm guessing these filenames are being kept somewhere in an object in the > object store? Would this also pose issues with xattrs and other metadata? > No, with fscrypt, the filenames are encrypted in place, such that the filenames themselves are encrypted and you need a key to see their unencrypted forms in a readdir(). What is sort of cool is that you can still access the tree without a key at all, but all you see is encrypted filenames, and you can't do anything with the files (other than unlink(), assuming you have the correct permissions). The main downside of fscrypt is that it can't encrypt metadata at all. Stuff like the mtime or file size is under the purview of the filesystem (the Ceph MDS in our case). It'd also be dangerous to (e.g.) show a scrambled mode or ownership for the file, as you might not be able to predict how applications might behave. > Solely as an exercise, should the client encrypt the data and write it to the > cluster, and write the metadata to an encrypted namespace, I believe the > object itself could simply be encrypted as a whole, transparently for the > client. > Encrypting metadata is going to pretty much be impossible from a solely client-side solution. The client doesn't have much control over stuff like mtime. That's all managed on the server side. > Granted, at this point I'm not entirely sure how hard it would be to, > or if it's even feasible to, encrypt the internal metadata used by bluestore, > but from my conversation with Igor it seemed that encrypting the object's > metadata would be feasible. Yeah, if you have the OSD doing the encryption for you, then it could (theoretically) store encrypted metadata too. One thing you should consider as well: What will you do if someone has a bunch of encrypted data and the key is (somehow) lost? You'll need some way to be able to blow away old objects that can't be accessed anymore. -- Jeff Layton <jlayton@xxxxxxxxxx> _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
