Stefan, how do you have this implemented? Earlier this year I submitted https://tracker.ceph.com/issues/58569 asking to enable just this. > On Jun 2, 2023, at 10:09, Stefan Kooman <stefan@xxxxxx> wrote: > > On 5/26/23 23:09, Alexander E. Patrakov wrote: >> Hello Frank, >> On Fri, May 26, 2023 at 6:27 PM Frank Schilder <frans@xxxxxx> wrote: >>> >>> Hi all, >>> >>> jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense: >>> >>>> What kind of security to you want to achieve with encryption keys stored >>>> on the server side? >>> >>> One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest. >>> >> I would disagree about the unavoidable performance impact of at-rest >> encryption of OSDs. Read the CloudFlare blog article which shows how >> they make the encryption impact on their (non-Ceph) drives negligible: >> https://blog.cloudflare.com/speeding-up-linux-disk-encryption/. The >> main part of their improvements (the ability to disable dm-crypt >> workqueues) is already in the mainline kernel. There is also a Ceph >> pull request that disables dm-crypt workqueues on certain drives: >> https://github.com/ceph/ceph/pull/49554 > > Indeed. With the bypass workqueue option enabled for flash devices the overhead of crypto is really low. Here a partial repost from an email I send earlier > > I repeated the tests from Cloudflare and could draw the same conclusions: TL;DR: performance is increased a lot and less CPU is used. Some fio 4k write, iodepth=1, performance numbers on a Samsung PM983 3.84 TB drive )Ubuntu 22.04 with HWE kernel, 5.15.0-52-generic, AMD EPYC 7302P 16-Core Processor, C-state pinning, CPU performance mode on, Samsung PM 983 firmware: EDA5702Q): > > Unencrypted NVMe: > > write: IOPS=63.3k, BW=247MiB/s (259MB/s)(62.6GiB/259207msec); 0 zone resets > clat (nsec): min=13190, max=56400, avg=15397.89, stdev=1506.45 > lat (nsec): min=13250, max=56940, avg=15462.03, stdev=1507.88 > > > Encrypted (without no_write_workqueue / no_read_workqueue): > > write: IOPS=34.8k, BW=136MiB/s (143MB/s)(47.4GiB/357175msec); 0 zone resets > clat (usec): min=24, max=1221, avg=28.12, stdev= 2.98 > lat (usec): min=24, max=1221, avg=28.37, stdev= 2.99 > > > Encrypted (with no_write_workqueue / no_read_workqueue enabled): > > write: IOPS=55.7k, BW=218MiB/s (228MB/s)(57.3GiB/269574msec); 0 zone resets > clat (nsec): min=15710, max=87090, avg=17550.99, stdev=875.72 > lat (nsec): min=15770, max=87150, avg=17614.82, stdev=876.85 > > So encryption does have a performance impact, but the added latency is only a few micro seconds. And these tests are on NVMe drives, not Ceph OSDS. Compared to the latency Ceph itself adds to (client) IO this seems negligible. At least, when the work queues are bypassed, otherwise a lot of CPU seems to be involved (loads of kcryptd threads). And that might hurt max performance on a system (especially if CPU bound). > > > So, today I did a comparison on a production cluster while draining an OSD with 10 concurrent backfills: > > without no_write_workqueue / no_read_workqueue: 32 krcryptd threads each doing on average 5.5% CPU, and dmcrypt write thread doing ~ 9% CPU. So that's almost 2 CPU cores > > with no_write_workqueue / no_read_workqueue: dmcrypt / cryptd threads do not even show up in top ... > > So if encryption is important, even for a subset of your users, I'm pretty sure you can enable it and won't have a negative impact. > It does require reprovisioning of all your OSDs ... which is not a small feat. Although this thread started with "per user" encryption. If your users do not trust your Ceph cluster ... client side encryption (i.e. CephFS fscrypt) with a key _they_ manage is still the only way to go. > > Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |