Re: Encryption per user Howto

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 5/26/23 23:09, Alexander E. Patrakov wrote:
Hello Frank,

On Fri, May 26, 2023 at 6:27 PM Frank Schilder <frans@xxxxxx> wrote:

Hi all,

jumping on this thread as we have requests for which per-client fs mount encryption makes a lot of sense:

What kind of security to you want to achieve with encryption keys stored
on the server side?

One of the use cases is if a user requests a share with encryption at rest. Since encryption has an unavoidable performance impact, it is impractical to make 100% of users pay for the requirements that only 1% of users really have. Instead of all-OSD back-end encryption hitting everyone for little reason, encrypting only some user-buckets/fs-shares on the front-end application level will ensure that the data is encrypted at rest.


I would disagree about the unavoidable performance impact of at-rest
encryption of OSDs. Read the CloudFlare blog article which shows how
they make the encryption impact on their (non-Ceph) drives negligible:
https://blog.cloudflare.com/speeding-up-linux-disk-encryption/. The
main part of their improvements (the ability to disable dm-crypt
workqueues) is already in the mainline kernel. There is also a Ceph
pull request that disables dm-crypt workqueues on certain drives:
https://github.com/ceph/ceph/pull/49554

Indeed. With the bypass workqueue option enabled for flash devices the overhead of crypto is really low. Here a partial repost from an email I send earlier

I repeated the tests from Cloudflare and could draw the same conclusions: TL;DR: performance is increased a lot and less CPU is used. Some fio 4k write, iodepth=1, performance numbers on a Samsung PM983 3.84 TB drive )Ubuntu 22.04 with HWE kernel, 5.15.0-52-generic, AMD EPYC 7302P 16-Core Processor, C-state pinning, CPU performance mode on, Samsung PM 983 firmware: EDA5702Q):

Unencrypted NVMe:

write: IOPS=63.3k, BW=247MiB/s (259MB/s)(62.6GiB/259207msec); 0 zone resets
    clat (nsec): min=13190, max=56400, avg=15397.89, stdev=1506.45
     lat (nsec): min=13250, max=56940, avg=15462.03, stdev=1507.88


Encrypted (without no_write_workqueue / no_read_workqueue):

write: IOPS=34.8k, BW=136MiB/s (143MB/s)(47.4GiB/357175msec); 0 zone resets
    clat (usec): min=24, max=1221, avg=28.12, stdev= 2.98
     lat (usec): min=24, max=1221, avg=28.37, stdev= 2.99


Encrypted (with no_write_workqueue / no_read_workqueue enabled):

write: IOPS=55.7k, BW=218MiB/s (228MB/s)(57.3GiB/269574msec); 0 zone resets
    clat (nsec): min=15710, max=87090, avg=17550.99, stdev=875.72
     lat (nsec): min=15770, max=87150, avg=17614.82, stdev=876.85

So encryption does have a performance impact, but the added latency is only a few micro seconds. And these tests are on NVMe drives, not Ceph OSDS. Compared to the latency Ceph itself adds to (client) IO this seems negligible. At least, when the work queues are bypassed, otherwise a lot of CPU seems to be involved (loads of kcryptd threads). And that might hurt max performance on a system (especially if CPU bound).


So, today I did a comparison on a production cluster while draining an OSD with 10 concurrent backfills:

without no_write_workqueue / no_read_workqueue: 32 krcryptd threads each doing on average 5.5% CPU, and dmcrypt write thread doing ~ 9% CPU. So that's almost 2 CPU cores

with no_write_workqueue / no_read_workqueue: dmcrypt / cryptd threads do not even show up in top ...

So if encryption is important, even for a subset of your users, I'm pretty sure you can enable it and won't have a negative impact. It does require reprovisioning of all your OSDs ... which is not a small feat. Although this thread started with "per user" encryption. If your users do not trust your Ceph cluster ... client side encryption (i.e. CephFS fscrypt) with a key _they_ manage is still the only way to go.

Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux