On 11/8/22 21:20, Mark Nelson wrote:
2.
https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/
<https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/>
You tested network encryption impact on performance. It would be nice to
see how OSD encryption (encryption at rest) impacts performance. As far
as I can see there is not much public information available on this.
However there is one thread with this exact question asked [1]. And it
contains an interesting blog post from Cloudlare [2]. I repeated the
tests from [2] and could draw the same conclusions: TL;DR: performance
is increased a lot and less CPU is used. Some fio 4k write, iodepth=1,
performance numbers on a Samsung PM983 3.84 TB drive )Ubuntu 22.04 with
HWE kernel, 5.15.0-52-generic, AMD EPYC 7302P 16-Core Processor, C-state
pinning, CPU performance mode on, Samsung PM 983 firmware: EDA5702Q):
Unencrypted NVMe:
write: IOPS=63.3k, BW=247MiB/s (259MB/s)(62.6GiB/259207msec); 0 zone resets
clat (nsec): min=13190, max=56400, avg=15397.89, stdev=1506.45
lat (nsec): min=13250, max=56940, avg=15462.03, stdev=1507.88
Encrypted (without no_write_workqueue / no_read_workqueue):
write: IOPS=34.8k, BW=136MiB/s (143MB/s)(47.4GiB/357175msec); 0 zone
resets
clat (usec): min=24, max=1221, avg=28.12, stdev= 2.98
lat (usec): min=24, max=1221, avg=28.37, stdev= 2.99
Encrypted (with no_write_workqueue / no_read_workqueue enabled):
write: IOPS=55.7k, BW=218MiB/s (228MB/s)(57.3GiB/269574msec); 0 zone resets
clat (nsec): min=15710, max=87090, avg=17550.99, stdev=875.72
lat (nsec): min=15770, max=87150, avg=17614.82, stdev=876.85
So encryption does have a performance impact, but the added latency
compared to the latency Ceph itself adds to (client) IO seems
negligible. At least, when the work queues are bypassed, otherwise a lot
of CPU seems to be involved (loads of kcryptd threads). And that might
hurt max performance on a system that is CPU bound.
From the talk "DATA SECURITY AND STORAGE HARDENING IN ROOK AND CEPH"
[3] I understood that > 95% of all Red Hat's customers use OSD
encryption. So this might potentially be a big performance gain for all
(RH) users of Ceph that use encryption (and have fast SSD / NVMe drives).
I'm currently busy on enabling OSD encryption on a test cluster. As soon
as that is finished (will probably take a couple of days) I want test
real life impact of the bypass read / write work queue on a Ceph
cluster. Adding support for this in Ceph seems pretty trivial, as all
that would be required is adding the options to the command that opens
the crypt device in ceph-volume [4].
Gr. Stefan
[1]:
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/44GMO5UGOXDZKFSOQMCPPHYTREUEA3ZI/
[2]: https://blog.cloudflare.com/speeding-up-linux-disk-encryption/
[3]:
https://virtual-event-2022.ceph.io/en/community/events/2022/ceph-virtual/
[4]:
https://github.com/ceph/ceph/blob/main/src/ceph-volume/ceph_volume/util/encryption.py#L70
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
