Hi Shin,
a side note - you disk choice might be inappropriate for Ceph - AFAIK
this is a consumer model which apparently lacks power loss protection.
This causes dramatic performance drop when each write request is
followed by fsync one. And this is the mode RocksDB/BlueFS work in.
Hence expect pretty low performance numbers for BlueStore - especially
when small writes are benchmarked.
There were plenty of discussion on this topic on the web (e.g. at
ceph-users mailing list or at
https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#CAPACITORS.21).
Thanks,
Igor
On 8/19/2021 10:32 AM, 신희원 / 학생 / 컴퓨터공학부 wrote:
Hi Mark.
Thanks for answering.
I tested the performance on Samsung NVMe SSD 960 PRO 512GB, 2-way
E5-2650(2.2GHz, 12-core) CPU , not HDD...
Is it a problem that I check the performance on ceph that is deployed by
'vstart'..?
Please tell me how to deploy ceph to achieve similar results..!
Thank you
Shin
ᐧ
2021년 8월 19일 (목) 오후 3:55, Mark Nelson <mnelson@xxxxxxxxxx>님이 작성:
On 8/18/21 9:22 PM, 신희원 / 학생 / 컴퓨터공학부 wrote:
Hi,
I measured the performance of ceph-osd and crimson-osd with same single
core affinity.
I checked IOPS, Latency by rados-bench write , and crimson-osd has lower
performance than ceph-osd about 3 times. (ceph-osd(BlueStore): 228 IOPS,
crimson-osd(AlienStore): 73 IOPS)
-> " $ rados bench -p rbd 10 write --no-cleanup "
Then, crimson-osd's CPU utilization is almost 100%.
I think this is the reason of performance degradation..
Crimson-osd is known for lower CPU consumption than ceph-osd.
I wonder why crimson-osd has more CPU usage in this experiment.
Please let me know how to fix it,
Shin
Those are pretty low numbers in general. Is that on HDD? You are
correct that crimson right now is only using a single reactor core.
That means that tests done with cyanstore will (almost!) always be
limited to ~100% CPU usage. With alienstore you'll still have bluestore
worker threads, but we are often still limited by work being done in the
reactor thread. Here's the most recent performance data we've got
internally (from a ~july build of crimson-osd vs ceph-osd) on NVMe
drives using fio:
https://docs.google.com/spreadsheets/d/1AXj9h0yDc2ztFWuptqcTrNU2Ui3wMyAn6QUft3CPdcc/edit?usp=sharing
The gist of it is that on the read path, crimson+cyanstore is
significantly more efficient than crimson+alienstore and any classic
setup. We are slower in terms of absolute performance, but that's
expected to be the case until the multi-reactor work is done. The
thinking right now is that we probably have some optimization we can do
alienstore and of course bluestore as well (We have ongoing work
there). On the write path things are a little murkier. Cyanstore for
some reason is more efficient with very small and very large datasets
but not the middle size case. alienstore/bluestore and classic memstore
efficiency seems to drop overall as the dataset size grows. In fact
classic memstore is significantly less efficient on the write path than
bluestore is and this isn't the first dataset to show this.
I guess all of this is a round about way of saying that you are testing
very "wild west" code right now. 73 iops is pretty abysmal, but if this
is on HDD you might be the only person that's ever tried
crimson+alienstore+hdd so far. We might be more sensitive to backend
latency on HDD with crimson+alienstore, but that's just a guess.
Mark
ᐧ
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
