Hi,
I'm testing CEPH with the RBD/QEMU driver through libvirt to store my VM
images on. Installation and configuration all went very well with the
ceph-deploy tool. I have set up authx authentication in libvirt and that
works like a charm too.
However, when coming to performance I have big issues getting expected
results inside the hosted VM. I see high latency and bad write
performance, down to 20MB/s in VM.
My setup:
3xDELL R410,
2xXeon X5650,
48 GB RAM,
2xSATA RAID1 for System,
2x250GB Samsung Evo SSD for OSD's (with XFS on each one)
ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de)
Linux server1 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 13.10
In total:
6 OSD
1 MON
3 MDS
So, question is; is there anyone out there that have experience of
running the RBD/QEMU driver in production, and getting any good
performance inside the VM?
I suspect the main performance issue to be caused by high latency, since
it all feels quite high when running those tests below with bonnie++.
(bonnie++ -s 4096 -r 2048 -u root -d X -m BenchClient)
Inside VPS running on native image in RBD pool:
-- Without any Cache
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 733 96 64919 8 20271 3 3013 97 30770 3
2887 82
Latency 17425us 1093ms 894ms 16789us 19390us
89203us
Version 1.96 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 27951 52 +++++ +++ +++++ +++ 24921 45 +++++ +++
22535 29
Latency 1986us 826us 1065us 216us 41us
611us
--With Writeback Cache(QEMU)
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 872 96 67327 8 22424 3 2516 94 32013 3
2800 82
Latency 16196us 657ms 843ms 37889us 19207us
85407us
Version 1.96 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 27225 51 +++++ +++ +++++ +++ 27325 47 +++++ +++
21645 28
Latency 1986us 852us 874us 252us 34us
595us
--With Writethrough Cache(QEMU)
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 833 95 27469 3 6520 1 2743 93 33003 3
1912 61
Latency 17330us 2388ms 1165ms 48442us 19577us
91228us
Version 1.96 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 16378 31 +++++ +++ 18864 24 18024 33 +++++ +++
14734 19
Latency 2028us 761us 1188us 271us 36us
567us
---With Writeback Cache (CEPH)
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 785 95 67573 8 19906 3 2777 96 32681 3
2764 80
Latency 17410us 729ms 737ms 15103us 22802us
88876us
Version 1.96 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 24286 46 +++++ +++ +++++ +++ 31392 57 +++++ +++
+++++ +++
Latency 1925us 760us 1136us 191us 65us
612us
--- Without cache (CEPH)
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 743 95 53350 6 6568 1 2400 90 28769 2
2024 67
Latency 18056us 1503ms 2408ms 97616us 42963us
89855us
Version 1.96 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 20070 40 +++++ +++ 18488 24 20123 36 +++++ +++
15856 20
Latency 1926us 833us 1386us 207us 64us
591us
--- Without Cache test 2 (CEPH)
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 731 88 47184 6 6461 1 2926 97 27001 2
1915 61
Latency 17084us 2106ms 947ms 5563us 21173us
88365us
Version 1.96 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 13473 27 +++++ +++ 13531 17 15646 28 +++++ +++
17251 21
Latency 1979us 841us 1034us 190us 66us
696us
With Mounted RBD image to /mnt on host system
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 1531 86 106478 7 106441 5 1881 72 4820502
100 8202 132
Latency 7167us 226us 211us 4198us 185us
3115us
Version 1.97 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 26881 58 +++++ +++ 22656 79 21652 76 +++++ +++
14217 37
Latency 1043us 144us 838us 830us 8us
114ms
Directly to SSD drive
Version 1.97 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
BenchClient 4G 1687 98 121456 8 124699 5 2942 99 5465973
99 8527 142
Latency 7323us 221us 214us 3605us 205us
3402us
Version 1.97 ------Sequential Create------ --------Random
Create--------
BenchClient -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 24850 57 +++++ +++ 22672 79 18802 72 +++++ +++
28463 74
Latency 129us 223us 223us 459us 15us
212us
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
