Hello again,
thank all for the very very helpful advices.
Now i have reinstalled my ceph cluster.
Three nodes with ceph version 0.80.7 and for every single disk an osd. The journal will be saved on a ssd.
My ceph.conf
[global]
fsid = bceade34-3c54-4a35-a759-7af631a19df7
mon_initial_members = ceph01
mon_host = 10.0.0.20,10.0.0.21,10.0.0.22
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
public_network = 10.0.0.0/24
cluster_network = 10.0.1.0/24
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 4096
osd_pool_default_pgp_num = 4096
filestore_max_sync_interval = 30
ceph osd tree
-1 6.76 root default
-2 2.44 host ceph01
0 0.55 osd.0 up 1
3 0.27 osd.3 up 1
4 0.27 osd.4 up 1
5 0.27 osd.5 up 1
6 0.27 osd.6 up 1
7 0.27 osd.7 up 1
1 0.27 osd.1 up 1
2 0.27 osd.2 up 1
-3 2.16 host ceph02
9 0.27 osd.9 up 1
11 0.27 osd.11 up 1
12 0.27 osd.12 up 1
13 0.27 osd.13 up 1
14 0.27 osd.14 up 1
15 0.27 osd.15 up 1
8 0.27 osd.8 up 1
10 0.27 osd.10 up 1
-4 2.16 host ceph03
17 0.27 osd.17 up 1
18 0.27 osd.18 up 1
19 0.27 osd.19 up 1
20 0.27 osd.20 up 1
21 0.27 osd.21 up 1
22 0.27 osd.22 up 1
23 0.27 osd.23 up 1
16 0.27 osd.16 up 1
rados bench -p kvm 50 write --no-cleanup
Total time run: 50.494855
Total writes made: 1180
Write size: 4194304
Bandwidth (MB/sec): 93.475
Stddev Bandwidth: 16.3955
Max bandwidth (MB/sec): 112
Min bandwidth (MB/sec): 0
Average Latency: 0.684571
Stddev Latency: 0.216088
Max latency: 1.86831
Min latency: 0.234673
rados bench -p kvm 50 seq
Total time run: 15.009855
Total reads made: 1180
Read size: 4194304
Bandwidth (MB/sec): 314.460
Average Latency: 0.20296
Max latency: 1.06341
Min latency: 0.02983
I am really happy, these values above are enough for my little amount of vms. Inside the vms I get now for write 80mb/s and read 130mb/s, with write-cache enabled.
But there is one little problem.
Are there some tuning parameters for small files?
For 4kb to 50kb files the cluster is very slow.
thank you
best regards
-----Original message-----
From: Lindsay Mathieson <lindsay.mathieson@xxxxxxxxx>
Sent: Friday 9th January 2015 0:59
To: ceph-users@xxxxxxxxxxxxxx
Cc: Patrik Plank <patrik@xxxxxxxx>
Subject: Re: slow read-performance inside the vm
On Thu, 8 Jan 2015 05:36:43 PM Patrik Plank wrote:
Hi Patrick, just a beginner myself, but have been through a similar process
recently :)
> With these values above, I get a write performance of 90Mb/s and read
> performance of 29Mb/s, inside the VM. (Windows 2008/R2 with virtio driver
> and writeback-cache enabled) Are these values normal with my configuration
> and hardware? ->
They do seem *very* odd. Your write performance is pretty good, your read
performance is abysmal - with a similar setup, with 3 OSD's slower than yours
I was getting 200 MB/s reads.
Maybe your network setup is dodgy? Jumbo frames can be tricky. Have you run
iperf between the nodes?
What are you using for benchmark testing on the windows guest?
Also, probably more useful to turn writeback caching off for benchmarking, the
cache will totally obscure the real performance.
How is the VM mounted? rbd driver?
> The read-performance seems slow. Would the
> read-performance better if I run for every single disk a osd?
I think so - in general the more OSD's the better. Also having 8 HD's in RAID0
is a recipe for disaster, you'll lost the entire OSD is one of those disks
fails.
I'd be creating an OSD for each HD (8 per node), with a 5-10GB SSD partition
per OSD for journal. Tedious, but should make a big difference to reads and
writes.
Might be worth while trying
[global]
filestore max sync interval = 30
as well.
--
Lindsay
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
