On 04/24/2013 02:23 PM, Mark Nelson wrote:
On 04/24/2013 06:17 AM, Wido den Hollander wrote:
Hi,
I've been working with a Ceph 0.56.4 setup and I've been seeing some RBD
read performance issues with single processes / threads.
The setup is:
- 36 OSDs (2TB WD RE drives)
- 9 hosts (4 per OSD)
- 120GB Intel SSD as a journal per host
- 32GB Ram per host
- Quad Core Xeon CPU (E3-1220 V2 @ 3.10GHz)
- 2Gbit LACP link
The client (3.8.8 kernel) in this case is a single node connected with
20Gbit LACP to the same switches.
To sum it up, with "rados bench" I'm seeing about 918MB/sec read (LACP
doesn't balance well with one client) and 400MB/sec write.
Note: 2 RADOS bench processes with 64 threads each.
While doing those RADOS benches the disks nor the SSDs are really busy,
so it seems that can be tuned a bit further.
The problem is that when using either kernel RBD or librbd the read
speeds are a lot slower then a write in a single process:
dd if=/dev/zero of=/dev/rbd1 bs=4M count=1024: 290MB/sec
dd if=/dev/rbd1 of=/dev/null bs=4M count=1024: 65MB/sec
When running multiple writers I max out at somewhere around 400MB/sec,
the same as RADOS bench was telling me, but the reads go up to 300MB/sec
when running multiple readers.
Running multiple dd instances will still achieve about 60MB/sec per dd,
but it sums up to somewhere around 300MB/sec. (5 readers)
I changed the following settings:
osd op threads = 8
journal aio = true
The AIO journal showed a huge increase in write performance as expected,
but increasing the op threads didn't change that much. Going from 2
(default) to 4 gave me about 5MB/sec and going to 8 added another
3MB/sec.
Since I'm hitting the same RBD image over and over I'd expected these
blocks to be in the cache of that OSDs and have the read speeds reach
near line performance.
The big difference seems to be in the amount of threads. I noticed the
same with RADOS bench. With a smaller number of threads I wouldn't get
to the 918MB/sec and I had to spawn multiple processes to get there.
However, 65MB/sec write per RBD device doesn't seem like a lot.
I also tried with librbd, but that gives a similar read performance as
kernel RBD.
The end-goal is to run with librbd (OpenStack), but for now I just want
to crank up the read performance of a single process.
I found multiple threads regarding the read performance, one showed that
AMD systems where a problem with the hypertransport, but since these are
Intel systems that isn't the case.
Any suggestions? I'm not trying to touch any kernel settings (yet) since
the RADOS bench shows me a pretty high read performance.
Hi Wido,
I did some RBD testing with fio recently. This was 1 client node talking
to 1 server with 24 OSDs over 2 round-robin bonded 10GbE interfaces. No
replication. Multiple rados bench instances from the client node tops
out at like ~1.8GB/s writes and ~1.4GB/s reads. I'm planning on doing a
more complete write up, but for now, here are some of the single volume
fio results. The big thing here is that concurrency, even with a single
IO process, is needed to get good performance. With more clients (even
just VMs on the same node), we can get throughput within about 80% of
the RADOS bench numbers.
In my case I'm using 3x replication, so when writing I have 36/3 the
write performance available.
Issue is that I don't see a 100% util at all. But it's not about
writing, the reading is just to slow.
4MB write performance using libaio:
1 volume, 1 process, and iodepth = 1
ceph 0.58, krbd: 164MB/s
ceph 0.58, qemu/kvm, no cache: 84MB/s
ceph 0.58, qemu/kvm, rbd cache: 240MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache: 244MB/s
I tried with wip-bobtail-rbd-backports-req-order and with the recent
patch for Qemu ( http://patchwork.ozlabs.org/patch/232489/ ) and get
about 90MB/sec write, but again, it's about reads.
1 volume, 1 process, and iodepth = 16
ceph 0.58, krbd: 711MB/s
ceph 0.58, qemu/kvm, no cache: 899MB/s
ceph 0.58, qemu/kvm, rbd cache: 227MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache: 680MB/s
4MB read performance using libaio:
1 volume, 1 process, and iodepth = 1
ceph 0.58, krbd: 108MB/s
ceph 0.58, qemu/kvm, no cache: 85MB/s
ceph 0.58, qemu/kvm, rbd cache: 85MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache: 89MB/s
1 volume, 1 process, and iodepth = 16
ceph 0.58, krbd: 516MB/s
ceph 0.58, qemu/kvm, no cache: 839MB/s
ceph 0.58, qemu/kvm, rbd cache: 823MB/s
ceph wip-rbd-cache-aio, qemu/kvm, rbd cache: 830MB/s
With 4m size and an iodepth of 16 I'm maxing out a 90MB/sec inside a
Qemu VM.
The whole reading seems sluggish. For example "man fio" took about 4
seconds to show up. Even running apt-get update is rather slow.
The VM doesn't feel responsive at all, so trying to figure out where
that comes from.
To get single request performance to scale farther, you'll have to
diagnose if there are places that you can lower latency rather than hide
it with concurrency. That's not an easy task in a distributed system
like Ceph. There are probably opportunities for optimization, but I
suspect it may take more than tweaking the ceph.conf file.
I fully get that the distributed nature has it's drawbacks in serial
performance and that Ceph excels in parallel performance, however, just
60 ~ 80MB/sec seems rather slow. On a pretty idle cluster that should be
better, especially when all the OSDs have everything in their page cache.
Mark
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
