Nick, all,
fantastic, that did it!
I installed kernel 4.5.2 on the client, now the single threaded read
performance using krbd mount is up to about 370 MB/s with standard 256
readahead size, and touching 400 MB/s with larger readahead sizes.
All single threaded.
Multi-threaded krbd read on the same mount also improved, a 10 GBit/s
network connection is easily saturated.
Thank you all so much for the discussion and the hints.
Regards,
Mike
On 4/23/16 9:51 AM, nick@xxxxxxxxxx wrote:
I've just looked through github for the Linux kernel and it looks like
that read ahead fix was introduced in 4.4, so I'm not sure if it's worth
trying a slightly newer kernel?
Sent from Nine
<http://xo4t.mj.am/link/xo4t/x07j8u1663zy/1/QWFy6wfcMj4vvr1tAIpuQA/aHR0cDovL3d3dy45Zm9sZGVycy5jb20v>
*From:* Mike Miller <millermike287@xxxxxxxxx>
*Sent:* 21 Apr 2016 2:20 pm
*To:* ceph-users@xxxxxxxxxxxxxx
*Subject:* Re: Slow read on RBD mount, Hammer 0.94.5
Hi Udo,
thanks, just to make sure, further increased the readahead:
$ sudo blockdev --getra /dev/rbd0
1048576
$ cat /sys/block/rbd0/queue/read_ahead_kb
524288
No difference here. First one is sectors (512 bytes), second one KB.
The second read (after drop cache) is somewhat faster (10%-20%) but not
much.
I also found this info
http://tracker.ceph.com/issues/9192
Maybe Ilya can help us, he knows probably best how this can be improved.
Thanks and cheers,
Mike
On 4/21/16 4:32 PM, Udo Lembke wrote:
Hi Mike,
Am 21.04.2016 um 09:07 schrieb Mike Miller:
Hi Nick and Udo,
thanks, very helpful, I tweaked some of the config parameters along
the line Udo suggests, but still only some 80 MB/s or so.
this mean you have reached factor 3 (this are round about the value I
see with single thread on RBD too). Better than nothing.
Kernel 4.3.4 running on the client machine and comfortable readahead
configured
$ sudo blockdev --getra /dev/rbd0
262144
Still not more than about 80-90 MB/s.
they are two possibilities for read-ahead.
Take a look here (and change with echo)
cat /sys/block/rbd0/queue/read_ahead_kb
Perhaps there are slightly differences?
For writing the parallelization is amazing and I see very impressive
speeds, but why is reading performance so much behind? Why is it not
parallelized the same way writing is? Is this something coming up in
the jewel release? Or is it planned further down the road?
If you read an big file and clear your cache ("echo 3 >
/proc/sys/vm/drop_caches") on the client, is the second read very fast?
I assume yes.
In this case the readed data is in the cache on the osd-nodes... so
tuning must be there (and I'm very interesting in improvements).
Udo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
