Running 3.10 but I don't think i can change the depth on this older kernel.
I see that config option on my 4.9 test machine. I wonder if that will help a lot! My cluster has wait but it seems entirely limited by the RBD client.. the OSD are not busy and i don't really have any iowait at all.
_____________________________________________
![]()
Tyler Bishop
EST 2007
O: 513-299-7108 x1000
This email is intended only for the recipient(s) above and/or otherwise authorized personnel. The information contained herein and attached is confidential and the property of Beyond Hosting. Any unauthorized copying, forwarding, printing, and/or disclosing any information related to this email is prohibited. If you received this message in error, please contact the sender and destroy all copies of this email and any attachment(s).
On Sat, Sep 8, 2018 at 4:56 AM Ilya Dryomov <idryomov@xxxxxxxxx> wrote:
On Sat, Sep 8, 2018 at 1:52 AM Tyler Bishop
<tyler.bishop@xxxxxxxxxxxxxxxxx> wrote:
>
> I have a fairly large cluster running ceph bluestore with extremely fast SAS ssd for the metadata. Doing FIO benchmarks I am getting 200k-300k random write iops but during sustained workloads of ElasticSearch my clients seem to hit a wall of around 1100 IO/s per RBD device. I've tried 1 RBD and 4 RBD devices and I still only get 1100 IO per device, so 4 devices gets me around 4k.
>
> Is there some sort of setting that limits each RBD devices performance? I've tried playing with nr_requests but that don't seem to change it at all... I'm just looking for another 20-30% performance on random write io... I even thought about doing raid 0 across 4-8 rbd devices just to get the io performance.
What is the I/O profile of that workload? How did you arrive at the
20-30% number?
Which kernel are you running? Increasing nr_requests doesn't actually
increase the queue depth, at least on anything moderately recent. You
need to map with queue_depth=X for that, see [1] for details.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b55841807fb864eccca0167650a65722fd7cd553
Thanks,
Ilya
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
