Re: Performance doesn't scale well on a full ssd cluster.

Mark Nelson <mark.nelson@xxxxxxxxxxx> · Fri, 17 Oct 2014 07:49:02 -0500

At least historically, high CPU usage and likely context switching and 
lock contention have been the limiting factor during high IOPS workloads 
on the test hardware at Inktank (and now RH).  I ran benchmarks with a 
parametric sweep of ceph parameters a while back on SSDs to see if 
changing any of our tuning parameters had any unexpected benefits. I 
didn't really see any obviously universally beneficial changes to our 
default tunings.  It's possible this has changed though with the work 
that Somnath and others have been doing for Giant though.

Mark

On 10/17/2014 05:13 AM, Mark Wu wrote:
The client doesn't hit any bottleneck.  I also tried to run multiple
clients on different host. There's no change.

2014-10-17 14:36 GMT+08:00 Alexandre DERUMIER <aderumier@xxxxxxxxx
<mailto:aderumier@xxxxxxxxx>>:

    Hi,
    >>Thanks for the detailed information. but I am already using fio with rbd engine. Almost 4 volumes can reach the peak.

    What is your cpu usage of fio-rbd ?
    Myself I'm cpu bound on 8cores with around 40000iops read 4K.

    ----- Mail original -----

    De: "Mark Wu" <wudx05@xxxxxxxxx <mailto:wudx05@xxxxxxxxx>>
    À: "Daniel Schwager" <Daniel.Schwager@xxxxxxxx
    <mailto:Daniel.Schwager@xxxxxxxx>>
    Cc: ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
    Envoyé: Jeudi 16 Octobre 2014 19:19:17
    Objet: Re:  Performance doesn't scale well on a full ssd
    cluster.

    Thanks for the detailed information. but I am already using fio with
    rbd engine. Almost 4 volumes can reach the peak.
    2014 年 10 月 17 日 上午 1:03于 wudx05@xxxxxxxxx
    <mailto:wudx05@xxxxxxxxx> 写道：

    Thanks for the detailed information. but I am already using fio with
    rbd engine. Almost 4 volumes can reach the peak.
    2014 年 10 月 17 日 上午 12:55于 "Daniel Schwager" <
    Daniel.Schwager@xxxxxxxx <mailto:Daniel.Schwager@xxxxxxxx> >写道：

    <blockquote>

    Hi Mark,

    maybe you will check rbd-enabled fio
    http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html

    yum install ceph-devel
    git clone git:// git.kernel.dk/fio.git <http://git.kernel.dk/fio.git>
    cd fio ; ./configure ; make -j5 ; make install

    Setup the number of jobs (==clients) inside fio config to
    numjobs=8
    for simulating multiple clients.

    regards
    Danny

    my test.fio:

    [global]
    #logging
    #write_iops_log=write_iops_log
    #write_bw_log=write_bw_log
    #write_lat_log=write_lat_log
    ioengine=rbd
    clientname=admin
    pool=rbd
    rbdname=myimage
    invalidate=0 # mandatory
    rw=randwrite
    bs=1m
    runtime=120
    iodepth=8
    numjobs=8

    time_based
    #direct=0

    [seq-write]
    stonewall
    rw=write

    #[seq-read]
    #stonewall
    #rw=read

    </blockquote>

    _______________________________________________
    ceph-users mailing list
    ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com