Re: RBD with SSD journals and SAS OSDs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



hi nick, I earlier did cpupower frequency-set --cpu-governor
performance on all my hosts, which bumped all CPUs up to almost max
speed or more.

It didn't really help much, and I still experience 5-10ms latency in
my fio benchmarks in VMs with this job description.

Is there anything else I can do to force the SSDs to be used more? I
know DIRECT SYNCED WRITE may not be the most common application case,
however I need help to improve a worst case. Benchmarking these ssd
locally with fio and direct sync write, can do 40-50k IOPS.  I'm not
sure exactly what, but something is holding back the max performance.
I know the journals are sparely used from collectd graphs. appreciate
any advice. thx will

>> [global]
>> bs=4k
>> rw=write
>> sync=1
>> direct=1
>> iodepth=1
>> filename=/dev/vdb1
>> runtime=30
>> stonewall=1
>> group_reporting


grep "cpu MHz" /proc/cpuinfo
cpu MHz         : 2945.250
cpu MHz         : 2617.500
cpu MHz         : 3065.062
cpu MHz         : 2574.281
cpu MHz         : 2739.468
cpu MHz         : 2857.593
cpu MHz         : 2602.125
cpu MHz         : 2581.687
cpu MHz         : 2958.656
cpu MHz         : 2793.093
cpu MHz         : 2682.750
cpu MHz         : 2699.718
cpu MHz         : 2620.125
cpu MHz         : 2926.875
cpu MHz         : 2740.031
cpu MHz         : 2559.656
cpu MHz         : 2758.875
cpu MHz         : 2656.593
cpu MHz         : 1476.187
cpu MHz         : 2545.125
cpu MHz         : 2792.718
cpu MHz         : 2630.156
cpu MHz         : 3090.750
cpu MHz         : 2951.906
cpu MHz         : 2845.875
cpu MHz         : 2553.281
cpu MHz         : 2602.125
cpu MHz         : 2600.906
cpu MHz         : 2737.031
cpu MHz         : 2552.156
cpu MHz         : 2624.625
cpu MHz         : 2614.125




On Mon, Oct 17, 2016 at 5:17 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:
>> -----Original Message-----
>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of William Josefsson
>> Sent: 17 October 2016 09:31
>> To: Christian Balzer <chibi@xxxxxxx>
>> Cc: ceph-users@xxxxxxxxxxxxxx
>> Subject: Re:  RBD with SSD journals and SAS OSDs
>>
>> Thx Christian for helping troubleshooting the latency issues. I have attached my fio job template below.
>>
>> I thought to eliminate the factor that the VM is the bottleneck, I've created a 128GB 32 cCPU flavor. Here's the latest fio
> benchmark.
>> http://pastebin.ca/raw/3729693   I'm trying to benchmark the clusters
>> performance for SYNCED WRITEs and how well suited it would be for disk intensive workloads or DBs
>>
>>
>> > The size (45GB) of these journals is only going to be used by a little
>> > fraction, unlikely to be more than 1GB in normal operations and with
>> > default filestore/journal parameters.
>>
>> To consume more of the SSDs in the hope to achieve lower latency, can you pls advice what parameters I should be looking at? I
> have
>> already tried to what's mentioned in RaySun's ceph blog, which eventually lowered my overall sync write IOPs performance by 1-2k.
>
> You biggest gains will probably be around forcing the CPU's to max frequency and forcing c-state to 1.
>
> intel_idle.max_cstate=0 on kernel parameters
> and
> echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct ( I think this is the same as performance governor)
>
> Use something like powertop to check that all cores are running at max freq and are staying in cstate1
>
> I have managed to get the latency on my cluster down to about 600us, but with your hardware I don't suspect you would be able to get
> it below ~1-1.5ms best case.
>
>>
>> # These are from RaySun's  write up, and worsen my total IOPs.
>> # http://xiaoquqi.github.io/blog/2015/06/28/ceph-performance-optimization-summary/
>>
>> filestore xattr use omap = true
>> filestore min sync interval = 10
>> filestore max sync interval = 15
>> filestore queue max ops = 25000
>> filestore queue max bytes = 10485760
>> filestore queue committing max ops = 5000 filestore queue committing max bytes = 10485760000 journal max write bytes =
>> 1073714824 journal max write entries = 10000 journal queue max ops = 50000 journal queue max bytes = 10485760000
>>
>> My Journals are Intel s3610 200GB, split in 4-5 partitions each. When I did FIO on the disks locally with direct=1 and sync=1 the
> WRITE
>> performance was 50k iops for 7 threads.
>>
>> My hardware specs:
>>
>> - 3 Controllers, The mons run here
>> Dell PE R630, 64GB, Intel SSD s3610
>> - 9 Storage nodes
>> Dell 730xd, 2x2630v4 2.2Ghz, 512GB, Journal: 5x200GB Intel 3610 SSD,
>> OSD: 18x1.8TB Hitachi 10krpm SAS
>>
>> RAID Controller is PERC 730
>>
>> All servers have 2x10GbE bonds, Intel ixgbe X540 copper connecting to Arista 7050X 10Gbit Switches with VARP, and LACP interfaces.
> I
>> have from my VM pinged all hosts and the RTT is 0.3ms on the LAN. I did iperf, and I can do 10Gbps from the VM to the storage
> nodes.
>>
>> I've already been tuning, CPU scaling governor to 'performance' on all hosts for all cores. My CEPH release is latest hammer on
>> CentOS7.
>>
>> The best write currently happens at 62 threads it seems, the IOPS is 8.3k for the direct synced writes. The latency and stddev are
> still
>> concerning.. :(
>>
>> simple-write-62: (groupid=14, jobs=62): err= 0: pid=2748: Mon Oct 17
>> 15:20:05 2016
>>   write: io=978.64MB, bw=33397KB/s, iops=8349, runt= 30006msec
>>     clat (msec): min=3, max=20, avg= 7.42, stdev= 2.50
>>      lat (msec): min=3, max=20, avg= 7.42, stdev= 2.50
>>     clat percentiles (usec):
>>      |  1.00th=[ 3888],  5.00th=[ 4256], 10.00th=[ 4448], 20.00th=[ 4768],
>>      | 30.00th=[ 5088], 40.00th=[ 5984], 50.00th=[ 7904], 60.00th=[ 8384],
>>      | 70.00th=[ 8768], 80.00th=[ 9408], 90.00th=[10432], 95.00th=[11584],
>>      | 99.00th=[13760], 99.50th=[14784], 99.90th=[16320], 99.95th=[16512],
>>      | 99.99th=[17792]
>>     bw (KB  /s): min=  315, max=  761, per=1.61%, avg=537.06, stdev=77.13
>>     lat (msec) : 4=1.99%, 10=84.54%, 20=13.47%, 50=0.01%
>>   cpu          : usr=0.05%, sys=0.35%, ctx=509542, majf=0, minf=1902
>>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>>      issued    : total=r=0/w=250527/d=0, short=r=0/w=0/d=0
>>
>>
>> From the above we can tell that the latency for clients doing synced writes, is somewhere 5-10ms which seems very high, especially
>> with quite high performing hardware, network, and SSD journals. I'm not sure whether it may be the syncing from Journal to OSD
> that
>> causes these fluctuations or high latencies.
>>
>> Any help or advice would be much appreciates. thx will
>>
>>
>> [global]
>> bs=4k
>> rw=write
>> sync=1
>> direct=1
>> iodepth=1
>> filename=${FILE}
>> runtime=30
>> stonewall=1
>> group_reporting
>>
>> [simple-write-6]
>> numjobs=6
>> [simple-write-10]
>> numjobs=10
>> [simple-write-14]
>> numjobs=14
>> [simple-write-18]
>> numjobs=18
>> [simple-write-22]
>> numjobs=22
>> [simple-write-26]
>> numjobs=26
>> [simple-write-30]
>> numjobs=30
>> [simple-write-34]
>> numjobs=34
>> [simple-write-38]
>> numjobs=38
>> [simple-write-42]
>> numjobs=42
>> [simple-write-46]
>> numjobs=46
>> [simple-write-50]
>> numjobs=50
>> [simple-write-54]
>> numjobs=54
>> [simple-write-58]
>> numjobs=58
>> [simple-write-62]
>> numjobs=62
>> [simple-write-66]
>> numjobs=66
>> [simple-write-70]
>> numjobs=70
>>
>> On Mon, Oct 17, 2016 at 10:47 AM, Christian Balzer <chibi@xxxxxxx> wrote:
>> >
>> > Hello,
>> >
>> >
>> > On Sun, 16 Oct 2016 19:07:17 +0800 William Josefsson wrote:
>> >
>> >> Ok thanks for sharing. yes my journals are Intel S3610 200GB, which I
>> >> partition in 4 partitions each ~45GB. When I ceph-deploy I declare
>> >> these as the journals of the OSDs.
>> >>
>> > The size (45GB) of these journals is only going to be used by a little
>> > fraction, unlikely to be more than 1GB in normal operations and with
>> > default filestore/journal parameters.
>> >
>> > Because those defaults start flushing things (from RAM, the journal
>> > never gets read unless there is a crash) to the filestore (OSD HDD)
>> > pretty much immediately.
>> >
>> > Again, use google to search the ML archives.
>> >
>> >> I was trying to understand the blocking, and how much my SAS OSDs
>> >> affected my performance. I have a total of 9 hosts, 158 OSDs each
>> >> 1.8TB. The Servers are connected through copper 10Gbit LACP bonds.
>> >> My failure domain is by type RACK. The CRUSH rule set is by rack. 3
>> >> hosts in each rack. Pool size is =3. I'm running hammer on centos7.
>> >>
>> >
>> > Which begs the question to fully detail your HW (CPUs, RAM), network
>> > (topology, what switches, inter-rack/switch links), etc.
>> > The reason for this will become obvious below.
>> >
>> >> I did a simple fio test from one of my xl instances, and got the
>> >> results below. The Latency 7.21ms is worrying, is this expected
>> >> results? Or is there any way I can further tune my cluster to achieve
>> >> better results? thx will
>> >>
>> >
>> >> FIO: sync=1, direct=1, bs=4k
>> >>
>> > Full command line, please.
>> >
>> > Small, sync I/Os are by far the hardest thing for Ceph.
>> >
>> > I can guess what some of the rest was, but it's better to know for sure.
>> > Alternatively, additionally, try this please:
>> >
>> > "fio --size=1G --ioengine=libaio --invalidate=1  --direct=1
>> > --numjobs=1 --rw=randwrite --name=fiojob --blocksize=4K --iodepth=32"
>> >
>> >>
>> >> write-50: (groupid=11, jobs=50): err= 0: pid=3945: Sun Oct 16 08:41:15 2016
>> >>   write: io=832092KB, bw=27721KB/s, iops=6930, runt= 30017msec
>> >>     clat (msec): min=2, max=253, avg= 7.21, stdev= 4.97
>> >>      lat (msec): min=2, max=253, avg= 7.21, stdev= 4.97
>> >
>> > These numbers suggest you did randwrite and aren't all that surprising.
>> > If you were to run atop on your OSD nodes while doing that fio run,
>> > you'll likely see that both CPUs and individual disk (HDDs) get very busy.
>> >
>> > There are several things conspiring against Ceph here, the latency of
>> > it's own code, the network latency of getting all the individual
>> > writes to each replica, the fact that 1000 of these 4K blocks will hit
>> > one typical RBD object (4MB) and thus one PG, make 3 OSDs very busy, etc.
>> >
>> > If you absolutely need low latencies with Ceph, consider dedicated SSD
>> > only pools for special need applications (DB) or a cache tier if it
>> > fits the profile and avtive working set.
>> > Lower Ceph latency in general by having fast CPUs which are have
>> > powersaving (frequency throttling) disabled or set to "performance"
>> > instead of "ondemand".
>> >
>> > Christan
>> >
>> >>     clat percentiles (msec):
>> >>      |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
>> >>      | 30.00th=[    5], 40.00th=[    6], 50.00th=[    7], 60.00th=[    8],
>> >>      | 70.00th=[    9], 80.00th=[   10], 90.00th=[   12], 95.00th=[   14],
>> >>      | 99.00th=[   17], 99.50th=[   19], 99.90th=[   21], 99.95th=[   23],
>> >>      | 99.99th=[  253]
>> >>     bw (KB  /s): min=  341, max=  870, per=2.01%, avg=556.60, stdev=136.98
>> >>     lat (msec) : 4=8.24%, 10=74.10%, 20=17.52%, 50=0.12%, 500=0.02%
>> >>   cpu          : usr=0.04%, sys=0.23%, ctx=425242, majf=0, minf=1570
>> >>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>> >>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> >>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>> >>      issued    : total=r=0/w=208023/d=0, short=r=0/w=0/d=0
>> >>
>> >> On Sun, Oct 16, 2016 at 4:18 PM, Christian Balzer <chibi@xxxxxxx> wrote:
>> >> >
>> >> > Hello,
>> >> >
>> >> > On Sun, 16 Oct 2016 15:03:24 +0800 William Josefsson wrote:
>> >> >
>> >> >> Hi list, while I know that writes in the RADOS backend are sync()
>> >> >> can anyone please explain when the cluster will return on a write
>> >> >> call for RBD from VMs? Will data be considered synced one written
>> >> >> to the journal or all the way to the OSD drive?
>> >> >>
>> >> > This has been answered countless (really) here, the Ceph
>> >> > Architecture documentation should really be more detailed about
>> >> > this, as well as how parallel the data is being sent to the secondary OSDs.
>> >> >
>> >> > It is of course ack'ed to the client once all journals have
>> >> > successfully written the data, otherwise journal SSDs would make a LOT less sense.
>> >> >
>> >> >> Each host in my cluster has 5x Intel S3610, and 18x1.8TB Hitachi 10krpm SAS.
>> >> >>
>> >> > The size of your SSDs (you didn't mention) will determine the
>> >> > speed, for journal purposes the sequential write speed is basically it.
>> >> >
>> >> > A 5:18 ratio implies that some of your SSDs hold more journals than others.
>> >> >
>> >> > You emphatically do NOT want that, because eventually the busier
>> >> > ones will run out of endurance while the other ones still have plenty left.
>> >> >
>> >> > If possible change this to a 5:20 or 6:18 ratio (depending on your
>> >> > SSDs and expected write volume).
>> >> >
>> >> > Christian
>> >> >> I have size=3 for my pool. Will Ceph return once the data is
>> >> >> written to at least 3 designated journals, or will it in fact wait
>> >> >> until the data is written to the OSD drives? thx will
>> >> >> _______________________________________________
>> >> >> ceph-users mailing list
>> >> >> ceph-users@xxxxxxxxxxxxxx
>> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > Christian Balzer        Network/Systems Engineer
>> >> > chibi@xxxxxxx           Global OnLine Japan/Rakuten Communications
>> >> > http://www.gol.com/
>> >>
>> >
>> >
>> > --
>> > Christian Balzer        Network/Systems Engineer
>> > chibi@xxxxxxx           Global OnLine Japan/Rakuten Communications
>> > http://www.gol.com/
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux