Thanks Christian .
Still
"So yes, your numbers are normal for single client, low depth reads, as many
threads in this ML confirm."
threads in this ML confirm."
we're facing very high latency ( i expect much less latency from ssd cluster ) :
clat percentiles (usec):
| 1.00th=[ 350], 5.00th=[ 390], 10.00th=[ 414], 20.00th=[ 454],
| 30.00th=[ 494], 40.00th=[ 540], 50.00th=[ 612], 60.00th=[ 732],
| 70.00th=[ 1064], 80.00th=[10304], 90.00th=[37632], 95.00th=[38656],
| 99.00th=[40192], 99.50th=[41216], 99.90th=[43264], 99.95th=[43776],
Thanks
Yair Magnezi
Storage & Data Protection TL // Kenshoo
Office +972 7 32862423 // Mobile +972 50 575-2955
__________________________________________
On Tue, Mar 15, 2016 at 2:28 AM, Christian Balzer <chibi@xxxxxxx> wrote:
Hello,
On Mon, 14 Mar 2016 15:51:11 +0200 Yair Magnezi wrote:
> On Fri, Mar 11, 2016 at 2:01 AM, Christian Balzer <chibi@xxxxxxx> wrote:
>
> >
> > Hello,
> >
> > As alway there are many similar threads in here, googling and reading
> > up stuff are good for you.
> >
> > On Thu, 10 Mar 2016 16:55:03 +0200 Yair Magnezi wrote:
> >
> > > Hello Cephers .
> > >
> > > I wonder if anyone has some experience with full ssd cluster .
> > > We're testing ceph ( "firefly" ) with 4 nodes ( supermicro
> > > SYS-F628R3-R72BPT ) * 1TB SSD , total of 12 osds .
> > > Our network is 10 gig .
> > Much more, relevant details, from SW versions (kernel, OS, Ceph) and
> > configuration (replica size of your pool) to precise HW info.
> >
>
> H/W --> 4 nodes supermicro ( SYS-F628R3-R72BPT ) , every node has
> 64 GB mem ,
> MegaRAID SAS 2208 : RAID0 , 4 * 1 TB ssd ( SAMSUNG
> MZ7KM960HAHP-00005 )
>
SM863, they should be fine.
However I've never seen any results of them with sync writes, if you have
the time, something to test.
> Cluster --. 4 nodes , 12 OSD's , replica size = 2 , ubuntu 14.04.1
> LTS ,
>
Otherwise similar to my cache pool against which I tested below,
2 nodes with 4x 800GB Intel DC S3610 each, replica of 2, thus 8 OSDs.
2 E5-2623 (3GHz base speed) per node.
Network is QDR Infiniband, IPoIB.
Debian Jessie and Ceph Hammer, though.
> >
> > In particular your SSDs, exact maker/version/size.
> > Where are your journals?
> >
> > SAMSUNG MZ7KM960HAHP-00005 , 893.752 GB
> Journals on the same drive data ( all SSD as mentioned )
>
Again, should be fine but test these with sync writes.
And of course monitor their wearout over time.
>
> > Also Firefly is EOL, Hammer and even more so the upcoming Jewel have
> > significant improvements with SSDs.
> >
> > > We used the ceph_deploy for installation with all defaults
> > > ( followed ceph documentation for integration with open-stack )
> > > As much as we understand there is no need to enable the rbd cache as
> > > we're running on full ssd.
> > RBD cache as in the client side librbd cache is always very helpful,
> > fast backing storage or not.
> > It can significantly reduce the number of small writes, something Ceph
> > has to do a lot of heavy lifting for.
> >
> > > bench marking the cluster shows very poor performance write but
> > > mostly read ( clients are open-stack but also vmware instances ) .
> >
> > Benchmarking how (exact command line for fio for example) and with what
> > results?
> > You say poor, but that might be "normal" for your situation, we can't
> > really tell w/o hard data.
> >
>
>
>
> fio --name=randread --ioengine=libaio --iodepth=1 --rw=randread
> --bs=4k --direct=1 --size=256M --numjobs=10 --runtime=120
> --group_reporting --directory=/ceph_test2
>
Just to make sure, this is run inside your VM?
Here is the result of a functionally identical fio run inside one of my
> root@open-compute1:~# fio --name=randread --ioengine=libaio
> --iodepth=1 --rw=randread --bs=4k --direct=1 --size=256M --numjobs=10
> --runtime=120 --group_reporting --directory=/ceph_test2
> randread: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=1
> ...
> randread: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=1
> fio-2.1.3
> Starting 10 processes
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> randread: Laying out IO file(s) (1 file(s) / 256MB)
> Jobs: 10 (f=10): [rrrrrrrrrr] [100.0% done] [4616KB/0KB/0KB /s] [1154/0/0
> iops] [eta 00m:00s]
> randread: (groupid=0, jobs=10): err= 0: pid=25393: Mon Mar 14 09:17:24
> 2016 read : io=597360KB, bw=4976.5KB/s, iops=1244, runt=120038msec
> slat (usec): min=4, max=497, avg=22.91, stdev=14.70
> clat (usec): min=154, max=57106, avg=8007.97, stdev=14477.89
> lat (usec): min=276, max=57125, avg=8031.36, stdev=14477.36
> clat percentiles (usec):
> | 1.00th=[ 350], 5.00th=[ 390], 10.00th=[ 414],
> 20.00th=[ 454], | 30.00th=[ 494], 40.00th=[ 540], 50.00th=[ 612],
> 60.00th=[ 732], | 70.00th=[ 1064], 80.00th=[10304], 90.00th=[37632],
> 95.00th=[38656], | 99.00th=[40192], 99.50th=[41216], 99.90th=[43264],
> 99.95th=[43776], | 99.99th=[44800]
> bw (KB /s): min= 314, max= 967, per=10.01%, avg=498.08,
> stdev=83.91 lat (usec) : 250=0.01%, 500=31.64%, 750=29.32%, 1000=8.21%
> lat (msec) : 2=5.22%, 4=3.35%, 10=2.22%, 20=0.46%, 50=19.56%
> lat (msec) : 100=0.01%
> cpu : usr=0.14%, sys=0.41%, ctx=153613, majf=0, minf=78
> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> issued : total=r=149340/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
> READ: io=597360KB, aggrb=4976KB/s, minb=4976KB/s, maxb=4976KB/s,
> mint=120038msec, maxt=120038msec
>
> Disk stats (read/write):
> rbd0: ios=149207/3, merge=0/3, ticks=1194356/0, in_queue=1194452,
> util=100.00%
>
VMs (entirely against the cache pool/nodes):
---
root@tvm-03:~# fio --size=128MB --ioengine=libaio --invalidate=1 --direct=1 --numjobs=1 --rw=randread --name=fiojob --blocksize=4k --iodepth=1
fiojob: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 process
fiojob: (groupid=0, jobs=1): err= 0: pid=18099
read : io=131072KB, bw=4128.1KB/s, iops=1032 , runt= 31745msec
slat (usec): min=19 , max=393 , avg=35.58, stdev=11.76
clat (usec): min=117 , max=7786 , avg=924.96, stdev=422.77
lat (usec): min=146 , max=7835 , avg=961.91, stdev=423.99
clat percentiles (usec):
| 1.00th=[ 149], 5.00th=[ 175], 10.00th=[ 197], 20.00th=[ 772],
| 30.00th=[ 884], 40.00th=[ 956], 50.00th=[ 1012], 60.00th=[ 1064],
| 70.00th=[ 1128], 80.00th=[ 1192], 90.00th=[ 1288], 95.00th=[ 1400],
| 99.00th=[ 1800], 99.50th=[ 2224], 99.90th=[ 3600], 99.95th=[ 3952],
| 99.99th=[ 6880]
bw (KB/s) : min= 3440, max= 8120, per=100.00%, avg=4135.22, stdev=572.73
lat (usec) : 250=17.95%, 500=0.79%, 750=0.60%, 1000=28.67%
lat (msec) : 2=51.34%, 4=0.60%, 10=0.05%
cpu : usr=1.30%, sys=5.61%, ctx=32985, majf=0, minf=23
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=131072KB, aggrb=4128KB/s, minb=4128KB/s, maxb=4128KB/s, mint=31745msec, maxt=31745msec
Disk stats (read/write):
vda: ios=32718/0, merge=0/0, ticks=29312/0, in_queue=29228, util=92.06%
---
So same ballpark, note the much lower clat/lat times.
Note that I use "block/vda/queue/read_ahead_kb = 2048" in sysfs.conf
instead of Ceph client configurations.
But then again that's basically identical, supposedly only helping with
sequential reads and not much of a help with SSDs (and fast CPUs).
So yes, your numbers are normal for single client, low depth reads, as many
threads in this ML confirm.
Cranking up iodepth to 32 gives me up to 7200 IOPS per client (VM), with
the storage nodes still very much bored.
Christian
>
> conf file ( client side ) -->
>
> [global]
> fsid = 609317d9-c8ee-462f-a82f-f5c28c6c561b
> mon_initial_members = open-ceph1,open-ceph2,open-ceph3
> mon_host = 10.63.4.101,10.63.4.102,10.63.4.103
> auth_cluster_required = none
> auth_service_required = none
> auth_client_required = none
> filestore_xattr_use_omap = true
> public_network = 10.63.4.0/23
>
> filestore_flusher = false
>
> [client]
> rbd cache = true
> cache writethrough until flush = true
> rbd_readahead_trigger_requests = 50
> rbd_readahead_max_bytes = 4096
> rbd_readahead_disable_after_bytes = 0
> admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
> log file = /var/log/ceph/
> rbd concurrent management ops = 20
>
>
>
> > "Poor" write performance would indicative of SSDs that are unsuitable
> > for Ceph.
> >
> > > any input is much appreciated ( especially want to know which
> > > parameter is crucial for read performance in full ssd cluster )
> > >
> >
> > read_ahead in your clients can improve things, but I guess your cluster
> > has more fundamental problems than this.
> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-April/028552.html
> >
> >
> > Thanks
>
>
>
> > Christian
> > --
> > Christian Balzer Network/Systems Engineer
> > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications
> > http://www.gol.com/
> >
>
--
Christian Balzer Network/Systems Engineer
chibi@xxxxxxx Global OnLine Japan/Rakuten Communications
http://www.gol.com/
This e-mail, as well as any attached document, may contain material which is confidential and privileged and may include trademark, copyright and other intellectual property rights that are proprietary to Kenshoo Ltd, its subsidiaries or affiliates ("Kenshoo"). This e-mail and its attachments may be read, copied and used only by the addressee for the purpose(s) for which it was disclosed herein. If you have received it in error, please destroy the message and any attachment, and contact us immediately. If you are not the intended recipient, be aware that any review, reliance, disclosure, copying, distribution or use of the contents of this message without Kenshoo's express permission is strictly prohibited.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
