Hi list I have tested suse enterprise storage3 using 2 iscsi gateway attached to vmware. The performance is bad. I have turn off VAAI following the (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665). My cluster 3 ceph nodes :2*E5-2620 64G , mem 2*1Gbps (3*10K SAS, 1*480G SSD) per node, SSD as journal 1 vmware node 2*E5-2620 64G , mem 2*1Gbps # ceph -s cluster 0199f68d-a745-4da3-9670-15f2981e7a15 health HEALTH_OK monmap e1: 3 mons at {node1=192.168.50.91:6789/0,node2=192.168.50.92:6789/0,node3=192.168.50.93:6789/0} election epoch 22, quorum 0,1,2 node1,node2,node3 osdmap e200: 9 osds: 9 up, 9 in flags sortbitwise pgmap v1162: 448 pgs, 1 pools, 14337 MB data, 4935 objects 18339 MB used, 5005 GB / 5023 GB avail 448 active+clean client io 87438 kB/s wr, 0 op/s rd, 213 op/s wr sudo ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 4.90581 root default -2 1.63527 host node1 0 0.54509 osd.0 up 1.00000 1.00000 1 0.54509 osd.1 up 1.00000 1.00000 2 0.54509 osd.2 up 1.00000 1.00000 -3 1.63527 host node2 3 0.54509 osd.3 up 1.00000 1.00000 4 0.54509 osd.4 up 1.00000 1.00000 5 0.54509 osd.5 up 1.00000 1.00000 -4 1.63527 host node3 6 0.54509 osd.6 up 1.00000 1.00000 7 0.54509 osd.7 up 1.00000 1.00000 8 0.54509 osd.8 up 1.00000 1.00000 An linux vm in vmmare， running fio. 4k randwrite result just 64 IOPS lantency is high，dd test just 11MB／s. fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=100G -filename=/dev/sdb -name="EBS 4KB randwrite test" -iodepth=32 -runtime=60 EBS 4KB randwrite test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32 fio-2.0.13 Starting 1 thread Jobs: 1 (f=1): [w] [100.0% done] [0K/131K/0K /s] [0 /32 /0 iops] [eta 00m:00s] EBS 4KB randwrite test: (groupid=0, jobs=1): err= 0: pid=6766: Wed Jun 29 21:28:06 2016 write: io=15696KB, bw=264627 B/s, iops=64 , runt= 60737msec slat (usec): min=10 , max=213 , avg=35.54, stdev=16.41 clat (msec): min=1 , max=31368 , avg=495.01, stdev=1862.52 lat (msec): min=2 , max=31368 , avg=495.04, stdev=1862.52 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 198], 60.00th=[ 204], | 70.00th=[ 208], 80.00th=[ 217], 90.00th=[ 799], 95.00th=[ 1795], | 99.00th=[ 7177], 99.50th=[12649], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw (KB/s) : min= 36, max=11960, per=100.00%, avg=264.77, stdev=1110.81 lat (msec) : 2=0.03%, 4=0.23%, 10=40.93%, 20=0.48%, 50=0.03% lat (msec) : 100=0.08%, 250=39.55%, 500=5.63%, 750=2.91%, 1000=1.35% lat (msec) : 2000=4.03%, >=2000=4.77% cpu : usr=0.02%, sys=0.22%, ctx=2973, majf=0, minf=18446744073709538907 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued : total=r=0/w=3924/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=15696KB, aggrb=258KB/s, minb=258KB/s, maxb=258KB/s, mint=60737msec, maxt=60737msec Disk stats (read/write): sdb: ios=83/3921, merge=0/0, ticks=60/1903085, in_queue=1931694, util=100.00% anyone can give me some suggestion to improve the performance ? Regards MQ |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
