Hi, my experience: ceph + iscsi ( multipath ) + vmware == worst Better you search for another solution. vmware + nfs + vmware might have a much better performance. -------- If you are able to get vmware run with iscsi and ceph, i would be >>very<< intrested in what/how you did that. -- Mit freundlichen Gruessen / Best regards Oliver Dzombic IP-Interactive mailto:info@xxxxxxxxxxxxxxxxx Anschrift: IP Interactive UG ( haftungsbeschraenkt ) Zum Sonnenberg 1-3 63571 Gelnhausen HRB 93402 beim Amtsgericht Hanau Geschäftsführung: Oliver Dzombic Steuer Nr.: 35 236 3622 1 UST ID: DE274086107 Am 01.07.2016 um 07:04 schrieb mq: > Hi list > I have tested suse enterprise storage3 using 2 iscsi gateway attached > to vmware. The performance is bad. I have turn off VAAI following the > (https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) > <https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665%29>. > My cluster > 3 ceph nodes :2*E5-2620 64G , mem 2*1Gbps > (3*10K SAS, 1*480G SSD) per node, SSD as journal > 1 vmware node 2*E5-2620 64G , mem 2*1Gbps > > # ceph -s > cluster 0199f68d-a745-4da3-9670-15f2981e7a15 > health HEALTH_OK > monmap e1: 3 mons at > {node1=192.168.50.91:6789/0,node2=192.168.50.92:6789/0,node3=192.168.50.93:6789/0} > election epoch 22, quorum 0,1,2 node1,node2,node3 > osdmap e200: 9 osds: 9 up, 9 in > flags sortbitwise > pgmap v1162: 448 pgs, 1 pools, 14337 MB data, 4935 objects > 18339 MB used, 5005 GB / 5023 GB avail > 448 active+clean > client io 87438 kB/s wr, 0 op/s rd, 213 op/s wr > > sudo ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -1 4.90581 root default > -2 1.63527 host node1 > 0 0.54509 osd.0 up 1.00000 1.00000 > 1 0.54509 osd.1 up 1.00000 1.00000 > 2 0.54509 osd.2 up 1.00000 1.00000 > -3 1.63527 host node2 > 3 0.54509 osd.3 up 1.00000 1.00000 > 4 0.54509 osd.4 up 1.00000 1.00000 > 5 0.54509 osd.5 up 1.00000 1.00000 > -4 1.63527 host node3 > 6 0.54509 osd.6 up 1.00000 1.00000 > 7 0.54509 osd.7 up 1.00000 1.00000 > 8 0.54509 osd.8 up 1.00000 1.00000 > > > > An linux vm in vmmare， running fio. 4k randwrite result just 64 IOPS > lantency is high，dd test just 11MB／s. > > fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=100G > -filename=/dev/sdb -name="EBS 4KB randwrite test" -iodepth=32 -runtime=60 > EBS 4KB randwrite test: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, > ioengine=libaio, iodepth=32 > fio-2.0.13 > Starting 1 thread > Jobs: 1 (f=1): [w] [100.0% done] [0K/131K/0K /s] [0 /32 /0 iops] [eta > 00m:00s] > EBS 4KB randwrite test: (groupid=0, jobs=1): err= 0: pid=6766: Wed Jun > 29 21:28:06 2016 > write: io=15696KB, bw=264627 B/s, iops=64 , runt= 60737msec > slat (usec): min=10 , max=213 , avg=35.54, stdev=16.41 > clat (msec): min=1 , max=31368 , avg=495.01, stdev=1862.52 > lat (msec): min=2 , max=31368 , avg=495.04, stdev=1862.52 > clat percentiles (msec): > | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 9], > | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 198], 60.00th=[ 204], > | 70.00th=[ 208], 80.00th=[ 217], 90.00th=[ 799], 95.00th=[ 1795], > | 99.00th=[ 7177], 99.50th=[12649], 99.90th=[16712], 99.95th=[16712], > | 99.99th=[16712] > bw (KB/s) : min= 36, max=11960, per=100.00%, avg=264.77, > stdev=1110.81 > lat (msec) : 2=0.03%, 4=0.23%, 10=40.93%, 20=0.48%, 50=0.03% > lat (msec) : 100=0.08%, 250=39.55%, 500=5.63%, 750=2.91%, 1000=1.35% > lat (msec) : 2000=4.03%, >=2000=4.77% > cpu : usr=0.02%, sys=0.22%, ctx=2973, majf=0, > minf=18446744073709538907 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >>=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >>=64=0.0% > issued : total=r=0/w=3924/d=0, short=r=0/w=0/d=0 > > Run status group 0 (all jobs): > WRITE: io=15696KB, aggrb=258KB/s, minb=258KB/s, maxb=258KB/s, > mint=60737msec, maxt=60737msec > > Disk stats (read/write): > sdb: ios=83/3921, merge=0/0, ticks=60/1903085, in_queue=1931694, > util=100.00% > > anyone can give me some suggestion to improve the performance ? > > Regards > > MQ > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |