I compared the profile dumps while write and read is separately running; writing: ------------------------------------------------ Interval 58 Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 27120 10500 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 100.00 133.51 us 36.00 us 1339.00 us 37619 WRITE Duration: 12 seconds Data Read: 0 bytes Data Written: 3153854464 bytes ------------------------------------------------ read: ------------------------------------------------ Interval 63 Stats: Block Size: 131072b+ No. of Reads: 3529 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.54 87.86 us 68.00 us 127.00 us 42 FSTAT 99.46 193.68 us 89.00 us 2121.00 us 3529 READ Duration: 12 seconds Data Read: 462553088 bytes Data Written: 0 bytes ------------------------------------------------ two server brick avg dumps: ================================ Brick: 192.168.101.133:/dcsdata/d0 ---------------------------------- Cumulative Stats: Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 2 1 1 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 0 1613832 0 No. of Writes: 2282474 1148962 227 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 14 FORGET 0.00 0.00 us 0.00 us 0.00 us 39 RELEASE 0.00 0.00 us 0.00 us 0.00 us 114 RELEASEDIR 0.00 84.50 us 54.00 us 115.00 us 2 OPENDIR 0.00 79.00 us 52.00 us 127.00 us 4 OPEN 0.00 47.00 us 14.00 us 130.00 us 8 FLUSH 0.00 342.00 us 311.00 us 373.00 us 2 CREATE 0.00 104.77 us 26.00 us 281.00 us 13 STATFS 0.01 131.75 us 35.00 us 285.00 us 93 LOOKUP 0.02 7446.00 us 104.00 us 29191.00 us 4 READDIRP 0.07 2784.89 us 49.00 us 49224.00 us 36 GETXATTR 0.20 64.49 us 29.00 us 164.00 us 4506 FSTAT 1.07 399482.25 us 361616.00 us 450370.00 us 4 UNLINK 42.87 167.36 us 56.00 us 44827.00 us 381080 READ 55.76 71.51 us 35.00 us 7032.00 us 1159912 WRITE Duration: 22156 seconds Data Read: 211528187904 bytes Data Written: 300276908032 bytes Interval 71 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR 0.18 54.00 us 54.00 us 54.00 us 1 OPENDIR 1.05 107.33 us 40.00 us 217.00 us 3 STATFS 2.90 126.57 us 81.00 us 256.00 us 7 LOOKUP 95.88 14669.00 us 147.00 us 29191.00 us 2 READDIRP Duration: 581 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: 192.168.101.134:/dcsdata/d0 ---------------------------------- Cumulative Stats: Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 2 3 24 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 22 1563063 0 No. of Writes: 1522412 1525007 184 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 14 FORGET 0.00 0.00 us 0.00 us 0.00 us 39 RELEASE 0.00 0.00 us 0.00 us 0.00 us 114 RELEASEDIR 0.00 116.50 us 111.00 us 122.00 us 2 OPENDIR 0.00 69.25 us 23.00 us 95.00 us 8 FLUSH 0.00 418.00 us 285.00 us 551.00 us 2 CREATE 0.00 239.25 us 101.00 us 396.00 us 4 READDIRP 0.00 93.00 us 39.00 us 249.00 us 13 STATFS 0.01 142.89 us 78.00 us 241.00 us 87 LOOKUP 0.09 48402.25 us 114.00 us 99173.00 us 4 OPEN 0.19 10974.42 us 60.00 us 345979.00 us 36 GETXATTR 0.20 94.33 us 41.00 us 200.00 us 4387 FSTAT 0.85 440436.25 us 381525.00 us 582989.00 us 4 UNLINK 35.80 193.96 us 57.00 us 23312.00 us 380869 READ 62.86 134.89 us 29.00 us 9976.00 us 961593 WRITE Duration: 22155 seconds Data Read: 204875400152 bytes Data Written: 299728837956 bytes ================================ Kane ? 2013-9-18???2:45?Anand Avati <avati at gluster.org> ??? > Can you get the volume profile dumps for both the runs and compare them? > > Avati > > > > On Tue, Sep 17, 2013 at 10:46 PM, kane <stef_9k at 163.com> wrote: > I have already used "kernel oplocks = no" in the smb.conf, next is my original smb.conf file global settings: > [global] > workgroup = MYGROUP > server string = DCS Samba Server > log file = /var/log/samba/log.vfs > max log size = 500000 > aio read size = 262144 > aio write size = 262144 > aio write behind = true > security = user > passdb backend = tdbsam > load printers = yes > cups options = raw > read raw = yes > write raw = yes > max xmit = 262144 > socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144 > # max protocol = SMB2 > kernel oplocks = no > stat cache = no > > thank you > -Kane > ? 2013-9-18???1:38?Anand Avati <avati at redhat.com> ??? > > > On 9/17/13 10:34 PM, kane wrote: > >> Hi Anand, > >> > >> I use 2 gluster server , this is my volume info: > >> Volume Name: soul > >> Type: Distribute > >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6 > >> Status: Started > >> Number of Bricks: 2 > >> Transport-type: tcp > >> Bricks: > >> Brick1: 192.168.101.133:/dcsdata/d0 > >> Brick2: 192.168.101.134:/dcsdata/d0 > >> > >> each brick use a raid 5 logic disk with 8*2TSATA hdd. > >> > >> smb.conf: > >> [gvol] > >> comment = For samba export of volume test > >> vfs objects = glusterfs > >> glusterfs:volfile_server = localhost > >> glusterfs:volume = soul > >> path = / > >> read only = no > >> guest ok = yes > >> > >> this my testparm result: > >> [global] > >> workgroup = MYGROUP > >> server string = DCS Samba Server > >> log file = /var/log/samba/log.vfs > >> max log size = 500000 > >> max xmit = 262144 > >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 > >> SO_SNDBUF=262144 > >> stat cache = No > >> kernel oplocks = No > >> idmap config * : backend = tdb > >> aio read size = 262144 > >> aio write size = 262144 > >> aio write behind = true > >> cups options = raw > >> > >> in client mount the smb share with cifs to dir /mnt/vfs, > >> then use iozone executed in the cifs mount dir "/mnt/vfs": > >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4 > >> File size set to 10485760 KB > >> Record Size 128 KB > >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4 > >> Output is in Kbytes/sec > >> Time Resolution = 0.000001 seconds. > >> Processor cache size set to 1024 Kbytes. > >> Processor cache line size set to 32 bytes. > >> File stride size set to 17 * record size. > >> Throughput test with 4 processes > >> Each process writes a 10485760 Kbyte file in 128 Kbyte records > >> > >> Children see throughput for 4 initial writers = 534315.84 KB/sec > >> Parent sees throughput for 4 initial writers = 519428.83 KB/sec > >> Min throughput per process = 133154.69 KB/sec > >> Max throughput per process = 134341.05 KB/sec > >> Avg throughput per process = 133578.96 KB/sec > >> Min xfer = 10391296.00 KB > >> > >> Children see throughput for 4 rewriters = 536634.88 KB/sec > >> Parent sees throughput for 4 rewriters = 522618.54 KB/sec > >> Min throughput per process = 133408.80 KB/sec > >> Max throughput per process = 134721.36 KB/sec > >> Avg throughput per process = 134158.72 KB/sec > >> Min xfer = 10384384.00 KB > >> > >> Children see throughput for 4 readers = 77403.54 KB/sec > >> Parent sees throughput for 4 readers = 77402.86 KB/sec > >> Min throughput per process = 19349.42 KB/sec > >> Max throughput per process = 19353.42 KB/sec > >> Avg throughput per process = 19350.88 KB/sec > >> Min xfer = 10483712.00 KB > >> > >> Children see throughput for 4 re-readers = 77424.40 KB/sec > >> Parent sees throughput for 4 re-readers = 77423.89 KB/sec > >> Min throughput per process = 19354.75 KB/sec > >> Max throughput per process = 19358.50 KB/sec > >> Avg throughput per process = 19356.10 KB/sec > >> Min xfer = 10483840.00 KB > >> > >> then the use the same command test in the dir mounted with glister fuse: > >> File size set to 10485760 KB > >> Record Size 128 KB > >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4 > >> Output is in Kbytes/sec > >> Time Resolution = 0.000001 seconds. > >> Processor cache size set to 1024 Kbytes. > >> Processor cache line size set to 32 bytes. > >> File stride size set to 17 * record size. > >> Throughput test with 4 processes > >> Each process writes a 10485760 Kbyte file in 128 Kbyte records > >> > >> Children see throughput for 4 initial writers = 887534.72 KB/sec > >> Parent sees throughput for 4 initial writers = 848830.39 KB/sec > >> Min throughput per process = 220140.91 KB/sec > >> Max throughput per process = 223690.45 KB/sec > >> Avg throughput per process = 221883.68 KB/sec > >> Min xfer = 10319360.00 KB > >> > >> Children see throughput for 4 rewriters = 892774.92 KB/sec > >> Parent sees throughput for 4 rewriters = 871186.83 KB/sec > >> Min throughput per process = 222326.44 KB/sec > >> Max throughput per process = 223970.17 KB/sec > >> Avg throughput per process = 223193.73 KB/sec > >> Min xfer = 10431360.00 KB > >> > >> Children see throughput for 4 readers = 605889.12 KB/sec > >> Parent sees throughput for 4 readers = 601767.96 KB/sec > >> Min throughput per process = 143133.14 KB/sec > >> Max throughput per process = 159550.88 KB/sec > >> Avg throughput per process = 151472.28 KB/sec > >> Min xfer = 9406848.00 KB > >> > >> it shows much higher perf. > >> > >> any places i did wrong? > >> > >> > >> thank you > >> -Kane > >> > >> ? 2013-9-18???1:19?Anand Avati <avati at gluster.org > >> <mailto:avati at gluster.org>> ??? > >> > >>> How are you testing this? What tool are you using? > >>> > >>> Avati > >>> > >>> > >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com > >>> <mailto:stef_9k at 163.com>> wrote: > >>> > >>> Hi Vijay > >>> > >>> I used the code in > >>> https://github.com/gluster/glusterfs.git with the lasted commit: > >>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a > >>> Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>> > >>> Date: Tue Sep 17 16:45:03 2013 -0700 > >>> > >>> Revert "cluster/distribute: Rebalance should also verify free > >>> inodes" > >>> > >>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa > >>> > >>> Realized soon after merging, ?. > >>> > >>> which include the patch you mentioned last time improve read perf, > >>> written by Anand. > >>> > >>> but the read perf was still slow: > >>> write: 500MB/s > >>> read: 77MB/s > >>> > >>> while via fuse : > >>> write 800MB/s > >>> read 600MB/s > >>> > >>> any advises? > >>> > >>> > >>> Thank you. > >>> -Kane > >>> > >>> ? 2013-9-13???10:37?kane <stef_9k at 163.com > >>> <mailto:stef_9k at 163.com>> ??? > >>> > >>>> Hi Vijay? > >>>> > >>>> thank you for post this message, i will try it soon > >>>> > >>>> -kane > >>>> > >>>> > >>>> > >>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur at redhat.com > >>> <mailto:vbellur at redhat.com>> ??? > >>>> > >>>>> On 09/13/2013 06:10 PM, kane wrote: > >>>>>> Hi > >>>>>> > >>>>>> We use gluster samba vfs test io,but the read performance via > >>> vfs is > >>>>>> half of write perfomance, > >>>>>> but via fuse the read and write performance is almost the same. > >>>>>> > >>>>>> this is our smb.conf: > >>>>>> [global] > >>>>>> workgroup = MYGROUP > >>>>>> server string = DCS Samba Server > >>>>>> log file = /var/log/samba/log.vfs > >>>>>> max log size = 500000 > >>>>>> # use sendfile = true > >>>>>> aio read size = 262144 > >>>>>> aio write size = 262144 > >>>>>> aio write behind = true > >>>>>> min receivefile size = 262144 > >>>>>> write cache size = 268435456 > >>>>>> security = user > >>>>>> passdb backend = tdbsam > >>>>>> load printers = yes > >>>>>> cups options = raw > >>>>>> read raw = yes > >>>>>> write raw = yes > >>>>>> max xmit = 262144 > >>>>>> socket options = TCP_NODELAY IPTOS_LOWDELAY > >>> SO_RCVBUF=262144 > >>>>>> SO_SNDBUF=262144 > >>>>>> kernel oplocks = no > >>>>>> stat cache = no > >>>>>> > >>>>>> any advises helpful? > >>>>>> > >>>>> > >>>>> This patch has shown improvement in read performance with libgfapi: > >>>>> > >>>>> http://review.gluster.org/#/c/5897/ > >>>>> > >>>>> Would it be possible for you to try this patch and check if it > >>> improves performance in your case? > >>>>> > >>>>> -Vijay > >>>>> > >>>> > >>> > >>> > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users > >>> > >>> > >> > > > > Please add 'kernel oplocks = no' in the [gvol] section and try again. > > > > Avati > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/ab613dc7/attachment.html>