I have already used "kernel oplocks = no" in the smb.conf, next is my original smb.conf file global settings: [global] workgroup = MYGROUP server string = DCS Samba Server log file = /var/log/samba/log.vfs max log size = 500000 aio read size = 262144 aio write size = 262144 aio write behind = true security = user passdb backend = tdbsam load printers = yes cups options = raw read raw = yes write raw = yes max xmit = 262144 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144 # max protocol = SMB2 kernel oplocks = no stat cache = no thank you -Kane ? 2013-9-18???1:38?Anand Avati <avati at redhat.com> ??? > On 9/17/13 10:34 PM, kane wrote: >> Hi Anand, >> >> I use 2 gluster server , this is my volume info: >> Volume Name: soul >> Type: Distribute >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6 >> Status: Started >> Number of Bricks: 2 >> Transport-type: tcp >> Bricks: >> Brick1: 192.168.101.133:/dcsdata/d0 >> Brick2: 192.168.101.134:/dcsdata/d0 >> >> each brick use a raid 5 logic disk with 8*2TSATA hdd. >> >> smb.conf: >> [gvol] >> comment = For samba export of volume test >> vfs objects = glusterfs >> glusterfs:volfile_server = localhost >> glusterfs:volume = soul >> path = / >> read only = no >> guest ok = yes >> >> this my testparm result: >> [global] >> workgroup = MYGROUP >> server string = DCS Samba Server >> log file = /var/log/samba/log.vfs >> max log size = 500000 >> max xmit = 262144 >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 >> SO_SNDBUF=262144 >> stat cache = No >> kernel oplocks = No >> idmap config * : backend = tdb >> aio read size = 262144 >> aio write size = 262144 >> aio write behind = true >> cups options = raw >> >> in client mount the smb share with cifs to dir /mnt/vfs, >> then use iozone executed in the cifs mount dir "/mnt/vfs": >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4 >> File size set to 10485760 KB >> Record Size 128 KB >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4 >> Output is in Kbytes/sec >> Time Resolution = 0.000001 seconds. >> Processor cache size set to 1024 Kbytes. >> Processor cache line size set to 32 bytes. >> File stride size set to 17 * record size. >> Throughput test with 4 processes >> Each process writes a 10485760 Kbyte file in 128 Kbyte records >> >> Children see throughput for 4 initial writers = 534315.84 KB/sec >> Parent sees throughput for 4 initial writers = 519428.83 KB/sec >> Min throughput per process = 133154.69 KB/sec >> Max throughput per process = 134341.05 KB/sec >> Avg throughput per process = 133578.96 KB/sec >> Min xfer = 10391296.00 KB >> >> Children see throughput for 4 rewriters = 536634.88 KB/sec >> Parent sees throughput for 4 rewriters = 522618.54 KB/sec >> Min throughput per process = 133408.80 KB/sec >> Max throughput per process = 134721.36 KB/sec >> Avg throughput per process = 134158.72 KB/sec >> Min xfer = 10384384.00 KB >> >> Children see throughput for 4 readers = 77403.54 KB/sec >> Parent sees throughput for 4 readers = 77402.86 KB/sec >> Min throughput per process = 19349.42 KB/sec >> Max throughput per process = 19353.42 KB/sec >> Avg throughput per process = 19350.88 KB/sec >> Min xfer = 10483712.00 KB >> >> Children see throughput for 4 re-readers = 77424.40 KB/sec >> Parent sees throughput for 4 re-readers = 77423.89 KB/sec >> Min throughput per process = 19354.75 KB/sec >> Max throughput per process = 19358.50 KB/sec >> Avg throughput per process = 19356.10 KB/sec >> Min xfer = 10483840.00 KB >> >> then the use the same command test in the dir mounted with glister fuse: >> File size set to 10485760 KB >> Record Size 128 KB >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4 >> Output is in Kbytes/sec >> Time Resolution = 0.000001 seconds. >> Processor cache size set to 1024 Kbytes. >> Processor cache line size set to 32 bytes. >> File stride size set to 17 * record size. >> Throughput test with 4 processes >> Each process writes a 10485760 Kbyte file in 128 Kbyte records >> >> Children see throughput for 4 initial writers = 887534.72 KB/sec >> Parent sees throughput for 4 initial writers = 848830.39 KB/sec >> Min throughput per process = 220140.91 KB/sec >> Max throughput per process = 223690.45 KB/sec >> Avg throughput per process = 221883.68 KB/sec >> Min xfer = 10319360.00 KB >> >> Children see throughput for 4 rewriters = 892774.92 KB/sec >> Parent sees throughput for 4 rewriters = 871186.83 KB/sec >> Min throughput per process = 222326.44 KB/sec >> Max throughput per process = 223970.17 KB/sec >> Avg throughput per process = 223193.73 KB/sec >> Min xfer = 10431360.00 KB >> >> Children see throughput for 4 readers = 605889.12 KB/sec >> Parent sees throughput for 4 readers = 601767.96 KB/sec >> Min throughput per process = 143133.14 KB/sec >> Max throughput per process = 159550.88 KB/sec >> Avg throughput per process = 151472.28 KB/sec >> Min xfer = 9406848.00 KB >> >> it shows much higher perf. >> >> any places i did wrong? >> >> >> thank you >> -Kane >> >> ? 2013-9-18???1:19?Anand Avati <avati at gluster.org >> <mailto:avati at gluster.org>> ??? >> >>> How are you testing this? What tool are you using? >>> >>> Avati >>> >>> >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com >>> <mailto:stef_9k at 163.com>> wrote: >>> >>> Hi Vijay >>> >>> I used the code in >>> https://github.com/gluster/glusterfs.git with the lasted commit: >>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a >>> Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>> >>> Date: Tue Sep 17 16:45:03 2013 -0700 >>> >>> Revert "cluster/distribute: Rebalance should also verify free >>> inodes" >>> >>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa >>> >>> Realized soon after merging, ?. >>> >>> which include the patch you mentioned last time improve read perf, >>> written by Anand. >>> >>> but the read perf was still slow: >>> write: 500MB/s >>> read: 77MB/s >>> >>> while via fuse : >>> write 800MB/s >>> read 600MB/s >>> >>> any advises? >>> >>> >>> Thank you. >>> -Kane >>> >>> ? 2013-9-13???10:37?kane <stef_9k at 163.com >>> <mailto:stef_9k at 163.com>> ??? >>> >>>> Hi Vijay? >>>> >>>> thank you for post this message, i will try it soon >>>> >>>> -kane >>>> >>>> >>>> >>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur at redhat.com >>> <mailto:vbellur at redhat.com>> ??? >>>> >>>>> On 09/13/2013 06:10 PM, kane wrote: >>>>>> Hi >>>>>> >>>>>> We use gluster samba vfs test io,but the read performance via >>> vfs is >>>>>> half of write perfomance, >>>>>> but via fuse the read and write performance is almost the same. >>>>>> >>>>>> this is our smb.conf: >>>>>> [global] >>>>>> workgroup = MYGROUP >>>>>> server string = DCS Samba Server >>>>>> log file = /var/log/samba/log.vfs >>>>>> max log size = 500000 >>>>>> # use sendfile = true >>>>>> aio read size = 262144 >>>>>> aio write size = 262144 >>>>>> aio write behind = true >>>>>> min receivefile size = 262144 >>>>>> write cache size = 268435456 >>>>>> security = user >>>>>> passdb backend = tdbsam >>>>>> load printers = yes >>>>>> cups options = raw >>>>>> read raw = yes >>>>>> write raw = yes >>>>>> max xmit = 262144 >>>>>> socket options = TCP_NODELAY IPTOS_LOWDELAY >>> SO_RCVBUF=262144 >>>>>> SO_SNDBUF=262144 >>>>>> kernel oplocks = no >>>>>> stat cache = no >>>>>> >>>>>> any advises helpful? >>>>>> >>>>> >>>>> This patch has shown improvement in read performance with libgfapi: >>>>> >>>>> http://review.gluster.org/#/c/5897/ >>>>> >>>>> Would it be possible for you to try this patch and check if it >>> improves performance in your case? >>>>> >>>>> -Vijay >>>>> >>>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users >>> >>> >> > > Please add 'kernel oplocks = no' in the [gvol] section and try again. > > Avati >