Gluster samba vfs read performance slow

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have already used "kernel oplocks = no" in the smb.conf, next is my original smb.conf file global settings:
[global]
        workgroup = MYGROUP
        server string = DCS Samba Server
        log file = /var/log/samba/log.vfs
        max log size = 500000
        aio read size = 262144
        aio write size = 262144
        aio write behind = true
        security = user
        passdb backend = tdbsam
        load printers = yes
        cups options = raw
        read raw = yes
        write raw = yes
        max xmit = 262144
        socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144
#       max protocol = SMB2
        kernel oplocks = no
        stat cache = no

thank you
-Kane
? 2013-9-18???1:38?Anand Avati <avati at redhat.com> ???

> On 9/17/13 10:34 PM, kane wrote:
>> Hi Anand,
>> 
>> I use 2 gluster server , this is my volume info:
>> Volume Name: soul
>> Type: Distribute
>> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: 192.168.101.133:/dcsdata/d0
>> Brick2: 192.168.101.134:/dcsdata/d0
>> 
>> each brick use a raid 5 logic disk with 8*2TSATA hdd.
>> 
>> smb.conf:
>> [gvol]
>>         comment = For samba export of volume  test
>>         vfs objects = glusterfs
>>         glusterfs:volfile_server = localhost
>>         glusterfs:volume = soul
>>         path = /
>>         read only = no
>>         guest ok = yes
>> 
>> this my testparm result:
>> [global]
>> workgroup = MYGROUP
>> server string = DCS Samba Server
>> log file = /var/log/samba/log.vfs
>> max log size = 500000
>> max xmit = 262144
>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 
>> SO_SNDBUF=262144
>> stat cache = No
>> kernel oplocks = No
>> idmap config * : backend = tdb
>> aio read size = 262144
>> aio write size = 262144
>> aio write behind = true
>> cups options = raw
>> 
>> in client mount the smb share with cifs to dir /mnt/vfs,
>> then use iozone executed in the cifs mount dir "/mnt/vfs":
>> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4
>> File size set to 10485760 KB
>> Record Size 128 KB
>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
>> Output is in Kbytes/sec
>> Time Resolution = 0.000001 seconds.
>> Processor cache size set to 1024 Kbytes.
>> Processor cache line size set to 32 bytes.
>> File stride size set to 17 * record size.
>> Throughput test with 4 processes
>> Each process writes a 10485760 Kbyte file in 128 Kbyte records
>> 
>> Children see throughput for  4 initial writers =  534315.84 KB/sec
>> Parent sees throughput for  4 initial writers =  519428.83 KB/sec
>> Min throughput per process =  133154.69 KB/sec
>> Max throughput per process =  134341.05 KB/sec
>> Avg throughput per process =  133578.96 KB/sec
>> Min xfer = 10391296.00 KB
>> 
>> Children see throughput for  4 rewriters =  536634.88 KB/sec
>> Parent sees throughput for  4 rewriters =  522618.54 KB/sec
>> Min throughput per process =  133408.80 KB/sec
>> Max throughput per process =  134721.36 KB/sec
>> Avg throughput per process =  134158.72 KB/sec
>> Min xfer = 10384384.00 KB
>> 
>> Children see throughput for  4 readers =   77403.54 KB/sec
>> Parent sees throughput for  4 readers =   77402.86 KB/sec
>> Min throughput per process =   19349.42 KB/sec
>> Max throughput per process =   19353.42 KB/sec
>> Avg throughput per process =   19350.88 KB/sec
>> Min xfer = 10483712.00 KB
>> 
>> Children see throughput for 4 re-readers =   77424.40 KB/sec
>> Parent sees throughput for 4 re-readers =   77423.89 KB/sec
>> Min throughput per process =   19354.75 KB/sec
>> Max throughput per process =   19358.50 KB/sec
>> Avg throughput per process =   19356.10 KB/sec
>> Min xfer = 10483840.00 KB
>> 
>> then the use the same command test in the dir mounted with glister fuse:
>> File size set to 10485760 KB
>> Record Size 128 KB
>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
>> Output is in Kbytes/sec
>> Time Resolution = 0.000001 seconds.
>> Processor cache size set to 1024 Kbytes.
>> Processor cache line size set to 32 bytes.
>> File stride size set to 17 * record size.
>> Throughput test with 4 processes
>> Each process writes a 10485760 Kbyte file in 128 Kbyte records
>> 
>> Children see throughput for  4 initial writers =  887534.72 KB/sec
>> Parent sees throughput for  4 initial writers =  848830.39 KB/sec
>> Min throughput per process =  220140.91 KB/sec
>> Max throughput per process =  223690.45 KB/sec
>> Avg throughput per process =  221883.68 KB/sec
>> Min xfer = 10319360.00 KB
>> 
>> Children see throughput for  4 rewriters =  892774.92 KB/sec
>> Parent sees throughput for  4 rewriters =  871186.83 KB/sec
>> Min throughput per process =  222326.44 KB/sec
>> Max throughput per process =  223970.17 KB/sec
>> Avg throughput per process =  223193.73 KB/sec
>> Min xfer = 10431360.00 KB
>> 
>> Children see throughput for  4 readers =  605889.12 KB/sec
>> Parent sees throughput for  4 readers =  601767.96 KB/sec
>> Min throughput per process =  143133.14 KB/sec
>> Max throughput per process =  159550.88 KB/sec
>> Avg throughput per process =  151472.28 KB/sec
>> Min xfer = 9406848.00 KB
>> 
>> it shows much higher perf.
>> 
>> any places i did wrong?
>> 
>> 
>> thank you
>> -Kane
>> 
>> ? 2013-9-18???1:19?Anand Avati <avati at gluster.org 
>> <mailto:avati at gluster.org>> ???
>> 
>>> How are you testing this? What tool are you using?
>>> 
>>> Avati
>>> 
>>> 
>>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com 
>>> <mailto:stef_9k at 163.com>> wrote:
>>> 
>>>    Hi Vijay
>>> 
>>>            I used the code in
>>>    https://github.com/gluster/glusterfs.git with the lasted commit:
>>>    commit de2a8d303311bd600cb93a775bc79a0edea1ee1a
>>>    Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>>
>>>    Date:   Tue Sep 17 16:45:03 2013 -0700
>>> 
>>>        Revert "cluster/distribute: Rebalance should also verify free
>>>    inodes"
>>> 
>>>        This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa
>>> 
>>>        Realized soon after merging, ?.
>>> 
>>>    which include the patch you mentioned last time improve read perf,
>>>    written by Anand.
>>> 
>>>    but the read perf was still slow:
>>>    write: 500MB/s
>>>    read: 77MB/s
>>> 
>>>    while via fuse :
>>>    write 800MB/s
>>>    read 600MB/s
>>> 
>>>    any advises?
>>> 
>>> 
>>>    Thank you.
>>>    -Kane
>>> 
>>>    ? 2013-9-13???10:37?kane <stef_9k at 163.com
>>>    <mailto:stef_9k at 163.com>> ???
>>> 
>>>> Hi Vijay?
>>>> 
>>>>      thank you for post this message, i will try it soon
>>>> 
>>>> -kane
>>>> 
>>>> 
>>>> 
>>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur at redhat.com
>>>    <mailto:vbellur at redhat.com>> ???
>>>> 
>>>>> On 09/13/2013 06:10 PM, kane wrote:
>>>>>> Hi
>>>>>> 
>>>>>> We use gluster samba vfs test io,but the read performance via
>>>    vfs is
>>>>>> half of write perfomance,
>>>>>> but via fuse the read and write performance is almost the same.
>>>>>> 
>>>>>> this is our smb.conf:
>>>>>> [global]
>>>>>>       workgroup = MYGROUP
>>>>>>       server string = DCS Samba Server
>>>>>>       log file = /var/log/samba/log.vfs
>>>>>>       max log size = 500000
>>>>>> #       use sendfile = true
>>>>>>       aio read size = 262144
>>>>>>       aio write size = 262144
>>>>>>       aio write behind = true
>>>>>>       min receivefile size = 262144
>>>>>>       write cache size = 268435456
>>>>>>       security = user
>>>>>>       passdb backend = tdbsam
>>>>>>       load printers = yes
>>>>>>       cups options = raw
>>>>>>       read raw = yes
>>>>>>       write raw = yes
>>>>>>       max xmit = 262144
>>>>>>       socket options = TCP_NODELAY IPTOS_LOWDELAY
>>>    SO_RCVBUF=262144
>>>>>> SO_SNDBUF=262144
>>>>>>       kernel oplocks = no
>>>>>>       stat cache = no
>>>>>> 
>>>>>> any advises helpful?
>>>>>> 
>>>>> 
>>>>> This patch has shown improvement in read performance with libgfapi:
>>>>> 
>>>>> http://review.gluster.org/#/c/5897/
>>>>> 
>>>>> Would it be possible for you to try this patch and check if it
>>>    improves performance in your case?
>>>>> 
>>>>> -Vijay
>>>>> 
>>>> 
>>> 
>>> 
>>>    _______________________________________________
>>>    Gluster-users mailing list
>>>    Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>    http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>> 
>>> 
>> 
> 
> Please add 'kernel oplocks = no' in the [gvol] section and try again.
> 
> Avati
> 




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux