Gluster samba vfs read performance slow

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Can you get the volume profile dumps for both the runs and compare them?

Avati



On Tue, Sep 17, 2013 at 10:46 PM, kane <stef_9k at 163.com> wrote:

> I have already used "kernel oplocks = no" in the smb.conf, next is my
> original smb.conf file global settings:
> [global]
>         workgroup = MYGROUP
>         server string = DCS Samba Server
>         log file = /var/log/samba/log.vfs
>         max log size = 500000
>         aio read size = 262144
>         aio write size = 262144
>         aio write behind = true
>         security = user
>         passdb backend = tdbsam
>         load printers = yes
>         cups options = raw
>         read raw = yes
>         write raw = yes
>         max xmit = 262144
>         socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> SO_SNDBUF=262144
> #       max protocol = SMB2
>         kernel oplocks = no
>         stat cache = no
>
> thank you
> -Kane
> ? 2013-9-18???1:38?Anand Avati <avati at redhat.com> ???
>
> > On 9/17/13 10:34 PM, kane wrote:
> >> Hi Anand,
> >>
> >> I use 2 gluster server , this is my volume info:
> >> Volume Name: soul
> >> Type: Distribute
> >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6
> >> Status: Started
> >> Number of Bricks: 2
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: 192.168.101.133:/dcsdata/d0
> >> Brick2: 192.168.101.134:/dcsdata/d0
> >>
> >> each brick use a raid 5 logic disk with 8*2TSATA hdd.
> >>
> >> smb.conf:
> >> [gvol]
> >>         comment = For samba export of volume  test
> >>         vfs objects = glusterfs
> >>         glusterfs:volfile_server = localhost
> >>         glusterfs:volume = soul
> >>         path = /
> >>         read only = no
> >>         guest ok = yes
> >>
> >> this my testparm result:
> >> [global]
> >> workgroup = MYGROUP
> >> server string = DCS Samba Server
> >> log file = /var/log/samba/log.vfs
> >> max log size = 500000
> >> max xmit = 262144
> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> >> SO_SNDBUF=262144
> >> stat cache = No
> >> kernel oplocks = No
> >> idmap config * : backend = tdb
> >> aio read size = 262144
> >> aio write size = 262144
> >> aio write behind = true
> >> cups options = raw
> >>
> >> in client mount the smb share with cifs to dir /mnt/vfs,
> >> then use iozone executed in the cifs mount dir "/mnt/vfs":
> >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> File size set to 10485760 KB
> >> Record Size 128 KB
> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> Output is in Kbytes/sec
> >> Time Resolution = 0.000001 seconds.
> >> Processor cache size set to 1024 Kbytes.
> >> Processor cache line size set to 32 bytes.
> >> File stride size set to 17 * record size.
> >> Throughput test with 4 processes
> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records
> >>
> >> Children see throughput for  4 initial writers =  534315.84 KB/sec
> >> Parent sees throughput for  4 initial writers =  519428.83 KB/sec
> >> Min throughput per process =  133154.69 KB/sec
> >> Max throughput per process =  134341.05 KB/sec
> >> Avg throughput per process =  133578.96 KB/sec
> >> Min xfer = 10391296.00 KB
> >>
> >> Children see throughput for  4 rewriters =  536634.88 KB/sec
> >> Parent sees throughput for  4 rewriters =  522618.54 KB/sec
> >> Min throughput per process =  133408.80 KB/sec
> >> Max throughput per process =  134721.36 KB/sec
> >> Avg throughput per process =  134158.72 KB/sec
> >> Min xfer = 10384384.00 KB
> >>
> >> Children see throughput for  4 readers =   77403.54 KB/sec
> >> Parent sees throughput for  4 readers =   77402.86 KB/sec
> >> Min throughput per process =   19349.42 KB/sec
> >> Max throughput per process =   19353.42 KB/sec
> >> Avg throughput per process =   19350.88 KB/sec
> >> Min xfer = 10483712.00 KB
> >>
> >> Children see throughput for 4 re-readers =   77424.40 KB/sec
> >> Parent sees throughput for 4 re-readers =   77423.89 KB/sec
> >> Min throughput per process =   19354.75 KB/sec
> >> Max throughput per process =   19358.50 KB/sec
> >> Avg throughput per process =   19356.10 KB/sec
> >> Min xfer = 10483840.00 KB
> >>
> >> then the use the same command test in the dir mounted with glister fuse:
> >> File size set to 10485760 KB
> >> Record Size 128 KB
> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> Output is in Kbytes/sec
> >> Time Resolution = 0.000001 seconds.
> >> Processor cache size set to 1024 Kbytes.
> >> Processor cache line size set to 32 bytes.
> >> File stride size set to 17 * record size.
> >> Throughput test with 4 processes
> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records
> >>
> >> Children see throughput for  4 initial writers =  887534.72 KB/sec
> >> Parent sees throughput for  4 initial writers =  848830.39 KB/sec
> >> Min throughput per process =  220140.91 KB/sec
> >> Max throughput per process =  223690.45 KB/sec
> >> Avg throughput per process =  221883.68 KB/sec
> >> Min xfer = 10319360.00 KB
> >>
> >> Children see throughput for  4 rewriters =  892774.92 KB/sec
> >> Parent sees throughput for  4 rewriters =  871186.83 KB/sec
> >> Min throughput per process =  222326.44 KB/sec
> >> Max throughput per process =  223970.17 KB/sec
> >> Avg throughput per process =  223193.73 KB/sec
> >> Min xfer = 10431360.00 KB
> >>
> >> Children see throughput for  4 readers =  605889.12 KB/sec
> >> Parent sees throughput for  4 readers =  601767.96 KB/sec
> >> Min throughput per process =  143133.14 KB/sec
> >> Max throughput per process =  159550.88 KB/sec
> >> Avg throughput per process =  151472.28 KB/sec
> >> Min xfer = 9406848.00 KB
> >>
> >> it shows much higher perf.
> >>
> >> any places i did wrong?
> >>
> >>
> >> thank you
> >> -Kane
> >>
> >> ? 2013-9-18???1:19?Anand Avati <avati at gluster.org
> >> <mailto:avati at gluster.org>> ???
> >>
> >>> How are you testing this? What tool are you using?
> >>>
> >>> Avati
> >>>
> >>>
> >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com
> >>> <mailto:stef_9k at 163.com>> wrote:
> >>>
> >>>    Hi Vijay
> >>>
> >>>            I used the code in
> >>>    https://github.com/gluster/glusterfs.git with the lasted commit:
> >>>    commit de2a8d303311bd600cb93a775bc79a0edea1ee1a
> >>>    Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>>
> >>>    Date:   Tue Sep 17 16:45:03 2013 -0700
> >>>
> >>>        Revert "cluster/distribute: Rebalance should also verify free
> >>>    inodes"
> >>>
> >>>        This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa
> >>>
> >>>        Realized soon after merging, ?.
> >>>
> >>>    which include the patch you mentioned last time improve read perf,
> >>>    written by Anand.
> >>>
> >>>    but the read perf was still slow:
> >>>    write: 500MB/s
> >>>    read: 77MB/s
> >>>
> >>>    while via fuse :
> >>>    write 800MB/s
> >>>    read 600MB/s
> >>>
> >>>    any advises?
> >>>
> >>>
> >>>    Thank you.
> >>>    -Kane
> >>>
> >>>    ? 2013-9-13???10:37?kane <stef_9k at 163.com
> >>>    <mailto:stef_9k at 163.com>> ???
> >>>
> >>>> Hi Vijay?
> >>>>
> >>>>      thank you for post this message, i will try it soon
> >>>>
> >>>> -kane
> >>>>
> >>>>
> >>>>
> >>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur at redhat.com
> >>>    <mailto:vbellur at redhat.com>> ???
> >>>>
> >>>>> On 09/13/2013 06:10 PM, kane wrote:
> >>>>>> Hi
> >>>>>>
> >>>>>> We use gluster samba vfs test io,but the read performance via
> >>>    vfs is
> >>>>>> half of write perfomance,
> >>>>>> but via fuse the read and write performance is almost the same.
> >>>>>>
> >>>>>> this is our smb.conf:
> >>>>>> [global]
> >>>>>>       workgroup = MYGROUP
> >>>>>>       server string = DCS Samba Server
> >>>>>>       log file = /var/log/samba/log.vfs
> >>>>>>       max log size = 500000
> >>>>>> #       use sendfile = true
> >>>>>>       aio read size = 262144
> >>>>>>       aio write size = 262144
> >>>>>>       aio write behind = true
> >>>>>>       min receivefile size = 262144
> >>>>>>       write cache size = 268435456
> >>>>>>       security = user
> >>>>>>       passdb backend = tdbsam
> >>>>>>       load printers = yes
> >>>>>>       cups options = raw
> >>>>>>       read raw = yes
> >>>>>>       write raw = yes
> >>>>>>       max xmit = 262144
> >>>>>>       socket options = TCP_NODELAY IPTOS_LOWDELAY
> >>>    SO_RCVBUF=262144
> >>>>>> SO_SNDBUF=262144
> >>>>>>       kernel oplocks = no
> >>>>>>       stat cache = no
> >>>>>>
> >>>>>> any advises helpful?
> >>>>>>
> >>>>>
> >>>>> This patch has shown improvement in read performance with libgfapi:
> >>>>>
> >>>>> http://review.gluster.org/#/c/5897/
> >>>>>
> >>>>> Would it be possible for you to try this patch and check if it
> >>>    improves performance in your case?
> >>>>>
> >>>>> -Vijay
> >>>>>
> >>>>
> >>>
> >>>
> >>>    _______________________________________________
> >>>    Gluster-users mailing list
> >>>    Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >>>    http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>>
> >>
> >
> > Please add 'kernel oplocks = no' in the [gvol] section and try again.
> >
> > Avati
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/7acaaa03/attachment.html>


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux