Gluster samba vfs read performance slow

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I compared the profile dumps while write and read is separately running;

writing:
------------------------------------------------
Interval 58 Stats:
   Block Size:              65536b+              131072b+ 
 No. of Reads:                    0                     0 
No. of Writes:                27120                 10500 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
    100.00     133.51 us      36.00 us    1339.00 us          37619       WRITE
 
    Duration: 12 seconds
   Data Read: 0 bytes
Data Written: 3153854464 bytes
------------------------------------------------


read:
------------------------------------------------
Interval 63 Stats:
   Block Size:             131072b+ 
 No. of Reads:                 3529 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.54      87.86 us      68.00 us     127.00 us             42       FSTAT
     99.46     193.68 us      89.00 us    2121.00 us           3529        READ
 
    Duration: 12 seconds
   Data Read: 462553088 bytes
Data Written: 0 bytes
------------------------------------------------



two server brick avg dumps:
================================
Brick: 192.168.101.133:/dcsdata/d0
----------------------------------
Cumulative Stats:
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    2                     1                     1 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0               1613832                     0 
No. of Writes:              2282474               1148962                   227 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             14      FORGET
      0.00       0.00 us       0.00 us       0.00 us             39     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            114  RELEASEDIR
      0.00      84.50 us      54.00 us     115.00 us              2     OPENDIR
      0.00      79.00 us      52.00 us     127.00 us              4        OPEN
      0.00      47.00 us      14.00 us     130.00 us              8       FLUSH
      0.00     342.00 us     311.00 us     373.00 us              2      CREATE
      0.00     104.77 us      26.00 us     281.00 us             13      STATFS
      0.01     131.75 us      35.00 us     285.00 us             93      LOOKUP
      0.02    7446.00 us     104.00 us   29191.00 us              4    READDIRP
      0.07    2784.89 us      49.00 us   49224.00 us             36    GETXATTR
      0.20      64.49 us      29.00 us     164.00 us           4506       FSTAT
      1.07  399482.25 us  361616.00 us  450370.00 us              4      UNLINK
     42.87     167.36 us      56.00 us   44827.00 us         381080        READ
     55.76      71.51 us      35.00 us    7032.00 us        1159912       WRITE
 
    Duration: 22156 seconds
   Data Read: 211528187904 bytes
Data Written: 300276908032 bytes
 
Interval 71 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR
      0.18      54.00 us      54.00 us      54.00 us              1     OPENDIR
      1.05     107.33 us      40.00 us     217.00 us              3      STATFS
      2.90     126.57 us      81.00 us     256.00 us              7      LOOKUP
     95.88   14669.00 us     147.00 us   29191.00 us              2    READDIRP
 
    Duration: 581 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: 192.168.101.134:/dcsdata/d0
----------------------------------
Cumulative Stats:
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    2                     3                    24 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                   22               1563063                     0 
No. of Writes:              1522412               1525007                   184 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             14      FORGET
      0.00       0.00 us       0.00 us       0.00 us             39     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            114  RELEASEDIR
      0.00     116.50 us     111.00 us     122.00 us              2     OPENDIR
      0.00      69.25 us      23.00 us      95.00 us              8       FLUSH
      0.00     418.00 us     285.00 us     551.00 us              2      CREATE
      0.00     239.25 us     101.00 us     396.00 us              4    READDIRP
      0.00      93.00 us      39.00 us     249.00 us             13      STATFS
      0.01     142.89 us      78.00 us     241.00 us             87      LOOKUP
      0.09   48402.25 us     114.00 us   99173.00 us              4        OPEN
      0.19   10974.42 us      60.00 us  345979.00 us             36    GETXATTR
      0.20      94.33 us      41.00 us     200.00 us           4387       FSTAT
      0.85  440436.25 us  381525.00 us  582989.00 us              4      UNLINK
     35.80     193.96 us      57.00 us   23312.00 us         380869        READ
     62.86     134.89 us      29.00 us    9976.00 us         961593       WRITE
 
    Duration: 22155 seconds
   Data Read: 204875400152 bytes
Data Written: 299728837956 bytes
================================


Kane

? 2013-9-18???2:45?Anand Avati <avati at gluster.org> ???

> Can you get the volume profile dumps for both the runs and compare them?
> 
> Avati
> 
> 
> 
> On Tue, Sep 17, 2013 at 10:46 PM, kane <stef_9k at 163.com> wrote:
> I have already used "kernel oplocks = no" in the smb.conf, next is my original smb.conf file global settings:
> [global]
>         workgroup = MYGROUP
>         server string = DCS Samba Server
>         log file = /var/log/samba/log.vfs
>         max log size = 500000
>         aio read size = 262144
>         aio write size = 262144
>         aio write behind = true
>         security = user
>         passdb backend = tdbsam
>         load printers = yes
>         cups options = raw
>         read raw = yes
>         write raw = yes
>         max xmit = 262144
>         socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144
> #       max protocol = SMB2
>         kernel oplocks = no
>         stat cache = no
> 
> thank you
> -Kane
> ? 2013-9-18???1:38?Anand Avati <avati at redhat.com> ???
> 
> > On 9/17/13 10:34 PM, kane wrote:
> >> Hi Anand,
> >>
> >> I use 2 gluster server , this is my volume info:
> >> Volume Name: soul
> >> Type: Distribute
> >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6
> >> Status: Started
> >> Number of Bricks: 2
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: 192.168.101.133:/dcsdata/d0
> >> Brick2: 192.168.101.134:/dcsdata/d0
> >>
> >> each brick use a raid 5 logic disk with 8*2TSATA hdd.
> >>
> >> smb.conf:
> >> [gvol]
> >>         comment = For samba export of volume  test
> >>         vfs objects = glusterfs
> >>         glusterfs:volfile_server = localhost
> >>         glusterfs:volume = soul
> >>         path = /
> >>         read only = no
> >>         guest ok = yes
> >>
> >> this my testparm result:
> >> [global]
> >> workgroup = MYGROUP
> >> server string = DCS Samba Server
> >> log file = /var/log/samba/log.vfs
> >> max log size = 500000
> >> max xmit = 262144
> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> >> SO_SNDBUF=262144
> >> stat cache = No
> >> kernel oplocks = No
> >> idmap config * : backend = tdb
> >> aio read size = 262144
> >> aio write size = 262144
> >> aio write behind = true
> >> cups options = raw
> >>
> >> in client mount the smb share with cifs to dir /mnt/vfs,
> >> then use iozone executed in the cifs mount dir "/mnt/vfs":
> >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> File size set to 10485760 KB
> >> Record Size 128 KB
> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> Output is in Kbytes/sec
> >> Time Resolution = 0.000001 seconds.
> >> Processor cache size set to 1024 Kbytes.
> >> Processor cache line size set to 32 bytes.
> >> File stride size set to 17 * record size.
> >> Throughput test with 4 processes
> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records
> >>
> >> Children see throughput for  4 initial writers =  534315.84 KB/sec
> >> Parent sees throughput for  4 initial writers =  519428.83 KB/sec
> >> Min throughput per process =  133154.69 KB/sec
> >> Max throughput per process =  134341.05 KB/sec
> >> Avg throughput per process =  133578.96 KB/sec
> >> Min xfer = 10391296.00 KB
> >>
> >> Children see throughput for  4 rewriters =  536634.88 KB/sec
> >> Parent sees throughput for  4 rewriters =  522618.54 KB/sec
> >> Min throughput per process =  133408.80 KB/sec
> >> Max throughput per process =  134721.36 KB/sec
> >> Avg throughput per process =  134158.72 KB/sec
> >> Min xfer = 10384384.00 KB
> >>
> >> Children see throughput for  4 readers =   77403.54 KB/sec
> >> Parent sees throughput for  4 readers =   77402.86 KB/sec
> >> Min throughput per process =   19349.42 KB/sec
> >> Max throughput per process =   19353.42 KB/sec
> >> Avg throughput per process =   19350.88 KB/sec
> >> Min xfer = 10483712.00 KB
> >>
> >> Children see throughput for 4 re-readers =   77424.40 KB/sec
> >> Parent sees throughput for 4 re-readers =   77423.89 KB/sec
> >> Min throughput per process =   19354.75 KB/sec
> >> Max throughput per process =   19358.50 KB/sec
> >> Avg throughput per process =   19356.10 KB/sec
> >> Min xfer = 10483840.00 KB
> >>
> >> then the use the same command test in the dir mounted with glister fuse:
> >> File size set to 10485760 KB
> >> Record Size 128 KB
> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> Output is in Kbytes/sec
> >> Time Resolution = 0.000001 seconds.
> >> Processor cache size set to 1024 Kbytes.
> >> Processor cache line size set to 32 bytes.
> >> File stride size set to 17 * record size.
> >> Throughput test with 4 processes
> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records
> >>
> >> Children see throughput for  4 initial writers =  887534.72 KB/sec
> >> Parent sees throughput for  4 initial writers =  848830.39 KB/sec
> >> Min throughput per process =  220140.91 KB/sec
> >> Max throughput per process =  223690.45 KB/sec
> >> Avg throughput per process =  221883.68 KB/sec
> >> Min xfer = 10319360.00 KB
> >>
> >> Children see throughput for  4 rewriters =  892774.92 KB/sec
> >> Parent sees throughput for  4 rewriters =  871186.83 KB/sec
> >> Min throughput per process =  222326.44 KB/sec
> >> Max throughput per process =  223970.17 KB/sec
> >> Avg throughput per process =  223193.73 KB/sec
> >> Min xfer = 10431360.00 KB
> >>
> >> Children see throughput for  4 readers =  605889.12 KB/sec
> >> Parent sees throughput for  4 readers =  601767.96 KB/sec
> >> Min throughput per process =  143133.14 KB/sec
> >> Max throughput per process =  159550.88 KB/sec
> >> Avg throughput per process =  151472.28 KB/sec
> >> Min xfer = 9406848.00 KB
> >>
> >> it shows much higher perf.
> >>
> >> any places i did wrong?
> >>
> >>
> >> thank you
> >> -Kane
> >>
> >> ? 2013-9-18???1:19?Anand Avati <avati at gluster.org
> >> <mailto:avati at gluster.org>> ???
> >>
> >>> How are you testing this? What tool are you using?
> >>>
> >>> Avati
> >>>
> >>>
> >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com
> >>> <mailto:stef_9k at 163.com>> wrote:
> >>>
> >>>    Hi Vijay
> >>>
> >>>            I used the code in
> >>>    https://github.com/gluster/glusterfs.git with the lasted commit:
> >>>    commit de2a8d303311bd600cb93a775bc79a0edea1ee1a
> >>>    Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>>
> >>>    Date:   Tue Sep 17 16:45:03 2013 -0700
> >>>
> >>>        Revert "cluster/distribute: Rebalance should also verify free
> >>>    inodes"
> >>>
> >>>        This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa
> >>>
> >>>        Realized soon after merging, ?.
> >>>
> >>>    which include the patch you mentioned last time improve read perf,
> >>>    written by Anand.
> >>>
> >>>    but the read perf was still slow:
> >>>    write: 500MB/s
> >>>    read: 77MB/s
> >>>
> >>>    while via fuse :
> >>>    write 800MB/s
> >>>    read 600MB/s
> >>>
> >>>    any advises?
> >>>
> >>>
> >>>    Thank you.
> >>>    -Kane
> >>>
> >>>    ? 2013-9-13???10:37?kane <stef_9k at 163.com
> >>>    <mailto:stef_9k at 163.com>> ???
> >>>
> >>>> Hi Vijay?
> >>>>
> >>>>      thank you for post this message, i will try it soon
> >>>>
> >>>> -kane
> >>>>
> >>>>
> >>>>
> >>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur at redhat.com
> >>>    <mailto:vbellur at redhat.com>> ???
> >>>>
> >>>>> On 09/13/2013 06:10 PM, kane wrote:
> >>>>>> Hi
> >>>>>>
> >>>>>> We use gluster samba vfs test io,but the read performance via
> >>>    vfs is
> >>>>>> half of write perfomance,
> >>>>>> but via fuse the read and write performance is almost the same.
> >>>>>>
> >>>>>> this is our smb.conf:
> >>>>>> [global]
> >>>>>>       workgroup = MYGROUP
> >>>>>>       server string = DCS Samba Server
> >>>>>>       log file = /var/log/samba/log.vfs
> >>>>>>       max log size = 500000
> >>>>>> #       use sendfile = true
> >>>>>>       aio read size = 262144
> >>>>>>       aio write size = 262144
> >>>>>>       aio write behind = true
> >>>>>>       min receivefile size = 262144
> >>>>>>       write cache size = 268435456
> >>>>>>       security = user
> >>>>>>       passdb backend = tdbsam
> >>>>>>       load printers = yes
> >>>>>>       cups options = raw
> >>>>>>       read raw = yes
> >>>>>>       write raw = yes
> >>>>>>       max xmit = 262144
> >>>>>>       socket options = TCP_NODELAY IPTOS_LOWDELAY
> >>>    SO_RCVBUF=262144
> >>>>>> SO_SNDBUF=262144
> >>>>>>       kernel oplocks = no
> >>>>>>       stat cache = no
> >>>>>>
> >>>>>> any advises helpful?
> >>>>>>
> >>>>>
> >>>>> This patch has shown improvement in read performance with libgfapi:
> >>>>>
> >>>>> http://review.gluster.org/#/c/5897/
> >>>>>
> >>>>> Would it be possible for you to try this patch and check if it
> >>>    improves performance in your case?
> >>>>>
> >>>>> -Vijay
> >>>>>
> >>>>
> >>>
> >>>
> >>>    _______________________________________________
> >>>    Gluster-users mailing list
> >>>    Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >>>    http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>>
> >>
> >
> > Please add 'kernel oplocks = no' in the [gvol] section and try again.
> >
> > Avati
> >
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/ab613dc7/attachment.html>


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux