Re: Fw: Re: Corvid gluster testing

Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx> · Thu, 07 Aug 2014 07:21:11 +0530



    On 08/07/2014 07:18 AM, Anand Avati
      wrote:

    
      It would be worth checking the perf numbers without
        -o acl (in case it was enabled, as seen in the other gid
        thread). Client side -o acl mount option can have a negative
        impact on performance because of the increased number of
        up-calls from FUSE for access().
    
    Actually it is all write intensive.

    here are the numbers they gave me from earlier runs:

     %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
    calls         Fop

     ---------   -----------   -----------   -----------  
    ------------        ----

          0.00       0.00 us       0.00 us       0.00 us            
    99      FORGET

          0.00       0.00 us       0.00 us       0.00 us          
    1093     RELEASE

          0.00       0.00 us       0.00 us       0.00 us            468 
    RELEASEDIR

          0.00      60.00 us      26.00 us     107.00 us             
    4     SETATTR

          0.00      91.56 us      42.00 us     157.00 us            
    27      UNLINK

          0.00      20.75 us      12.00 us      55.00 us           
    132    GETXATTR

          0.00      19.03 us       9.00 us      95.00 us           
    152    READLINK

          0.00      43.19 us      12.00 us     106.00 us            
    83        OPEN

          0.00      18.37 us       8.00 us      92.00 us           
    257      STATFS

          0.00      32.42 us      11.00 us     118.00 us           
    322     OPENDIR

          0.00      36.09 us       5.00 us     109.00 us           
    359       FSTAT

          0.00      51.14 us      37.00 us     183.00 us           
    663      RENAME

          0.00      33.32 us       6.00 us     123.00 us          
    1451        STAT

          0.00     821.79 us      21.00 us   22678.00 us            
    84        READ

          0.00      34.88 us       3.00 us     139.00 us          
    2326       FLUSH

          0.01     789.33 us      72.00 us   64054.00 us           
    347      CREATE

          0.01    1144.63 us      43.00 us  280735.00 us           
    337   FTRUNCATE

          0.01      47.82 us      16.00 us   19817.00 us         
    16513      LOOKUP

          0.02     604.85 us      11.00 us    1233.00 us          
    1423    READDIRP

         99.95      17.51 us       6.00 us  212701.00 us     
    300715967       WRITE

     
        Duration: 5390 seconds

       Data Read: 1495257497 bytes

    Data Written: 166546887668 bytes

    
    Pranith

    
        Thanks
      
      
        On Wed, Aug 6, 2014 at 6:26 PM, Pranith
          Kumar Karampuri <pkarampu@xxxxxxxxxx>
          wrote:

          
                On 08/07/2014 06:48 AM, Anand Avati wrote:

                
                      On Wed, Aug 6, 2014 at
                        6:05 PM, Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>
                        wrote:

                        
                          We
                            checked this performance with plain
                            distribute as well and on nfs it gave 25
                            minutes where as on nfs it gave around 90
                            minutes after disabling throttling in both
                            situations.
                        
                        
                        This sentence is very confusing. Can you
                          please state it more clearly?
                      
                    
              sorry :-D.

              We checked this performance on plain distribute volume by
              disabling throttling.

              On nfs the run took 25 minutes.

              On fuse the run took 90 minutes.

                  
                  Pranith
              
                
                          Thanks
                          

                             I was
                              wondering if any of you guys know what
                              could contribute to this difference.

                                  
                                  Pranith
                              
                                
                                  On 08/07/2014 01:33 AM, Anand
                                    Avati wrote:

                                  
                                    Seems like heavy
                                      FINODELK contention. As a
                                      diagnostic step, can you try
                                      disabling eager-locking and check
                                      the write performance again
                                      (gluster volume set $name
                                      cluster.eager-lock off)?
                                     

                                      On Tue,
                                        Aug 5, 2014 at 11:44 AM, David
                                        F. Robinson <david.robinson@xxxxxxxxxxxxx>
                                        wrote:

                                        
                                            Forgot to attach
                                              profile info in previous
                                              email.  Attached...  
                                            
                                                 
                                                David
                                              
                                            
                                                ------ Original
                                                  Message ------
                                                From: "David F.
                                                  Robinson" <david.robinson@xxxxxxxxxxxxx>
                                                To: gluster-devel@xxxxxxxxxxx
                                                Sent: 8/5/2014
                                                  2:41:34 PM
                                                Subject: Fw: Re:
                                                  Corvid gluster testing
                                                 
                                                
                                                    I have been
                                                      testing some of
                                                      the fixes that
                                                      Pranith incorporated
                                                      into the
                                                      3.5.2-beta to see
                                                      how they performed
                                                      for moderate
                                                      levels of i/o. All
                                                      of the stability
                                                      issues that I had
                                                      seen in previous
                                                      versions seem to
                                                      have been fixed in
                                                      3.5.2; however,
                                                      there still seem
                                                      to be some
                                                      significant
                                                      performance
                                                      issues.  Pranith
                                                      suggested that I
                                                      send this to the
                                                      gluster-devel
                                                      email list, so
                                                      here goes: 
                                                     
                                                    I am running an
                                                      MPI job that saves
                                                      a restart file to
                                                      the gluster file
                                                      system.  When I
                                                      use the following
                                                      in my fstab to
                                                      mount the gluster
                                                      volume, the i/o
                                                      time for the 2.5GB
                                                      file is roughly
                                                      45-seconds.
                                                     
                                                        gfsib01a.corvidtec.com:/homegfs


                                                          /homegfs
                                                          glusterfs
                                                          transport=tcp,_netdev
                                                          0 0

                                                        
                                                    When I switch
                                                      this to use the
                                                      NFS protocol (see
                                                      below), the i/o
                                                      time is
                                                      2.5-seconds.
                                                     
                                                     
                                                          gfsib01a.corvidtec.com:/homegfs
                                                          /homegfs nfs
                                                          vers=3,intr,bg,rsize=32768,wsize=32768
                                                          0 0
                                                     
                                                    The read-times
                                                      for gluster are
                                                      10-20% faster than
                                                      NFS, but the write
                                                      times are almost
                                                      20x slower.  
                                                     
                                                    I am running SL
                                                      6.4 and
                                                      glusterfs-3.5.2-0.1.beta1.el6.x86_64...
                                                    
                                                     
                                                    [root@gfs01a

                                                          glusterfs]#
                                                          gluster volume
                                                          info homegfs

                                                          Volume Name:
                                                          homegfs

                                                          Type:
                                                          Distributed-Replicate

                                                          Volume ID:
                                                          1e32672a-f1b7-4b58-ba94-58c085e59071

                                                          Status:
                                                          Started

                                                          Number of
                                                          Bricks: 2 x 2
                                                          = 4

                                                          Transport-type:
                                                          tcp

                                                          Bricks:

                                                          Brick1:
                                                          gfsib01a.corvidtec.com:/data/brick01a/homegfs

                                                          Brick2:
                                                          gfsib01b.corvidtec.com:/data/brick01b/homegfs

                                                          Brick3:
                                                          gfsib01a.corvidtec.com:/data/brick02a/homegfs

                                                          Brick4:
                                                          gfsib01b.corvidtec.com:/data/brick02b/homegfs

                                                    
                                                    David
                                                     
                                                    ------
                                                      Forwarded Message
                                                      ------
                                                    From: "Pranith
                                                      Kumar Karampuri"
                                                      <pkarampu@xxxxxxxxxx>
                                                    To: "David
                                                      Robinson" <david.robinson@xxxxxxxxxxxxx>
                                                    Cc: "Young
                                                      Thomas" <tom.young@xxxxxxxxxxxxx>
                                                    Sent: 8/5/2014
                                                      2:25:38 AM
                                                    Subject: Re:
                                                      Corvid gluster
                                                      testing
                                                     
                                                    
                                                      gluster-devel@xxxxxxxxxxx
                                                        is the email-id
                                                        for the mailing
                                                        list. We should
                                                        probably start
                                                        with the initial
                                                        run numbers and
                                                        the comparison
                                                        for glusterfs
                                                        mount and nfs
                                                        mounts. May be
                                                        something like 
                                                       
                                                      glusterfs
                                                        mount: 90
                                                        minutes 
                                                      nfs mount: 25
                                                        minutes 
                                                       
                                                      And profile
                                                        outputs, volume
                                                        config, number
                                                        of mounts,
                                                        hardware
                                                        configuration
                                                        should be a good
                                                        start. 
                                                       
                                                      Pranith 
                                                       
                                                      On 08/05/2014
                                                        09:28 AM, David
                                                        Robinson wrote:
                                                      
                                                      
                                                        Thanks
                                                          pranith 
                                                         
                                                         
                                                        ===============================

                                                        
                                                        David F.
                                                          Robinson,
                                                          Ph.D. 
                                                        President -
                                                          Corvid
                                                          Technologies 
                                                        704.799.6944


                                                          x101
                                                          [office] 
                                                        704.252.1310
                                                          [cell] 
                                                        704.799.7974
                                                          [fax] 
                                                        David.Robinson@xxxxxxxxxxxxx
                                                        
                                                        http://www.corvidtechnologies.com
                                                        
                                                         
                                                          On Aug 4,
                                                          2014, at 11:22
                                                          PM, Pranith
                                                          Kumar
                                                          Karampuri <pkarampu@xxxxxxxxxx>


                                                          wrote: 
                                                           
                                                           
                                                          On
                                                          08/05/2014
                                                          08:33 AM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote: 
                                                           
                                                          On
                                                          08/05/2014
                                                          08:29 AM,
                                                          David F.
                                                          Robinson
                                                          wrote: 
                                                          
                                                          
                                                          On
                                                          08/05/2014
                                                          12:51 AM,
                                                          David F.
                                                          Robinson
                                                          wrote: 
                                                          No. I
                                                          don't want to
                                                          use nfs. It
                                                          eliminates
                                                          most of the
                                                          benefits of
                                                          why I want to
                                                          use gluster.
                                                          Failover
                                                          redundancy of
                                                          the pair, load
                                                          balancing,
                                                          etc. 
                                                          
                                                          What is
                                                          the meaning of
                                                          'Failover
                                                          redundancy of
                                                          the pair, load
                                                          balancing '
                                                          Could you
                                                          elaborate
                                                          more?
                                                          smb/nfs/glusterfs
                                                          are just
                                                          access
                                                          protocols that
                                                          gluster
                                                          supports
                                                          functionality
                                                          is almost same
                                                          
                                                          
                                                          Here is
                                                          my
                                                          understanding.
                                                          Please correct
                                                          me where I am
                                                          wrong. 
                                                           
                                                          With
                                                          gluster, if I
                                                          am doing a
                                                          write and one
                                                          of the
                                                          replicated
                                                          pairs goes
                                                          down, there is
                                                          no
                                                          interruption
                                                          to the I/o.
                                                          The failover
                                                          is handled by
                                                          gluster and
                                                          the fuse
                                                          client. This
                                                          isn't done if
                                                          I use an nfs
                                                          mount unless
                                                          the component
                                                          of the pair
                                                          that goes down
                                                          isn't the one
                                                          I used for the
                                                          mount. 
                                                           
                                                          With nfs,
                                                          I will have to
                                                          mount one of
                                                          the bricks.
                                                          So, if I have
                                                          gfs01a,
                                                          gfs01b,
                                                          gfs02a,
                                                          gfs02b,
                                                          gfs03a,
                                                          gfs03b, etc
                                                          and my fstab
                                                          mounts gfs01a,
                                                          it is my
                                                          understanding
                                                          that all of my
                                                          I/o will go
                                                          through gfs01a
                                                          which then
                                                          gets
                                                          distributed to
                                                          all of the
                                                          other bricks.
                                                          Gfs01a
                                                          throughput
                                                          becomes a
                                                          bottleneck.
                                                          Where if I do
                                                          a gluster
                                                          mount using
                                                          fuse, the load
                                                          balancing is
                                                          handled at the
                                                          client side ,
                                                          not the server
                                                          side. If I
                                                          have
                                                          1000-nodes
                                                          accessing
                                                          20-gluster
                                                          bricks, I need
                                                          the load
                                                          balancing
                                                          aspect. I
                                                          cannot have
                                                          all traffic
                                                          going through
                                                          the network
                                                          interface on a
                                                          single brick.
                                                          
                                                           
                                                          If I am
                                                          wrong with the
                                                          above
                                                          assumptions, I
                                                          guess my
                                                          question is
                                                          why would one
                                                          ever use the
                                                          gluster mount
                                                          instead of nfs
                                                          and/or samba?
                                                          
                                                           
                                                          Tom: feel
                                                          free to chime
                                                          in if I have
                                                          missed
                                                          anything. 
                                                          
                                                          I see
                                                          your point
                                                          now. Yes the
                                                          gluster server
                                                          where you did
                                                          the mount is
                                                          kind of a
                                                          bottle neck. 
                                                          
                                                          Now that
                                                          we established
                                                          the problem is
                                                          in the
                                                          clients/protocols,
                                                          you should
                                                          send out a
                                                          detailed mail
                                                          on
                                                          gluster-devel
                                                          and see if
                                                          anyone can
                                                          help with you
                                                          on performance
                                                          xlators that
                                                          can improve it
                                                          a bit more. My
                                                          area of
                                                          expertise is
                                                          more on
                                                          replication. I
                                                          am
                                                          sub-maintainer
                                                          for
                                                          replication,locks
                                                          components. I
                                                          also know
                                                          connection
                                                          management/io-threads
                                                          related issues
                                                          which lead to
                                                          hangs as I
                                                          worked on them
                                                          before.
                                                          Performance
                                                          xlators are
                                                          black box to
                                                          me. 
                                                           
                                                          Performance
                                                          xlators are
                                                          enabled only
                                                          on fuse
                                                          gluster stack.
                                                          On nfs server
                                                          mounts we
                                                          disable all
                                                          the
                                                          performance
                                                          xlators except
                                                          write-behind
                                                          as nfs client
                                                          does lots of
                                                          things for
                                                          improving
                                                          performance. I
                                                          suggest you
                                                          guys follow up
                                                          more on
                                                          gluster-devel.
                                                          
                                                           
                                                          Appreciate
                                                          all the help
                                                          you did for
                                                          improving the
                                                          product :-).
                                                          Thanks a ton!
                                                          
                                                          Pranith 
                                                          
                                                          Pranith 
                                                          
                                                          David
                                                          (Sent from
                                                          mobile) 
                                                           
                                                          ===============================

                                                          
                                                          David F.
                                                          Robinson,
                                                          Ph.D. 
                                                          President
                                                          - Corvid
                                                          Technologies 
                                                          704.799.6944


                                                          x101
                                                          [office] 
                                                          704.252.1310
                                                          [cell] 
                                                          704.799.7974
                                                          [fax] 
                                                          David.Robinson@xxxxxxxxxxxxx
                                                          
                                                          http://www.corvidtechnologies.com
                                                          
                                                          
_______________________________________________

                                          Gluster-devel mailing list

                                          Gluster-devel@xxxxxxxxxxx

                                          http://supercolony.gluster.org/mailman/listinfo/gluster-devel

                                          
                                    _______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

                                  
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel