Re: Fw: Re: Corvid gluster testing

Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx> · Thu, 07 Aug 2014 06:56:02 +0530



    On 08/07/2014 06:48 AM, Anand Avati
      wrote:

    
          On Wed, Aug 6, 2014 at 6:05 PM,
            Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>
            wrote:

            
              We checked this
                performance with plain distribute as well and on nfs it
                gave 25 minutes where as on nfs it gave around 90
                minutes after disabling throttling in both situations.
            
            
            This sentence is very confusing. Can you please state
              it more clearly?
          
        
    sorry :-D.

    We checked this performance on plain distribute volume by disabling
    throttling.

    On nfs the run took 25 minutes.

    On fuse the run took 90 minutes.

    
    Pranith

    
            Thanks
            

               I was wondering if
                any of you guys know what could contribute to this
                difference.

                    
                    Pranith
                
                  
                    On 08/07/2014 01:33 AM, Anand Avati wrote:

                    
                      Seems like heavy FINODELK
                        contention. As a diagnostic step, can you try
                        disabling eager-locking and check the write
                        performance again (gluster volume set $name
                        cluster.eager-lock off)?
                       

                        On Tue, Aug 5, 2014 at
                          11:44 AM, David F. Robinson <david.robinson@xxxxxxxxxxxxx>
                          wrote:

                          
                              Forgot to attach profile info in
                                previous email.  Attached...  
                              
                                   
                                  David
                                
                              
                                  ------ Original Message ------
                                  From: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
                                  To: gluster-devel@xxxxxxxxxxx
                                  Sent: 8/5/2014 2:41:34 PM
                                  Subject: Fw: Re: Corvid gluster
                                    testing
                                   
                                  
                                      I have been testing some of
                                        the fixes that
                                        Pranith incorporated into the
                                        3.5.2-beta to see how they
                                        performed for moderate levels of
                                        i/o. All of the stability issues
                                        that I had seen in previous
                                        versions seem to have been fixed
                                        in 3.5.2; however, there still
                                        seem to be some significant
                                        performance issues.  Pranith
                                        suggested that I send this to
                                        the gluster-devel email list, so
                                        here goes: 
                                       
                                      I am running an MPI job that
                                        saves a restart file to the
                                        gluster file system.  When I use
                                        the following in my fstab to
                                        mount the gluster volume, the
                                        i/o time for the 2.5GB file is
                                        roughly 45-seconds.
                                       
                                          gfsib01a.corvidtec.com:/homegfs

                                            /homegfs glusterfs
                                            transport=tcp,_netdev 0 0

                                          
                                      When I switch this to use the
                                        NFS protocol (see below), the
                                        i/o time is 2.5-seconds.
                                       
                                       
                                            gfsib01a.corvidtec.com:/homegfs
                                            /homegfs nfs
                                            vers=3,intr,bg,rsize=32768,wsize=32768
                                            0 0
                                       
                                      The read-times for gluster
                                        are 10-20% faster than NFS, but
                                        the write times are almost 20x
                                        slower.  
                                       
                                      I am running SL 6.4 and
                                        glusterfs-3.5.2-0.1.beta1.el6.x86_64...
                                      
                                       
                                      [root@gfs01a
                                            glusterfs]# gluster volume
                                            info homegfs

                                            Volume Name: homegfs

                                            Type: Distributed-Replicate

                                            Volume ID:
                                            1e32672a-f1b7-4b58-ba94-58c085e59071

                                            Status: Started

                                            Number of Bricks: 2 x 2 = 4

                                            Transport-type: tcp

                                            Bricks:

                                            Brick1:
                                            gfsib01a.corvidtec.com:/data/brick01a/homegfs

                                            Brick2:
                                            gfsib01b.corvidtec.com:/data/brick01b/homegfs

                                            Brick3:
                                            gfsib01a.corvidtec.com:/data/brick02a/homegfs

                                            Brick4:
                                            gfsib01b.corvidtec.com:/data/brick02b/homegfs

                                      
                                      David
                                       
                                      ------ Forwarded Message
                                        ------
                                      From: "Pranith Kumar
                                        Karampuri" <pkarampu@xxxxxxxxxx>
                                      To: "David Robinson" <david.robinson@xxxxxxxxxxxxx>
                                      Cc: "Young Thomas" <tom.young@xxxxxxxxxxxxx>
                                      Sent: 8/5/2014 2:25:38 AM
                                      Subject: Re: Corvid gluster
                                        testing
                                       
                                      
                                        gluster-devel@xxxxxxxxxxx
                                          is the email-id for the
                                          mailing list. We should
                                          probably start with the
                                          initial run numbers and the
                                          comparison for glusterfs mount
                                          and nfs mounts. May be
                                          something like 
                                         
                                        glusterfs mount: 90 minutes
                                        
                                        nfs mount: 25 minutes 
                                         
                                        And profile outputs, volume
                                          config, number of mounts,
                                          hardware configuration should
                                          be a good start. 
                                         
                                        Pranith 
                                         
                                        On 08/05/2014 09:28 AM,
                                          David Robinson wrote: 
                                        
                                          Thanks pranith 
                                           
                                           
                                          ===============================
                                          
                                          David F. Robinson, Ph.D.
                                          
                                          President - Corvid
                                            Technologies 
                                          704.799.6944

                                              x101 [office] 
                                          704.252.1310
                                            [cell] 
                                          704.799.7974
                                            [fax] 
                                          David.Robinson@xxxxxxxxxxxxx
                                          
                                          http://www.corvidtechnologies.com
                                          
                                           
                                            On Aug 4, 2014, at
                                              11:22 PM, Pranith Kumar
                                              Karampuri <pkarampu@xxxxxxxxxx>

                                              wrote: 
                                             
                                             
                                              On 08/05/2014 08:33
                                                AM, Pranith Kumar
                                                Karampuri wrote: 
                                               
                                              On 08/05/2014 08:29
                                                AM, David F. Robinson
                                                wrote: 
                                              
                                                
                                                    On 08/05/2014
                                                      12:51 AM, David F.
                                                      Robinson wrote: 
                                                    No. I don't
                                                      want to use nfs.
                                                      It eliminates most
                                                      of the benefits of
                                                      why I want to use
                                                      gluster. Failover
                                                      redundancy of the
                                                      pair, load
                                                      balancing, etc. 
                                                  
                                                  What is the
                                                    meaning of 'Failover
                                                    redundancy of the
                                                    pair, load balancing
                                                    ' Could you
                                                    elaborate more?
                                                    smb/nfs/glusterfs
                                                    are just access
                                                    protocols that
                                                    gluster supports
                                                    functionality is
                                                    almost same 
                                                
                                                Here is my
                                                  understanding. Please
                                                  correct me where I am
                                                  wrong. 
                                                 
                                                With gluster, if I
                                                  am doing a write and
                                                  one of the replicated
                                                  pairs goes down, there
                                                  is no interruption to
                                                  the I/o. The failover
                                                  is handled by gluster
                                                  and the fuse client.
                                                  This isn't done if I
                                                  use an nfs mount
                                                  unless the component
                                                  of the pair that goes
                                                  down isn't the one I
                                                  used for the mount. 
                                                 
                                                With nfs, I will
                                                  have to mount one of
                                                  the bricks. So, if I
                                                  have gfs01a, gfs01b,
                                                  gfs02a, gfs02b,
                                                  gfs03a, gfs03b, etc
                                                  and my fstab mounts
                                                  gfs01a, it is my
                                                  understanding that all
                                                  of my I/o will go
                                                  through gfs01a which
                                                  then gets distributed
                                                  to all of the other
                                                  bricks. Gfs01a
                                                  throughput becomes a
                                                  bottleneck. Where if I
                                                  do a gluster mount
                                                  using fuse, the load
                                                  balancing is handled
                                                  at the client side ,
                                                  not the server side.
                                                  If I have 1000-nodes
                                                  accessing 20-gluster
                                                  bricks, I need the
                                                  load balancing aspect.
                                                  I cannot have all
                                                  traffic going through
                                                  the network interface
                                                  on a single brick. 
                                                 
                                                If I am wrong with
                                                  the above assumptions,
                                                  I guess my question is
                                                  why would one ever use
                                                  the gluster mount
                                                  instead of nfs and/or
                                                  samba? 
                                                 
                                                Tom: feel free to
                                                  chime in if I have
                                                  missed anything. 
                                              
                                              I see your point now.
                                                Yes the gluster server
                                                where you did the mount
                                                is kind of a bottle
                                                neck. 
                                            
                                            Now that we established
                                              the problem is in the
                                              clients/protocols, you
                                              should send out a detailed
                                              mail on gluster-devel and
                                              see if anyone can help
                                              with you on performance
                                              xlators that can improve
                                              it a bit more. My area of
                                              expertise is more on
                                              replication. I am
                                              sub-maintainer for
                                              replication,locks
                                              components. I also know
                                              connection
                                              management/io-threads
                                              related issues which lead
                                              to hangs as I worked on
                                              them before. Performance
                                              xlators are black box to
                                              me. 
                                             
                                            Performance xlators are
                                              enabled only on fuse
                                              gluster stack. On nfs
                                              server mounts we disable
                                              all the performance
                                              xlators except
                                              write-behind as nfs client
                                              does lots of things for
                                              improving performance. I
                                              suggest you guys follow up
                                              more on gluster-devel. 
                                             
                                            Appreciate all the help
                                              you did for improving the
                                              product :-). Thanks a ton!
                                            
                                            Pranith 
                                            
                                              Pranith 
                                              
                                                David (Sent from
                                                  mobile) 
                                                 
                                                ===============================
                                                
                                                David F. Robinson,
                                                  Ph.D. 
                                                President - Corvid
                                                  Technologies 
                                                704.799.6944

                                                    x101 [office] 
                                                704.252.1310
                                                  [cell] 
                                                704.799.7974
                                                  [fax] 
                                                David.Robinson@xxxxxxxxxxxxx
                                                
                                                http://www.corvidtechnologies.com
                                                
                                              
_______________________________________________

                            Gluster-devel mailing list

                            Gluster-devel@xxxxxxxxxxx

                            http://supercolony.gluster.org/mailman/listinfo/gluster-devel

                            
                      _______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel

                    
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel