Re: Slow write times to gluster disk

Pat Haley <phaley@xxxxxxx> · Fri, 5 May 2017 11:12:31 -0400



    Hi Pranith,

    
    I presume you are asking for some version of the profile data that
    just shows the dd test (or a repeat of the dd test).  If yes, how do
    I extract just that data?

    
    Thanks

    
    Pat

    
    On 05/05/2017 10:58 AM, Pranith Kumar
      Karampuri wrote:

    
            hi Pat,

            
                  Let us concentrate on the performance numbers part for
            now. We will look at the permissions one after this?

            
          As per the profile info, only 2.6% of the work-load is writes.
          There are too many Lookups.

          
        Would it be possible to get the data for just the dd test you
        were doing earlier?

        
        On Fri, May 5, 2017 at 8:14 PM, Pat
          Haley <phaley@xxxxxxx>
          wrote:

          
              Hi Pranith & Ravi,

              
              A couple of quick questions

              
              We have profile turned on. Are there specific queries we
              should make that would help debug our configuration?  (The
              default profile info was previously sent in http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html
              but I'm not sure if that is what you were looking for.)

              
              We also started to do a test on serving gluster over NFS. 
              We rediscovered an issue we previously reported (
              http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html
              ) in that the NFS mounted version was ignoring the group
              write permissions.  What specific information would be
              useful in debugging this?

              
              Thanks

                  
                  Pat
              
                
                  On
                    04/14/2017 03:01 AM, Ravishankar N wrote:

                  
                    On
                      04/14/2017 12:20 PM, Pranith Kumar Karampuri
                      wrote:

                    
                          On Sat, Apr 8, 2017
                            at 10:28 AM, Ravishankar N <ravishankar@xxxxxxxxxx>
                            wrote:

                            
                                Hi
                                  Pat,

                                  
                                  I'm assuming you are using gluster
                                  native (fuse mount). If it helps, you
                                  could try mounting it via gluster NFS
                                  (gnfs) and then see if there is an
                                  improvement in speed. Fuse mounts are
                                  slower than gnfs mounts but you get
                                  the benefit of avoiding a single point
                                  of failure. Unlike fuse mounts, if the
                                  gluster node containing the gnfs
                                  server goes down, all mounts done
                                  using that node will fail). For fuse
                                  mounts, you could try tweaking the
                                  write-behind xlator settings to see if
                                  it helps. See the
                                  performance.write-behind and
                                  performance.write-behind-window-size
                                  options in `gluster volume set help`.
                                  Of course, even for gnfs mounts, you
                                  can achieve fail-over by using CTDB.

                                
                            Ravi,

                            
                                  Do you have any data that
                              suggests fuse mounts are slower than gNFS
                              servers? 

                            
                    I have heard anecdotal evidence time and again on
                    the ML and IRC, which is why I wanted to compare it
                    with NFS numbers on his setup. 

                    
                            Pat,

                            
                                  I see that I am late to the
                              thread, but do you happen to have "profile
                              info" of the workload?

                              
                            You can follow https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
                              to get the information.

                            
                    Yeah, Let's see if profile info shows up anything
                    interesting.

                    -Ravi

                    
                                  Thanks,

                                  Ravi
                                  
                                    
                                      On 04/08/2017 12:07 AM, Pat Haley
                                      wrote:

                                    
                                      Hi,

                                      
                                      We noticed a dramatic slowness
                                      when writing to a gluster disk
                                      when compared to writing to an NFS
                                      disk. Specifically when using dd
                                      (data duplicator) to write a 4.3
                                      GB file of zeros:

                                      
                                        on NFS disk (/home): 9.5
                                          Gb/s
                                        on gluster disk (/gdata):
                                          508 Mb/s

                                        
                                      The gluser disk is 2 bricks joined
                                      together, no replication or
                                      anything else. The hardware is
                                      (literally) the same:

                                      
                                        one server with 70 hard
                                          disks  and a hardware RAID
                                          card.
                                        4 disks in a RAID-6 group
                                          (the NFS disk)
                                        32 disks in a RAID-6 group
                                          (the max allowed by the card,
                                          /mnt/brick1)
                                        32 disks in another RAID-6
                                          group (/mnt/brick2)
                                        2 hot spare

                                        
                                      Some additional information and
                                        more tests results (after
                                        changing the log level):

                                      
                                      glusterfs 3.7.11 built on
                                          Apr 27 2016 14:09:22

                                        CentOS release 6.8 (Final)

                                        RAID bus controller: LSI Logic /
                                        Symbios Logic MegaRAID SAS-3
                                        3108 [Invader] (rev 02)

                                        
                                        Create the file to /gdata
                                          (gluster)

                                        [root@mseas-data2 gdata]# dd
                                        if=/dev/zero of=/gdata/zero1
                                        bs=1M count=1000

                                        1000+0 records in

                                        1000+0 records out

                                        1048576000 bytes (1.0 GB)
                                        copied, 1.91876 s, 546 MB/s

                                        
                                        Create the file to /home
                                          (ext4)

                                        [root@mseas-data2 gdata]# dd
                                        if=/dev/zero of=/home/zero1
                                        bs=1M count=1000

                                        1000+0 records in

                                        1000+0 records out

                                        1048576000 bytes (1.0 GB)
                                        copied, 0.686021 s, 1.5 GB/s
                                          - 3 times as fast

                                          
                                          Copy from /gdata to /gdata
                                          (gluster to gluster)

                                        [root@mseas-data2 gdata]# dd
                                        if=/gdata/zero1 of=/gdata/zero2

                                        2048000+0 records in

                                        2048000+0 records out

                                        1048576000 bytes (1.0 GB)
                                        copied, 101.052 s, 10.4 MB/s
                                        - realllyyy slooowww

                                        
                                        Copy from /gdata to /gdata
                                        2nd time (gluster to
                                            gluster)

                                        [root@mseas-data2 gdata]# dd
                                        if=/gdata/zero1 of=/gdata/zero2

                                        2048000+0 records in

                                        2048000+0 records out

                                        1048576000 bytes (1.0 GB)
                                        copied, 92.4904 s, 11.3 MB/s
                                        - realllyyy slooowww
                                        again

                                        
                                        Copy from /home to /home
                                          (ext4 to ext4)

                                        [root@mseas-data2 gdata]# dd
                                        if=/home/zero1 of=/home/zero2

                                        2048000+0 records in

                                        2048000+0 records out

                                        1048576000 bytes (1.0 GB)
                                        copied, 3.53263 s, 297 MB/s
                                        30 times as fast

                                        
                                        Copy from /home to /home
                                          (ext4 to ext4)

                                        [root@mseas-data2 gdata]# dd
                                        if=/home/zero1 of=/home/zero3

                                        2048000+0 records in

                                        2048000+0 records out

                                        1048576000 bytes (1.0 GB)
                                        copied, 4.1737 s, 251 MB/s
                                        - 30 times as fast

                                          
                                          As a test, can we copy data
                                          directly to the xfs mountpoint
                                          (/mnt/brick1) and bypass
                                          gluster?

                                          
                                          Any help you could give us
                                          would be appreciated.

                                          
                                        Thanks

                                      
                                      -- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley@xxxxxxx
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

      
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
    
    
_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

http://lists.gluster.org/mailman/listinfo/gluster-users


-- 
Pranith


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley@xxxxxxx
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301


-- 
Pranith


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley@xxxxxxx
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users