Re: RDMA connectivity not available with GlusterFS 3.5.2

Mohammed Rafi K C <rkavunga@xxxxxxxxxx> · Wed, 08 Oct 2014 12:55:31 +0530



    On 10/08/2014 11:54 AM, Geoffrey
      Letessier wrote:

    
      Hello,
      

      First of all, thanks to spend some times to help me to fix my
        issues.
      

      Previously, some weeks ago, i’ve already installed a
        GlusterFS patched version (http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs-3.5/epel-6-x86_64/glusterfs-3.5.20140922.da1657d-1.autobuild/ )
        to fix quota daemon (and libc) issues thanks to Krutika
        Dhananjay. 
    
    
      Do this patched version you offer me is compatible with the
        previously said one? If yes, is it possible to get RPM packages?
      

       I hope it will be compatible with the quota patch. But we need to
    figure out the availability of rpm packages for 3.5,  hopefully i
    can let you know the availability of  RPM packages within 1 day.

    
      Currently we have around 42TB used disk space (per replica);
        so is there any risk to apply this patched version on a
        production environment?
    
    
        Currently rdma is a tech preview , which is aimed to release in
    next versions. Once you applied the patch, all the communication
    will go through rdma instead of tcp. I hope there will not be any
    major risks. 

    
      In addition, concerning my global IO performances, do you
        feel my settings as optimal?
    
    
       I'm not sure about IO performance and settings of your volume,
    perhaps some one from this group can help you about that. 

    
    Rafi KC

    
      Thanks in advance,
      Geoffrey
      

            Le 8 oct. 2014 à 06:58, Mohammed Rafi K C <rkavunga@xxxxxxxxxx>
              a écrit :
            

                On 10/08/2014 04:15 AM,
                  Geoffrey Letessier wrote:

                
                  Dears,
                  

                  I have a HPC cluster composed by 4 storage nodes
                    (8x 24TB RAID6 bricks, 2 per nodes) and 62 compute
                    nodes, interconnected via Infiniband QDR technology.
                  

                  NB: each brick provide around 1.2-1.5TBs write
                    performances.
                   
                  My main volume is defined as below
                  
                    Volume Name:
                        vol_home
                    Type:
                        Distributed-Replicate
                    Volume ID:
                        f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6
                    Status: Started
                    Number of
                        Bricks: 4 x 2 = 8
                    Transport-type:
                        tcp,rdma
                    Bricks:
                    Brick1:
                        ib-storage1:/export/brick_home/brick1
                    Brick2:
                        ib-storage2:/export/brick_home/brick1
                    Brick3:
                        ib-storage3:/export/brick_home/brick1
                    Brick4:
                        ib-storage4:/export/brick_home/brick1
                    Brick5:
                        ib-storage1:/export/brick_home/brick2
                    Brick6:
                        ib-storage2:/export/brick_home/brick2
                    Brick7:
                        ib-storage3:/export/brick_home/brick2
                    Brick8:
                        ib-storage4:/export/brick_home/brick2
                    Options
                        Reconfigured:
                    features.quota:
                        on
                    diagnostics.brick-log-level:

                        CRITICAL
                    auth.allow:
                        localhost,127.0.0.1,10.*
                    nfs.disable: on
                    performance.cache-size:
                        64MB
                    performance.write-behind-window-size:

                        1MB
                    performance.quick-read:
                        on
                    performance.io-cache:
                        on
                    performance.io-thread-count:
                        64
                    features.default-soft-limit:
                        90%
                    

                    But, in the cluster, when I try to mount my
                      volume specifying RDMA transport type, i notice
                      all my communication go through TCP stack (all
                      network packet are visible on ib0 network
                      interface with ifstat shell command), not through
                      RDMA
                    
                      [root@lucifer
                          ~]# mount -t glusterfs -o
                          transport=rdma,direct-io-mode=disable
                          localhost:vol_home /home
                    
                    
                      [root@lucifer ~]# mount|grep
                          vol_home.rdma
                      localhost:vol_home.rdma on /home type
                          fuse.glusterfs
                          (rw,default_permissions,allow_other,max_read=131072)
                      
                        [root@lucifer ~]#
                            ifstat -i ib0
                               ib0        
                         KB/s in  KB/s out
                        25313.60   6776.44
                        26258.96   9064.92
                        28272.97  10034.15
                        23495.09   8504.84
                        21842.41   7161.69
                        ^C
                      
                    
                    So, my best noticed throughput is around
                      400MBs, but basically around 200-250MBs, although
                      I can read on the net i can expect to achieve
                      around 800-900MBs -sometimes more- with RDMA
                      transport type.
                    

                    Can anyone help me to make it work?
                  
                
                    There is known issue in rdma that volume with
                transport type as tcp,rdma will mount as tcp. the fix
                for the same is under review. You can pull the patch
                from  git fetch https://review.gluster.org/glusterfs
                refs/changes/98/8498/7 && git format-patch -1 
                FETCH_HEAD. Applying the patch will help to mount the
                tcp,rdma volume as rdma. If you are mounting a tcp,rdma
                volume as RDMA fuse mount you can also append  .rdma

                    with volname
                    instead of -o option.

                    
                      Let me know your result, If possible I would like
                    to know the version of gluster you are currently
                    using. 

                    Rafi KC.

                
                    In addition, are my volume settings look like
                      optimal?
                    

                    Thanks in advance,
                    Geoffrey
                  
                  
                  _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
                
                
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users