ceph-fuse and its memory usage

Goncalo Borges <goncalo@xxxxxxxxxxxxxxxxxxx> · Fri, 2 Oct 2015 11:42:26 +1000



    Dear CephFS Gurus...

    
    I have a question regarding ceph-fuse and its memory usage.

    
    1./ My Ceph and CephFS setups are the following:

    Ceph:

      a. ceph 9.0.3

      b. 32 OSDs distributed in 4 servers (8 OSD per server).

      c. 'osd pool default size = 3' and 'osd pool default min size = 2'

      d. All servers running Centos6.7

      
      CephFS:

      e. a single mds

      f. dedicated pools for data and metadata

      g. clients in different locations / sites mounting CephFS via
      ceph-fuse

      h. All servers and clients running Centos6.7

      
    2./ I have been running fio tests in two CephFS clients:

        - Client A is in the same data center as all OSDs connected at 1
    GbE

        - Client B is in a different data center (in another city) also
    connected at 1 GbE. However, I've seen that the connection is
    problematic, and sometimes, the network performance is well bellow
    the theoretical 1 Gbps limit.

        - Client A has 24 GB RAM + 98 GB of SWAP and client B has 48 GB
    of RAM + 98 GB of SWAP 

    
    3./ I have been running some fio write tests (with 128 threads) in
    both clients, and surprisingly, the results show that the aggregated
    throughput is better for client B than client A.  

    CLIENT A results:

      # grep agg
      fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015558.out
      

      WRITE: io=1024.0GB, aggrb=114878KB/s, minb=897KB/s,
      maxb=1785KB/s, mint=4697347msec, maxt=9346754msec

      
      CLIENT B results:

      #  grep agg
      fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015555.out
      

      WRITE: io=1024.0GB, aggrb=483254KB/s, minb=3775KB/s,
      maxb=3782KB/s, mint=2217808msec, maxt=2221896msec

      
    4./ If I actually monitor the memory usage of ceph-fuse during the
    I/O tests, I see that

    CLIENT A: ceph-fuse does not seem to go behond 7GB of
      VMEM and 1 GB of RMEM.

      CLIENT B: ceph-fuse uses 11 GB of VMEM and 7 GB of RMEM.

      
    5./ These results make me think that caching is playing a critical
    role in these results. 

    
    My questions are the following:

    a./ Why CLIENT B uses more memory than CLIENT A? My hint
      is that there is a network bottleneck between CLIENT B and the
      Ceph Cluster, and memory is more used because of that.

      b/ Is the FIO write performance better in CLIENT B a consequence
      of the fact that it is using more memory than client A?

      c./ Is there a parameters we can set for the CEPHFS clients to
      limit the amount of memory they can use?

    
    Cheers

    Goncalo

    
    -- 
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW  2006
T: +61 2 93511937

  
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com