Mysql performance on CephFS vs RBD

Babu Shanmugam <babu@xxxxxxxx> · Sun, 30 Apr 2017 07:16:11 +0530



    Hi,

      I did some basic experiments with mysql and measured the time
      taken by a set of operations on CephFS and RBD. The RBD
      measurements are taken on a 1GB RBD disk with ext4 filesystem.
      Following are my observation. The time listed below are in
      seconds.

      
          Plain file system
          CephFS
          RBD
        
        
          Mysql install db
          7.9
          38.3
          36.4
        
        
          Create table
          0.43
          4.2
          2.5
        
        
          Drop table 
          0.14
          0.21
          0.40
        
        
          Create table + 1000 recs
          2.76
          4.69
          5.07
        
        
          Create table + 10000 recs
          

          7.69
          11.96
        
        
          Create table + 100K recs
          

          12.06
          29.65
        
      
    From the above numbers, CephFS seems to fare very
      well while creating records whereas RBD does well while creating a
      table. I tried measuring the syscalls of ceph-osd, ceph-mds and
      the mysqld while creating a table on CephFS and RBD. Following is
      how the key syscalls of mysqld performed while creating a table
      (time includes wait time as well).

      
          Syscalls of MYSQLD
          CephFS
          RBD
        
        
          fsync
          338.237 ms
          183.697 ms
        
        
          fdatasync
          75.635 ms
          96.359 ms
        
        
          io_submit
          50 us
          151 us
        
        
          open
          2266 us
          61 us
        
        
          close
          1186 us
          33 us
        
        
          write
          115 us
          51 us
        
      
    From the above numbers, open, close and fsync
      syscalls take too much time on CephFs as compared to RBD.

      
      Sysbench results are below;

      
            Sysbence 100K records in
                60 secs
            CephFS
            RBD
          
          
            Read Queries performed
            631876
            501690
          
          
            Other Queries performed
            90268
            71670
          
          
            No. of transactions
            45134
            35835
          
          
            No. of transactions per sec
            752.04
            597.17
          
          
            R/W requests per sec
            10528.55
            8360.37
          
          
            Other operations per sec
            1504.08
            1194.34
          
        
      Above numbers seems to indicate the CephFS does very well with
      MYSQL transactions, better than RBD.

      
      Following is my setup;

      
      Num MONs    : 1

      Num OSDs    : 1

      Num MDSs    : 1

      Disk              : 10 GB Qemu disk
      file (Both journal and data in the same disk)

      Ceph version : 10.2.5 (Built from source)

      Build config   : ./configure
      --without-debug --without-fuse --with-libaio \

                --without-libatomic-ops --without-hadoop --with-nss
      --without-cryptopp \

                --without-gtk2 --disable-static --with-jemalloc \

                --without-libzfs --without-lttng --without-babeltrace \

                --with-eventfd --with-python -without-kinetic
      --without-librocksdb \

                --without-openldap \

                CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -std=c++11 -fPIC

      
      Ceph conf : Apart from host and
      network settings nothing else is configured

      CephFS mount options:
      rw,relatime,name=cephfs,secret=<hidden>,acl

      RBD mount options:
      rw,relatime,stripe=1024,data="">
      

      All the processes were run in a Qemu virtual machine with Linux
      4.4.18 kernel

      
      Searching for "Mysql on CephFS" in google does not give any useful
      results. If this kind of experiments had been done previously and
      shared publicly, kindly share a link to it. 

      
      If you are aware of anything that I can do to optimise this,
      kindly let me know. I am willing to continue this experiment to see
      how well we can optimise CephFs for mysql.

      
      Thank you,

      Babu Shanmugam

      www.aalam.io
    
    
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com