Re: Mysql performance on CephFS vs RBD

[Babu Shanmugam <babu@xxxxxxxx>]

My intention is to just identify the root cause for the too much time spent on a "table create" operation on CephFS. I am *not* trying to benchmark with my testing. Sorry if it wasn't clear in my mail.

I am sure the time spent would be lesser if I had a proper CEPH setup. But, I believe even then, the "Table create" operation takes more time in CephFS than RBD. Please correct me if I am wrong.

On Monday 01 May 2017 08:58 PM, Scottix wrote:

I'm by no means a Ceph expert but I feel this is not a fair representation of Ceph, I am not saying numbers would be better or worse. Just the fact I see some major holes that don't represent a typical Ceph setup.

1 Mon? Most have a minimum of 3

1 OSD? basically all your reads and writes are going to 1 HDD? (I would say biggest flaw in the benchmark setup)

Is everything on a VM? worse, is it on 1 machine?

What is your network setup?

Why are you testing CephFS and RBD on an older kernel?

Why did you compile from source?

Journal And Data on same disk, is it a spinning drive, SSD, or other? (We need way more specs to understand)

I would suggest if you want to benchmark you need to get actual hardware to represent what you would do in production, to try to maximize the performance of this type of test. Otherwise these numbers are basically meaningless.

On Sun, Apr 30, 2017 at 9:57 PM Babu Shanmugam <babu@xxxxxxxx> wrote:

On Monday 01 May 2017 10:24 AM, David Turner wrote:

You don't have results that include the added network latency of having replica 3 replicating across multiple hosts. The reads would be very similar as the primary is the only thing that is read from, but writes will not return until after all 3 copies are written.

I started this as an experiment to see why table creation takes too much time on CephFS. That was my prime focus, David. So haven't tried it on pools with size > 1.

On Sat, Apr 29, 2017, 9:46 PM Babu Shanmugam <babu@xxxxxxxx> wrote:

Hi,
I did some basic experiments with mysql and measured the time taken by a set of operations on CephFS and RBD. The RBD measurements are taken on a 1GB RBD disk with ext4 filesystem. Following are my observation. The time listed below are in seconds.

	Plain file system	CephFS	RBD
Mysql install db	7.9	38.3	36.4
Create table	0.43	4.2	2.5
Drop table	0.14	0.21	0.40
Create table + 1000 recs	2.76	4.69	5.07
Create table + 10000 recs		7.69	11.96
Create table + 100K recs		12.06	29.65

From the above numbers, CephFS seems to fare very well while creating records whereas RBD does well while creating a table. I tried measuring the syscalls of ceph-osd, ceph-mds and the mysqld while creating a table on CephFS and RBD. Following is how the key syscalls of mysqld performed while creating a table (time includes wait time as well).

Syscalls of MYSQLD	CephFS	RBD
fsync	338.237 ms	183.697 ms
fdatasync	75.635 ms	96.359 ms
io_submit	50 us	151 us
open	2266 us	61 us
close	1186 us	33 us
write	115 us	51 us

From the above numbers, open, close and fsync syscalls take too much time on CephFs as compared to RBD.

Sysbench results are below;

Sysbence 100K records in 60 secs	CephFS	RBD
Read Queries performed	631876	501690
Other Queries performed	90268	71670
No. of transactions	45134	35835
No. of transactions per sec	752.04	597.17
R/W requests per sec	10528.55	8360.37
Other operations per sec	1504.08	1194.34

Above numbers seems to indicate the CephFS does very well with MYSQL transactions, better than RBD.


Following is my setup;

Num MONs    : 1
Num OSDs    : 1
Num MDSs    : 1
Disk              : 10 GB Qemu disk file (Both journal and data in the same disk)
Ceph version : 10.2.5 (Built from source)
Build config   : ./configure --without-debug --without-fuse --with-libaio \
          --without-libatomic-ops --without-hadoop --with-nss --without-cryptopp \
          --without-gtk2 --disable-static --with-jemalloc \
          --without-libzfs --without-lttng --without-babeltrace \
          --with-eventfd --with-python -without-kinetic --without-librocksdb \
          --without-openldap \
          CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -std=c++11 -fPIC

Ceph conf : Apart from host and network settings nothing else is configured
CephFS mount options: rw,relatime,name=cephfs,secret=<hidden>,acl
RBD mount options: rw,relatime,stripe=1024,data="">
All the processes were run in a Qemu virtual machine with Linux 4.4.18 kernel

Searching for "Mysql on CephFS" in google does not give any useful results. If this kind of experiments had been done previously and shared publicly, kindly share a link to it.

If you are aware of anything that I can do to optimise this, kindly let me know. I am willing to continue this experiment to see how well we can optimise CephFs for mysql.



Thank you,
Babu Shanmugam
www.aalam.io

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com