Dear Cephers,
I did a simple test to understand the performance loss of ceph. Here's my environment:
CPU: 2 * Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
Memory: 4 * 8G 1067 MHz
NIC: 2 * Intel Corporation 10-Gigabit X540-AT2
HDD:
1 * WDC WD1003FZEX ATA Disk 1TB
4 * Seagate ST2000NM0011 ATA Disk 2TB
SSD:
1 * INTEL SSDSC2BB800G4 800GB
# TEST 1: 4K Randwrite Testing
The first testing is to build a ceph cluster using only 1 HDD.
Firstly, I did a basic performance on one of my HDD disk: Seagate ST2000NM0011 ATA Disk 2TB. For 4k randwrite(iodepth=1), IOPS is around 350, and the latency distribution:
clat percentiles (usec):
| 1.00th=[ 756], 5.00th=[ 3312], 10.00th=[ 3728], 20.00th=[ 4192],
| 30.00th=[ 4448], 40.00th=[ 4768], 50.00th=[ 5024], 60.00th=[ 5280],
| 70.00th=[ 5600], 80.00th=[ 5920], 90.00th=[ 6816], 95.00th=[12352],
| 99.00th=[14016], 99.50th=[14656], 99.90th=[23168], 99.95th=[80384],
| 99.99th=[342016]
Then I tried to build up a ceph cluster using this HDD, Most of my ceph configurations are default. And I use fio rbd engine to test my ceph cluster, but the IOPS is only 83, and latency distribution became worse.
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 9], 20.00th=[ 9],
| 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 9],
| 70.00th=[ 9], 80.00th=[ 9], 90.00th=[ 16], 95.00th=[ 27],
| 99.00th=[ 99], 99.50th=[ 139], 99.90th=[ 215], 99.95th=[ 258],
| 99.99th=[ 347]
my fio command:
fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=16G -filename=/var/lib/ceph/osd/ceph-0/fio.test -name='4k randwrite test' -iodepth=1 -runtime=60 --numjobs 2 --group_reporting
fio -ioengine=rbd -pool=volumes -rbdname=testimg -bs=4k -direct=1 -thread -rw=randwrite -size=16g -directory=/tmp -name='randwrite-4k-d1-n1' -iodepth=1 -runtime=180 --numjobs 1 --group_reporting
As my understanding, the IO first do a fsync() to the journal and make sure all IO is written into the disk, then do a write() operation on the OSD, this maybe fast because of the page cache. So I guess the bottleneck is at the journal process, but why the performance lost more than half performance of the hard drive? Someone told me this is caused by the lock mechanism in journal, Is any document to explain this problem?
# TEST 2: 4K Randwrite Testing
Like the test above, I did a rand read test on the cluster. My IOPS of hard drive is 196 when iodepth=1. I clean the xfs cache using umount and mount. After that the performance of ceph is only 80. So the same question as above, why this happen?
Please correct me if any of my description is wrong. Thanks a lot.
Best Regards
-- Ray
-- Ray
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
