|
I have a 3 nodes, 15 osds ceph cluster setup:
* 15 7200 RPM SATA disks, 5 for each node. * 10G network * Intel(R) Xeon(R) CPU E5-2620(6 cores) 2.00GHz, for each node. * 64G Ram for each node. I deployed the cluster with ceph-deploy, and created a new data pool for cephfs. Both the data and metadata pools are set with replica size 3. Then mounted the cephfs on one of the three nodes, and tested the performance with fio. The sequential read performance looks good: fio -direct=1 -iodepth 1 -thread -rw=read -ioengine=libaio -bs=16K -size=1G -numjobs=16 -group_reporting -name=mytest -runtime 60 read : io=10630MB, bw=181389KB/s, iops=11336 , runt= 60012msec But the sequential write/random read/random write performance is very poor: fio -direct=1 -iodepth 1 -thread -rw=write -ioengine=libaio -bs=16K -size=256M -numjobs=16 -group_reporting -name=mytest -runtime 60 write: io=397280KB, bw=6618.2KB/s, iops=413 , runt= 60029msec fio -direct=1 -iodepth 1 -thread -rw=randread -ioengine=libaio -bs=16K -size=256M -numjobs=16 -group_reporting -name=mytest -runtime 60 read : io=665664KB, bw=11087KB/s, iops=692 , runt= 60041msec fio -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=libaio -bs=16K -size=256M -numjobs=16 -group_reporting -name=mytest -runtime 60 write: io=361056KB, bw=6001.1KB/s, iops=375 , runt= 60157msec I am mostly surprised by the seq write performance comparing to the raw sata disk performance(It can get 4127 IOPS when mounted with ext4). My cephfs only gets 1/10 performance of the raw disk. How can I tune my cluster to improve the sequential write/random read/random write performance? |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
