Hi Radoslav,
Is RBD cache enabled and in writeback mode? Do you have client side
readahead?
Both are doing better for writes than you'd expect from the native
performance of the disks assuming they are typical 7200RPM drives and
you are using 3X replication (~150IOPS * 27 / 3 = ~1350 IOPS). Given
the small file size, I'd expect that you might be getting better journal
coalescing in filestore.
Sadly I imagine you can't do a comparison test at this point, but I'd be
curious how it would look if you used libaio with a high iodepth and a
much bigger partition to do random writes over.
Mark
On 11/14/2017 01:54 PM, Milanov, Radoslav Nikiforov wrote:
Hi
We have 3 node, 27 OSDs cluster running Luminous 12.2.1
In filestore configuration there are 3 SSDs used for journals of 9 OSDs
on each hosts (1 SSD has 3 journal paritions for 3 OSDs).
I’ve converted filestore to bluestore by wiping 1 host a time and
waiting for recovery. SSDs now contain block-db – again one SSD serving
3 OSDs.
Cluster is used as storage for Openstack.
Running fio on a VM in that Openstack reveals bluestore performance
almost twice slower than filestore.
fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=1G
--numjobs=2 --time_based --runtime=180 --group_reporting
fio --name fio_test_file --direct=1 --rw=randread --bs=4k --size=1G
--numjobs=2 --time_based --runtime=180 --group_reporting
Filestore
write: io=3511.9MB, bw=19978KB/s, iops=4994, runt=180001msec
write: io=3525.6MB, bw=20057KB/s, iops=5014, runt=180001msec
write: io=3554.1MB, bw=20222KB/s, iops=5055, runt=180016msec
read : io=1995.7MB, bw=11353KB/s, iops=2838, runt=180001msec
read : io=1824.5MB, bw=10379KB/s, iops=2594, runt=180001msec
read : io=1966.5MB, bw=11187KB/s, iops=2796, runt=180001msec
Bluestore
write: io=1621.2MB, bw=9222.3KB/s, iops=2305, runt=180002msec
write: io=1576.3MB, bw=8965.6KB/s, iops=2241, runt=180029msec
write: io=1531.9MB, bw=8714.3KB/s, iops=2178, runt=180001msec
read : io=1279.4MB, bw=7276.5KB/s, iops=1819, runt=180006msec
read : io=773824KB, bw=4298.9KB/s, iops=1074, runt=180010msec
read : io=1018.5MB, bw=5793.7KB/s, iops=1448, runt=180001msec
- Rado
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
