Resending as my first try seems to have disappeared.
Hi,
We ran some benchmarks to assess the overhead caused by enabling
client-side RBD journaling in Luminous. The tests consists of:
- Create an image with journaling enabled (--image-feature journaling)
- Run randread, randwrite and randrw workloads sequentially from a
single client using fio
- Collect IOPS
More info:
- Feature exclusive-lock is enabled with journaling (required)
- Queue depth of 128 for fio
- With 1 and 2 threads
Cluster 1
================
- 5 OSD nodes
- 6 OSDs per node
- 3 monitors
- All SSD
- Bluestore + WAL
- 10GbE NIC
- Ceph version 12.0.3-1380-g6984d41b5d
(6984d41b5d142ce157216b6e757bcb547da2c7d2) luminous (dev)
Results:
Default Journaling Jour width 32
Jobs IOPS IOPS Slowdown IOPS Slowdown
RW
1 19521 9104 2.1x 16067 1.2x
2 30575 726 42.1x 488 62.6x
Read
1 22775 22946 0.9x 23601 0.9x
2 35955 1078 33.3x 446 80.2x
Write
1 18515 6054 3.0x 9765 1.9x
2 29586 1188 24.9x 534 55.4x
- "Default" is the baseline (with journaling disabled)
- "Journaling" is with journaling enabled
- "Jour width 32" is with a journal data width of 32 objects
(--journal-splay-width 32)
- The major slowdown for two jobs is due to locking
- With a journal width of 32, the 0.9x slowdown (which is actually a
speedup) is due to the read-only workload, which doesn't exercise the
journaling code.
- The randwrite workload exercises the journaling code the most, and is
expected to have the highest slowdown, which is 1.9x in this case.
Cluster 2
================
- 3 OSD nodes
- 10 OSDs per node
- 1 monitor
- All HDD
- Filestore
- 10GbE NIC
- Ceph version 12.1.0-289-g117b171715
(117b1717154e1236b2d37c405a86a9444cf7871d) luminous (dev)
Results:
Default Journaling Jour width 32
Jobs IOPS IOPS Slowdown IOPS Slowdown
RW
1 11869 3674 3.2x 4914 2.4x
2 13127 736 17.8x 432 30.4x
Read
1 14500 14700 1.0x 14703 1.0x
2 16673 3893 4.3x 307 54.3x
Write
1 8267 1925 4.3x 2591 3.2x
2 8283 1012 8.2x 417 19.9x
- The number of IOPS for the write workload is quite low, which is due
to HDDs and filestore
Mohamad
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
