Re: RBD journaling benchmarks

[Maged Mokhtar <mmokhtar@xxxxxxxxxxx>]

 

On 2017-07-10 18:14, Mohamad Gebai wrote:

Resending as my first try seems to have disappeared.

Hi,

We ran some benchmarks to assess the overhead caused by enabling
client-side RBD journaling in Luminous. The tests consists of:
- Create an image with journaling enabled  (--image-feature journaling)
- Run randread, randwrite and randrw workloads sequentially from a
single client using fio
- Collect IOPS

More info:
- Feature exclusive-lock is enabled with journaling (required)
- Queue depth of 128 for fio
- With 1 and 2 threads


Cluster 1
================

- 5 OSD nodes
- 6 OSDs per node
- 3 monitors
- All SSD
- Bluestore + WAL
- 10GbE NIC
- Ceph version 12.0.3-1380-g6984d41b5d
(6984d41b5d142ce157216b6e757bcb547da2c7d2) luminous (dev)


Results:

        Default        Journaling                Jour width 32      
Jobs    IOPS        IOPS        Slowdown        IOPS        Slowdown
RW
1        19521        9104       2.1x            16067        1.2x
2        30575        726       42.1x              488        62.6x
Read
1        22775        22946      0.9x            23601        0.9x
2        35955        1078      33.3x              446        80.2x
Write
1        18515        6054       3.0x             9765        1.9x
2        29586        1188      24.9x              534        55.4x

- "Default" is the baseline (with journaling disabled)
- "Journaling" is with journaling enabled
- "Jour width 32" is with a journal data width of 32 objects
(--journal-splay-width 32)
- The major slowdown for two jobs is due to locking
- With a journal width of 32, the 0.9x slowdown (which is actually a
speedup) is due to the read-only workload, which doesn't exercise the
journaling code.
- The randwrite workload exercises the journaling code the most, and is
expected to have the highest slowdown, which is 1.9x in this case.


Cluster 2
================

- 3 OSD nodes
- 10 OSDs per node
- 1 monitor
- All HDD
- Filestore
- 10GbE NIC
- Ceph version 12.1.0-289-g117b171715
(117b1717154e1236b2d37c405a86a9444cf7871d) luminous (dev)


Results:

        Default        Journaling                 Jour width 32      
Jobs      IOPS        IOPS     Slowdown          IOPS       Slowdown
RW                                      
1        11869        3674        3.2x           4914          2.4x
2        13127         736       17.8x            432         30.4x
Read                                      
1        14500       14700        1.0x          14703          1.0x
2        16673        3893        4.3x            307         54.3x
Write                                      
1         8267        1925        4.3x           2591          3.2x
2         8283        1012        8.2x            417         19.9x

- The number of IOPS for the write workload is quite low, which is due
to HDDs and filestore

Mohamad

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

These are significant differences, to the point where it may not make sense to use rbd journaling / mirroring unless there is only 1 active client. Could there be in the future enhancement that will try to make active/active possible ? Would it help if each active writer maintained their own queue and only lock for a sequence number / counter to try to minimize the lock overhead writing in the same journal queue  ?   

Maged

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com