Hi All,
We're testing the rbd cache setting for openstack(Ceph 14.2.5 Bluestore
3-replica), and an odd problem found:
1. Setting librbd cache
[client]
rbd cache = true
rbd cache size = 16777216
rbd cache max dirty = 12582912
rbd cache target dirty = 8388608
rbd cache max dirty age = 1
rbd cache writethrough until flush = true
2. Running rbd bench
rbd -c /etc/ceph/ceph.conf \
-k /etc/ceph/keyring2 \
-n client.rbd-openstack-002 bench \
--io-size 4K \
--io-threads 1 \
--io-pattern seq \
--io-type read \
--io-total 100G \
openstack-volumes/image-you-can-drop-me
3. Start another test
rbd -c /etc/ceph/ceph.conf \
-k /etc/ceph/keyring2 \
-n client.rbd-openstack-002 bench \
--io-size 4K \
--io-threads 1 \
--io-pattern rand \
--io-type write \
--io-total 100G \
openstack-volumes/image-you-can-drop-me2
Running for minutes, I found the read test almost hung for a while:
69 152069 2375.21 9728858.72
70 153627 2104.63 8620569.93
71 155748 1956.04 8011953.10
72 157665 1945.84 7970177.24
73 159661 1947.64 7977549.44
74 161522 1890.45 7743277.01
75 163583 1991.04 8155301.58
76 165791 2008.44 8226566.26
77 168433 2153.43 8820438.66
78 170269 2121.43 8689377.16
79 172511 2197.62 9001467.33
80 174845 2252.22 9225091.00
81 177089 2259.42 9254579.83
82 179675 2248.22 9208708.30
83 182053 2356.61 9652679.11
84 185087 2515.00 10301433.50
99 185345 550.16 2253434.96
101 185346 407.76 1670187.73
102 185348 282.44 1156878.38
103 185350 162.34 664931.53
104 185353 12.86 52681.27
105 185357 1.93 7916.89
106 185361 2.74 11235.38
107 185367 3.27 13379.95
108 185375 5.08 20794.43
109 185384 6.93 28365.91
110 185403 9.19 37650.06
111 185438 17.47 71544.17
128 185467 4.94 20243.53
129 185468 4.45 18210.82
131 185469 3.89 15928.44
132 185493 4.09 16764.16
133 185529 4.16 17037.21
134 185578 18.64 76329.67
135 185631 27.78 113768.65
Why this happened? It's a unacceptable performance for read.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
