hello I got a huge difference of read/write performance on my ceph cluster when I use rbd. rbd bench reach the limit of my cluster (1gbps network) when the performance in mapped rbd is very low. 30MB/s. ceph version : ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) kernel version : 4.19.0-5-amd64 ceph -s cluster: id: fdf6eba0-cde9-401c-8901-751eb1e8b7d2 health: HEALTH_OK services: mon: 3 daemons, quorum gra3-k8s001,sbg2-k8s001,rbx4-k8s001 (age 41h) mgr: sbg2-k8s001(active, since 46h), standbys: gra3-k8s001, rbx4-k8s001 osd: 10 osds: 10 up (since 41h), 10 in (since 41h) rgw: 3 daemons active (gra3-k8s001, rbx4-k8s001, sbg2-k8s001) data: pools: 8 pools, 240 pgs objects: 109.14k objects, 229 GiB usage: 694 GiB used, 3.0 TiB / 3.7 TiB avail pgs: 240 active+clean here the bench rbd create --size=10G kube/bench rbd feature disable kube/bench object-map fast-diff deep-flatten rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seq elapsed: 110 ops: 1310720 ops/sec: 11884.39 bytes/sec: 97356926.64 this reach the 1gbps limit of my network. Looks good. let's test random write rbd -p kube bench kube/bench --io-type write --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand elapsed: 125 ops: 1310720 ops/sec: 10416.31 bytes/sec: 85330446.58 this reach the 1gbps limit of my network. Looks good. rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern seq elapsed: 40 ops: 1310720 ops/sec: 32712.03 bytes/sec: 267976974.35 this is great perf ! let's try rand rbd -p kube bench kube/bench --io-type read --io-size 8192 --io-threads 256 --io-total 10G --io-pattern rand elapsed: 29 ops: 1310720 ops/sec: 43859.78 bytes/sec: 359299337.92 this is great perf ! But when I mount the image : rbd map bench --pool kube --name client.admin /sbin/mkfs.ext4 /dev/rbd/kube/bench mount /dev/rbd/kube/bench /mnt/ cd /mnt/ dd if=/dev/zero of=test bs=8192k count=100 oflag=direct 838860800 bytes (839 MB, 800 MiB) copied, 24.6185 s, 34.1 MB/s 34.1MB/s vs 85MB/s ....
|
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
