Hi, I have get a problem about rbd cache and xfs Write barrier. Evn =============== Ceph 9.2.1 Qemu 2.3 Rbd cache: true Kvm disk cache mode: write back Guest os: centos 7.1 Guest os kernel: 4.1.12 Ssd pool: 12 * intel dc s3700 400GB Sas pool: 70 * 4TB sas Symptoms =============== 1. Fio 4k performance test only ~3.5K iops(ssd pool) 2. When running zookeeper benchmark on a kvm vm, iostat only ~300iops on vm [root@test-centos7-rbd ~]# iostat -dxm 1 Linux 4.1.12-32.2.3.el7.centosbfd.x86_64 (test-centos7-rbd.novalocal) 04/12/2016 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.13 3.18 0.97 0.17 0.01 88.01 0.01 2.38 2.25 2.79 1.05 0.44 vdb 0.00 0.11 0.94 544.03 0.04 1.58 6.08 0.13 0.24 1.85 0.23 0.23 12.42 vdc 0.00 0.00 0.18 0.00 0.00 0.00 15.39 0.00 0.12 0.12 0.00 0.11 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 0.00 248.00 0.00 0.62 5.10 0.94 3.76 0.00 3.76 3.77 93.60 vdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 0.00 261.00 0.00 0.65 5.10 0.94 3.62 0.00 3.62 3.59 93.70 vdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3. Zookeeper benchmark CREATE/DELETE ops only ~150 ops/s(It’s very low) 4. When I run command: mount -o "rw,nodev,noatime,nodiratime,nobarrier" /dev/vdb /opt/, the Fio test/zk benchmark test will be perfect good. [root@test-centos7-rbd ~]# iostat -dxm 1 Linux 4.1.12-32.2.3.el7.centosbfd.x86_64 (test-centos7-rbd.novalocal) 04/12/2016 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.12 3.03 0.95 0.16 0.01 87.56 0.01 2.37 2.25 2.76 1.05 0.42 vdb 0.00 0.11 0.89 529.98 0.04 1.53 6.06 0.17 0.32 1.85 0.32 0.31 16.40 vdc 0.00 0.00 0.17 0.00 0.00 0.00 15.39 0.00 0.12 0.12 0.00 0.11 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 0.00 260.00 0.00 0.65 5.09 0.94 3.60 0.00 3.60 3.59 93.40 vdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 0.00 267.00 0.00 0.66 5.08 0.93 3.47 0.00 3.47 3.47 92.60 vdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ^C [root@test-centos7-rbd ~]# mount -o remount,rw,nodev,noatime,nodiratime,barrier /dev/vdb /opt/ [root@test-centos7-rbd ~]# mount -o remount,rw,nodev,noatime,nodiratime,nobarrier /dev/vdb /opt/ [root@test-centos7-rbd ~]# iostat -dxm 1 Linux 4.1.12-32.2.3.el7.centosbfd.x86_64 (test-centos7-rbd.novalocal) 04/12/2016 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.12 3.01 0.94 0.16 0.01 87.53 0.01 2.37 2.25 2.76 1.05 0.42 vdb 0.00 0.11 0.89 531.90 0.04 1.54 6.06 0.17 0.33 1.85 0.32 0.32 16.84 vdc 0.00 0.00 0.17 0.00 0.00 0.00 15.39 0.00 0.12 0.12 0.00 0.11 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 0.00 2392.00 0.00 6.56 5.62 0.50 0.21 0.00 0.21 0.21 49.10 vdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vdb 0.00 0.00 0.00 2605.00 0.00 7.35 5.78 0.49 0.19 0.00 0.19 0.18 47.50 vdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ============================= Based on my observations, xfs Write barrier affect the iops performance. I know nobarrier is definitely *NOT* recommended on vm，but if not, the iops is so bad. So, please give me a hand. Very appreciated to get your reply! Alan Zhang -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
