Hi All, We have recently deployed a new CEPH cluster Octopus 15.2.4 which consists of 12 OSD Nodes(16 Core + 200GB RAM, 30x14TB disks, CentOS 8) 3 Mon Nodes (8 Cores + 15GB, CentOS 8) We use Erasure Coded Pool and RBD block devices. 3 Ceph clients use the RBD devices, each has 25 RBDs and Each RBD size is 10TB. Each RBD is partitioned with the EXT4 file system. Cluster Health Is OK and Hardware is New and good. All the machines have 10Gbps (Active/Passive) bond Interface configured on it. Read operation of the cluster is OK, however, writes are very slow. One one of the RBDs we did the perf test. fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 -rw=randread -runtime=60 -filename=/dev/rbd40 Run status group 0 (all jobs): READ: bw=401MiB/s (420MB/s), 401MiB/s-401MiB/s (420MB/s-420MB/s), io=23.5GiB (25.2GB), run=60054-60054msec fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 -rw=randwrite -runtime=60 -filename=/dev/rbd40 Run status group 0 (all jobs): WRITE: bw=217KiB/s (222kB/s), 217KiB/s-217KiB/s (222kB/s-222kB/s), io=13.2MiB (13.9MB), run=62430-62430msec I see a High IO wait from the client. Any suggestions/pointers address this issue is really appreciated. Thanks and Regards, Athreya _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |