On Tue, Nov 10, 2020 at 1:52 PM athreyavc <athreyavc@xxxxxxxxx> wrote: > > Hi All, > > We have recently deployed a new CEPH cluster Octopus 15.2.4 which consists > of > > 12 OSD Nodes(16 Core + 200GB RAM, 30x14TB disks, CentOS 8) > 3 Mon Nodes (8 Cores + 15GB, CentOS 8) > > We use Erasure Coded Pool and RBD block devices. > > 3 Ceph clients use the RBD devices, each has 25 RBDs and Each RBD size is > 10TB. Each RBD is partitioned with the EXT4 file system. > > Cluster Health Is OK and Hardware is New and good. > > All the machines have 10Gbps (Active/Passive) bond Interface configured on > it. > > Read operation of the cluster is OK, however, writes are very slow. > > One one of the RBDs we did the perf test. > > fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 > -rw=randread -runtime=60 -filename=/dev/rbd40 > > Run status group 0 (all jobs): > READ: bw=401MiB/s (420MB/s), 401MiB/s-401MiB/s (420MB/s-420MB/s), > io=23.5GiB (25.2GB), run=60054-60054msec > > fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 > -rw=randwrite -runtime=60 -filename=/dev/rbd40 > > Run status group 0 (all jobs): > WRITE: bw=217KiB/s (222kB/s), 217KiB/s-217KiB/s (222kB/s-222kB/s), > io=13.2MiB (13.9MB), run=62430-62430msec > > I see a High IO wait from the client. > > Any suggestions/pointers address this issue is really appreciated. EC pools + small random writes + performance: pick two of the three. ;-) Writes against an EC pool require the chunk to be re-written via an expensive read/modify/write cycle. > Thanks and Regards, > > Athreya > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Jason _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |