Fabian said: > The output of "ceph osd pool stats" shows ~100 op/s, but our disks are doing: What does the iostat output look like on the old cluster? Thanks, Mark On Mon, Feb 24, 2020 at 11:02 AM Fabian Zimmermann <dev.faz@xxxxxxxxx> wrote: > > Hi, > > we currently creating a new cluster. This cluster is (as far as we can > tell) an config-copy (ansible) of our existing cluster, just 5 years later > - with new hardware (nvme instead of ssd, bigger disks, ...) > > The setup: > > * NVMe for Journals and "Cache"-Pool > * HDD with NVMe Journals for "Data"-Pool > * Cache-Pool as writeback-Tier on Data-Pool > * We are using 12.2.13 without bluestore. > > If we run an rados benchmark against this pool, everything seems fine, but > as soon as we start a fio-benchmark > > -<- > [global] > ioengine=rbd > clientname=cinder > pool=cinder > rbdname=fio_test > rw=write > bs=4M > > [rbd_iodepth32] > iodepth=32 > ->- > > after some seconds the bandwidth drops to <15 MB/s and our hdd-disks are > doing more IOs than our Journal-Disks. > We also unconfigured the caching completely, but the issue remains. > > The output of "ceph osd pool stats" shows ~100 op/s, but our disks are > doing: > -<- > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > nvme0n1 0.00 0.00 0.00 278.50 0.00 34.07 250.51 > 0.14 0.50 0.00 0.50 0.03 0.80 > nvme1n1 0.00 0.00 0.00 64.00 0.00 7.77 248.50 > 0.01 0.22 0.00 0.22 0.03 0.20 > sda 0.00 1.50 0.00 557.00 0.00 29.49 108.45 > 180.57 160.59 0.00 160.59 1.80 100.00 > sdb 0.00 42.00 0.00 592.00 0.00 28.21 97.60 > 176.51 1105.79 0.00 1105.79 1.69 100.00 > sdc 0.00 14.50 0.00 528.50 0.00 27.95 108.31 > 183.02 179.47 0.00 179.47 1.89 100.00 > sde 0.00 134.50 0.00 223.50 0.00 14.05 128.72 > 17.38 60.05 0.00 60.05 0.89 20.00 > sdg 0.00 76.00 0.00 492.00 0.00 26.32 109.54 > 191.81 1474.96 0.00 1474.96 2.03 100.00 > sdf 0.00 0.00 0.00 491.50 0.00 26.76 111.49 > 176.55 326.05 0.00 326.05 2.03 100.00 > sdh 0.00 0.00 0.00 548.50 0.00 26.71 99.75 > 204.39 327.57 0.00 327.57 1.82 100.00 > sdi 0.00 112.00 0.00 526.00 0.00 23.15 90.14 > 158.32 1325.61 0.00 1325.61 1.90 100.00 > sdj 0.00 12.00 0.00 641.00 0.00 34.78 111.13 > 185.51 278.29 0.00 278.29 1.56 100.00 > sdk 0.00 23.50 0.00 399.50 0.00 20.38 104.46 > 166.77 461.67 0.00 461.67 2.50 100.00 > sdl 0.00 267.00 0.00 498.50 0.00 34.46 141.58 > 200.37 490.80 0.00 490.80 2.01 100.00 > ->- > > Any hints how to debug the issue? > > Thanks a lot, > > Fabian > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |