On Fri, 2016-07-22 at 08:28 -0400, Jason Dillaman wrote: > You aren't, by chance, sharing the same RBD image between multiple > VMs, are you? An order-of-magnitude performance degradation would not > be unexpected if you have multiple clients concurrently accessing the > same image with the "exclusive-lock" feature enabled on the image. No, though I did perform a live migration of the VM between the tests as well. But there is only one client of it. > 4000 IOPS for 4K random writes also sounds suspiciously high to me. Are the replica writes of the primary OSD async/parallel? /M > On Thu, Jul 21, 2016 at 7:32 PM, Martin Millnert <martin@xxxxxxxxxxx> wrote: > > Hi, > > > > I just upgraded from Infernalis to Jewel and see an approximate 10x > > latency increase. > > > > Quick facts: > > - 3x replicated pool > > - 4x 2x-"E5-2690 v3 @ 2.60GHz", 128GB RAM, 6x 1.6 TB Intel S3610 SSDs, > > - LSI3008 controller with up-to-date firmware and upstream driver, and > > up-to-date firmware on SSDs. > > - 40GbE (Mellanox, with up-to-date drivers & firmware) > > - CentOS 7.2 > > > > Physical checks out, both iperf3 for network and e.g. fio over all the > > SSDs. Not done much of Linux tuning yet; but irqbalanced does a pretty > > good job with pairing both NIC and HBA with their respective CPUs. > > > > In performance hunting mode, and today took the next logical step of > > upgrading from Infernalis to Jewel. > > > > Tester is remote KVM/Qemu/libvirt guest (openstack) CentOS 7 image with > > fio. The test scenario is 4K randomwrite, libaio, directIO, QD=1, > > runtime=900s, test-file-size=40GiB. > > > > Went from a picture of [1] to [2]. In [1], the guest saw 98.25% of the > > I/O complete within maximum 250 µsec (~4000 IOPS). This, [2], sees > > 98.95% of the IO at ~4 msec (actually ~300 IOPs). > > > > Between [1] and [2] (simple plots of FIO's E2E-latency metrics), the > > entire cluster including compute nodes code went from Infernalis to > > 10.2.2 > > > > What's going on here? > > > > I haven't tuned Ceph OSDs either in config or via Linux kernel at all > > yet; upgrade to Jewel came first. I haven't changed any OSD configs > > between [1] and [2] myself (only minimally before [1], 0 effort on > > performance tuning) , other than updated to Jewel tunables. But the > > difference is very drastic, wouldn't you say? > > > > Best, > > Martin > > [1] http://martin.millnert.se/ceph/pngs/guest-ceph-fio-bench/test08/ceph-fio-bench_lat.1.png > > [2] http://martin.millnert.se/ceph/pngs/guest-ceph-fio-bench/test10/ceph-fio-bench_lat.1.png > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |