Hello, just forward this here, there is a big performance drop with kernel 4.4 compare to 4.2 on Jewel. Does anyone here have seen that before ? Best Regards, Yoann Moulin -------- Message transféré -------- Sujet : Re: [ceph-users] performance issue with jewel on ubuntu xenial (kernel) Date : Fri, 1 Jul 2016 16:37:41 +0200 De : Yoann Moulin <yoann.moulin@xxxxxxx> Hello, >>>>>>> I found a performance drop between kernel 3.13.0-88 (default kernel on Ubuntu >>>>>>> Trusty 14.04) and kernel 4.4.0.24.14 (default kernel on Ubuntu Xenial 16.04) >>>>>>> >>>>>>> ceph version is Jewel (10.2.2). >>>>>>> All tests have been done under Ubuntu 14.04 >>>>>> >>>>>> Knowing that you also have an internalis cluster on almost identical >>>>>> hardware, can you please let the list know whether you see the same >>>>>> behavior (severely reduced throughput on a 4.4 kernel, vs. 3.13) on >>>>>> that cluster as well? >>>>> >>>>> ceph version is infernalis (9.2.0) >>>>> >>>>> Ceph osd Benchmark: >>>>> >>>>> Kernel 3.13.0-88-generic : ceph tell osd.ID => average ~84MB/s >>>>> Kernel 4.2.0-38-generic : ceph tell osd.ID => average ~90MB/s >>>>> Kernel 4.4.0-24-generic : ceph tell osd.ID => average ~75MB/s >>>>> >>>>> The slow down is not as much as I have with Jewel but it is still present. >>>> >>>> But this is not on precisely identical hardware, is it? >>> >>> All the benchmarks were run on strictly identical hardware setups per node. >>> Clusters differ slightly in sizes (infernalis vs jewel) but nodes and OSDs are identical. >> >> One thing differ in the osd configuration, on the Jewel cluster, we have journal >> on disk, on the Infernalis cluster, we have journal on SSD (S3500) >> >> I can restart my test on a Jewel cluster with journal on SSD if needed. >> I can do as well a test on an Infernalis cluster with journal on disk. > > I'd suggest that the second option is probably more meaningful to test. I did new benchmarks on 3 clusters. Each cluster has 3 nodes strictly identical. Each node has 10 OSDs. Journals are on the disk. bench5 : Ubuntu 14.04 / Ceph Infernalis bench6 : Ubuntu 14.04 / Ceph Jewel bench7 : Ubuntu 16.04 / Ceph jewel this is the average of 2 runs of "ceph tell osd.* bench" on each cluster (2 x 30 OSDs) bench5 / 14.04 / Infernalis / kernel 3.13 : 54.35 MB/s bench6 / 14.04 / Jewel / kernel 3.13 : 86.47 MB/s bench5 / 14.04 / Infernalis / kernel 4.2 : 63.38 MB/s bench6 / 14.04 / Jewel / kernel 4.2 : 107.75 MB/s bench7 / 16.04 / Jewel / kernel 4.2 : 101.54 MB/s bench5 / 14.04 / Infernalis / kernel 4.4 : 53.61 MB/s bench6 / 14.04 / Jewel / kernel 4.4 : 65.82 MB/s bench7 / 16.04 / Jewel / kernel 4.4 : 61.57 MB/s If needed, I have the raw output of "ceph tell osd.* bench" > What I find curious is that no-one else on the list has apparently run > into this. Any Ubuntu xenial users out there, or perhaps folks on > trusty who choose to install linux-image-generic-lts-xenial? Anyone to try on their side if they have the same behaviour ? Cheers, -- Yoann Moulin EPFL IC-IT -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
