Could you share some of your testing methodology? I'd like to repeat your tests.
I have a cluster that is currently running mostly 3.13 kernels, but the latest patch of that version breaks the onboard 1Gb NIC in the servers I'm using. I recently had to redeploy several of these servers due to SSD failures, and so those ones I built around the 3.19 vivid kernel. I'm planning on upgrading all the machines to that kernel and Hammer (from Giant) in about a week.
I had planned on performing before and after testing on the cluster anyway, but if I could replicate your they would serve as an even better data point.
QH
On Tue, Oct 6, 2015 at 9:23 AM, Mark Nelson <mnelson@xxxxxxxxxx> wrote:
Hi,On 10/06/2015 10:14 AM, MailingLists - EWS wrote:
I have encountered a rather interesting issue with Ubuntu 14.04 LTS
running 3.19.0-30 kernel (Vivid) using Ceph Hammer (0.94.3).
With everything else identical in our testing cluster, no other changes
other than the kernel (apt-get install linux-image-generic-lts-vivid and
then a reboot), we are seeing a substantial drop in read performance as
indicated with rados bench.
Kernel 3.13.0.-65 (latest for Trusty): Read performance: 575MB/s
Kernel 3.19.0-30 (Vivid): Read performance: 175MB/s
Has anyone experienced this sort of performance drop?
Very interesting! Did you upgrade the kernel on both the OSDs and clients or just some of them? I remember there were some kernel performance regressions a little while back. You might try running perf during your tests and look for differences. Also, iperf might be worth trying to see if it's a network regression.
I also have a script that compares output from sysctl which might be worth trying to see if any defaults changes.
https://github.com/ceph/cbt/blob/master/tools/compare_sysctl.py
basically just save systctl -a with both kernels and pass them as arguments to the python script.
Mark
_______________________________________________
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
