I did a Ceph cluster install 2 weeks ago where I was getting great performance (~= PanFS) where I could write 100,000 1MB files in 61 Mins (Took PanFS 59 Mins). I thought I could increase the performance by adding a better MDS server so I redid the entire build. Now it takes 4 times as long to write the same data as it did before. The only thing that changed was the MDS server. (I even tried moving the MDS back on the old slower node and the performance was the same.) The first install was on CentOS 7. I tried going down to CentOS 6.6 and it's the same results. I use the same scripts to install the OSDs (which I created because I can never get ceph-deploy to behave correctly. Although, I did use ceph-deploy to create the MDS and MON and initial cluster creation.) I use btrfs on the OSDS as I can get 734 MB/s write and 1100 MB/s read with rados bench -p cephfs_data 500 write --no-cleanup && rados bench -p cephfs_data 500 seq (xfs was 734 MB/s write but only 200 MB/s read) Could anybody think of a reason as to why I am now getting a huge regression. Hardware Setup: [OSDs] 64 GB 2133 MHz Dual Proc E5-2630 v3 @ 2.40GHz (16 Cores) 40Gb Mellanox NIC [MDS/MON new] 128 GB 2133 MHz Dual Proc E5-2650 v3 @ 2.30GHz (20 Cores) 40Gb Mellanox NIC [MDS/MON old] 32 GB 800 MHz Dual Proc E5472 @ 3.00GHz (8 Cores) 10Gb Intel NIC _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |