On Mon, May 06, 2013 at 02:36:39PM +0200, Gandalf Corvotempesta wrote: > 2013/5/6 Andreas Friedrich <andreas.friedrich@xxxxxxxxxxxxxx>: > > To enable the LD_PRELOAD mechanism for the Ceph daemons only, a little > > generic extension in the global section of /etc/ceph/ceph.conf would > > be helpful, e.g.: > > > > [global] > > environment = LD_PRELOAD=/usr/lib64/libsdp.so.1 > > > > The appending patch adds 5 lines in the Bobtail (0.56.6) init script. > > The init script will then read the environment setting and - if present - > > call the Ceph daemons with the preceding environment string. > > Cool! We are planning the same infrastructure with IB as networking. > Could you share more details about this? > - which performance are you having and on which hardware? 6x Storage-Server, each with 3x Intel-910 PCIe-SSD (OSD-xfs), Journal in 1GB RAM-Disk per OSD plus 1x Client-server, each Server with 10GbE, Intel-QDR as well as MLX-FDR The performance values are still scattering ... (1 fio client job, 128 QD, 1TB of data, 60 seconds) e.g. - read_4m_128, read_8m_128, randread_4m_128, randread_8m_128 approx. 2.2 GB/s on 56 GbIPoIB_CM and 56 GbSDP - write_4m_128, write_8m_128, randwrite_4m_128, randwrite_8m_128 approx. 500 MB/s on 56 GbIPoIB_CM and 56 GbSDP, but 900 MS/s on 40 GbIPoIB_CM > - any issue with IB and SDP? Well, you have to compile the complete OFED stack (ib_core, ... ib_sdp) from one vendor. MLX is running, the Intel status is 'wip' ... the cluster is up, but jerking ... (OSDs are getting out/in ... timeouts) looks like a shortage on ressource ?! > - are you able to use the newer rsockets and not SDP? It also has a > preloader library and is still developed (SDP is deprecated) rsockets.so is up, but stuttering ... ... and for the moment rsockets is UserLand only, in contrast to sdp.ko Cheers, -Dieter > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
