do you follow this instruction(https://community.mellanox.com/docs/DOC-2693)? On Mon, Aug 28, 2017 at 6:40 AM, Jeroen Oldenhof <jeroen@xxxxxx> wrote: > Hi All! > > I'm trying to run CEPH over RDMA, using a batch of Infiniband Mellanox > MT25408 20GBit (4x DDR) cards. > > RDMA is running, rping works between all hosts, and I've configured 10.0.0.x > addressing on the ib0 interfaces. > > But when enabling RMDA in ceph.conf: > > ms_type = async+rdma > ms_async_rdma_device_name = mlx4_0 > > OSD and MON on all hosts barf: > -5> 2017-08-28 12:03:37.623110 7f2326c9de00 1 Processor -- start > -4> 2017-08-28 12:03:37.624232 7f23205fe700 1 Infiniband binding_port > found active port 1 > -3> 2017-08-28 12:03:37.624250 7f23205fe700 1 Infiniband init > assigning: 1024 receive buffers > -2> 2017-08-28 12:03:37.624260 7f23205fe700 1 Infiniband init > assigning: 1024 send buffers > -1> 2017-08-28 12:03:37.624262 7f23205fe700 1 Infiniband init device > allow 4194303 completion entries > 0> 2017-08-28 12:03:37.628379 7f23205fe700 -1 > /build/ceph-12.1.4/src/msg/async/rdma/Infiniband.cc: In function 'int > Infiniband::MemoryManager::Cluster::fill(uint32_t)' thread 7f23205fe700 time > 2017-08-28 12:03:37.624433 > /build/ceph-12.1.4/src/msg/async/rdma/Infiniband.cc: 599: FAILED assert(m) > > > As suggested in the thread > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-June/018943.html > I also tried with lower values for receive and send buffer: > -4> 2017-08-28 12:36:46.997001 7fa28b270700 1 Infiniband binding_port > found active port 1 > -3> 2017-08-28 12:36:46.997026 7fa28b270700 1 Infiniband init > assigning: 256 receive buffers > -2> 2017-08-28 12:36:46.997029 7fa28b270700 1 Infiniband init > assigning: 256 send buffers > -1> 2017-08-28 12:36:46.997030 7fa28b270700 1 Infiniband init device > allow 4194303 completion entries > 0> 2017-08-28 12:36:47.001835 7fa28b270700 -1 > /build/ceph-12.1.4/src/msg/async/rdma/Infiniband.cc: In function 'int > Infiniband::MemoryManager::Cluster::fill(uint32_t)' thread 7fa28b270700 time > 2017-08-28 12:36:46.997231 > /build/ceph-12.1.4/src/msg/async/rdma/Infiniband.cc: 599: FAILED assert(m) > > > > > $ ceph -v > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc) > > In the osd logs I also see some of these: > -236> 2017-08-28 12:17:34.507315 7f7f4815ce00 -1 RDMAStack RDMAStack!!! > WARNING !!! For RDMA to work properly user memlock (ulimit -l) must be big > enough to allow large amount of registered memory. We recommend setting this > parameter to infinity > > but, the memlock has been set to infinity: > > $ ulimit -l > unlimited > > Any suggestions..? > > Best regards, > Jeroen Oldenhof > The Netherlands > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |