So I've had a chance to re-visit this since Bécholey Alexandre was kind enough to let me know how to compile Ceph with the RDMACM library (thankyou again!).
At this stage it compiles and runs but there appears to be a problem with calling rshutdown in Pipe as it seems to just wait forever for the pipe to close which causes commands like 'ceph osd tree' to hang indefinitely after they work successfully. Debug MS is here - http://pastebin.com/WzMJNKZY2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter closing
2013-08-09 15:08:31.382075 7f4643cfd700 10 accepter.accepter stopping
2013-08-09 15:08:31.382115 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopped accepter thread
2013-08-09 15:08:31.382127 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopping reaper thread
2013-08-09 15:08:31.382146 7f4645500700 10 -- 172.16.0.1:6789/0 reaper_entry done
2013-08-09 15:08:31.382182 7f464bd397c0 20 -- 172.16.0.1:6789/0 wait: stopped reaper thread
2013-08-09 15:08:31.382194 7f464bd397c0 10 -- 172.16.0.1:6789/0 wait: closing pipes
2013-08-09 15:08:31.382200 7f464bd397c0 10 -- 172.16.0.1:6789/0 reaper
2013-08-09 15:08:31.382205 7f464bd397c0 10 -- 172.16.0.1:6789/0 reaper done
2013-08-09 15:08:31.382210 7f464bd397c0 10 -- 172.16.0.1:6789/0 wait: waiting for pipes 0x3014c80,0x3015180,0x3015400 to close
Thanks again
On Thu, Jun 20, 2013 at 5:09 PM, Matthew Anderson <manderson8787@xxxxxxxxx> wrote:
Hi All,I've had a few conversations on IRC about getting RDMA support into Ceph and thought I would give it a quick attempt to hopefully spur some interest. What I would like to accomplish is an RSockets only implementation so I'm able to use Ceph, RBD and QEMU at full speed over an Infiniband fabric.What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by replacing the regular socket calls with the rsocket equivalent. Unfortunately it doesn't compile and I get an error of -CXXLD ceph-osd./.libs/libglobal.a(libcommon_la-Accepter.o): In function `Accepter::stop()':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243: undefined reference to `rshutdown'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251: undefined reference to `rclose'./.libs/libglobal.a(libcommon_la-Accepter.o): In function `Accepter::entry()':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213: undefined reference to `raccept'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230: undefined reference to `rclose'./.libs/libglobal.a(libcommon_la-Accepter.o): In function `Accepter::bind(entity_addr_t const&, int, int)':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61: undefined reference to `rsocket'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80: undefined reference to `rsetsockopt'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: undefined reference to `rbind'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118: undefined reference to `rgetsockname'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128: undefined reference to `rlisten'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100: undefined reference to `rbind'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: undefined reference to `rbind'./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::tcp_write(char const*, int)':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined reference to `rsend'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined reference to `rshutdown'./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::do_sendmsg(msghdr*, int, bool)':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined reference to `rsendmsg'./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::tcp_read_nonblocking(char*, int)':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined reference to `rrecv'./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::tcp_read(char*, int)':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined reference to `rshutdown'./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined reference to `rclose'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined reference to `rsocket'/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined reference to `rconnect'./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()':/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined reference to `rwrite'collect2: error: ld returned 1 exit statusmake[3]: *** [ceph-mon] Error 1From the looks of it I need to include the 'rdma/rsocket.h' library somewhere else or add librdmacm but I'm not sure where.Full disclaimer, I am terrible at C++. If anyone has a few spare minutes to have a look into the error messages and can point out where I've gone wrong it would be greatly appreciated.I've put the code up at - https://github.com/funkBuild/ceph-rsocketsThanks again-Matt
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
