Hi All,
I've had a few conversations on IRC about getting RDMA support into Ceph and thought I would give it a quick attempt to hopefully spur some interest. What I would like to accomplish is an RSockets only implementation so I'm able to use Ceph, RBD and QEMU at full speed over an Infiniband fabric.
What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by replacing the regular socket calls with the rsocket equivalent. Unfortunately it doesn't compile and I get an error of -
CXXLD ceph-osd
./.libs/libglobal.a(libcommon_la-Accepter.o): In function `Accepter::stop()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243: undefined reference to `rshutdown'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251: undefined reference to `rclose'
./.libs/libglobal.a(libcommon_la-Accepter.o): In function `Accepter::entry()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213: undefined reference to `raccept'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230: undefined reference to `rclose'
./.libs/libglobal.a(libcommon_la-Accepter.o): In function `Accepter::bind(entity_addr_t const&, int, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61: undefined reference to `rsocket'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80: undefined reference to `rsetsockopt'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: undefined reference to `rbind'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118: undefined reference to `rgetsockname'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128: undefined reference to `rlisten'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100: undefined reference to `rbind'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87: undefined reference to `rbind'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::tcp_write(char const*, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined reference to `rsend'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined reference to `rshutdown'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::do_sendmsg(msghdr*, int, bool)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined reference to `rsendmsg'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::tcp_read_nonblocking(char*, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined reference to `rrecv'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::tcp_read(char*, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined reference to `rshutdown'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined reference to `rclose'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined reference to `rsocket'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined reference to `rconnect'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined reference to `rwrite'
collect2: error: ld returned 1 exit status
make[3]: *** [ceph-mon] Error 1
From the looks of it I need to include the 'rdma/rsocket.h' library somewhere else or add librdmacm but I'm not sure where.
Full disclaimer, I am terrible at C++. If anyone has a few spare minutes to have a look into the error messages and can point out where I've gone wrong it would be greatly appreciated.
I've put the code up at - https://github.com/funkBuild/ceph-rsockets
Thanks again
-Matt
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
