On 06/20/2013 10:09 AM, Matthew Anderson wrote:
Hi All,
I've had a few conversations on IRC about getting RDMA support into Ceph
and thought I would give it a quick attempt to hopefully spur some
interest. What I would like to accomplish is an RSockets only
implementation so I'm able to use Ceph, RBD and QEMU at full speed over
an Infiniband fabric.
What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by
replacing the regular socket calls with the rsocket equivalent.
Unfortunately it doesn't compile and I get an error of -
CXXLD ceph-osd
./.libs/libglobal.a(libcommon_la-Accepter.o): In function
`Accepter::stop()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243:
undefined reference to `rshutdown'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251:
undefined reference to `rclose'
./.libs/libglobal.a(libcommon_la-Accepter.o): In function
`Accepter::entry()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213:
undefined reference to `raccept'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230:
undefined reference to `rclose'
./.libs/libglobal.a(libcommon_la-Accepter.o): In function
`Accepter::bind(entity_addr_t const&, int, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61:
undefined reference to `rsocket'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80:
undefined reference to `rsetsockopt'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:
undefined reference to `rbind'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118:
undefined reference to `rgetsockname'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128:
undefined reference to `rlisten'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100:
undefined reference to `rbind'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:
undefined reference to `rbind'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function
`Pipe::tcp_write(char const*, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined
reference to `rsend'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined
reference to `rshutdown'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function
`Pipe::do_sendmsg(msghdr*, int, bool)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined
reference to `rsendmsg'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function
`Pipe::tcp_read_nonblocking(char*, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined
reference to `rrecv'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function
`Pipe::tcp_read(char*, int)':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined
reference to `rshutdown'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined
reference to `rclose'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined
reference to `rsocket'
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined
reference to `rconnect'
./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()':
/home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined
reference to `rwrite'
collect2: error: ld returned 1 exit status
make[3]: *** [ceph-mon] Error 1
From the looks of it I need to include the 'rdma/rsocket.h' library
somewhere else or add librdmacm but I'm not sure where.
Full disclaimer, I am terrible at C++. If anyone has a few spare minutes
to have a look into the error messages and can point out where I've gone
wrong it would be greatly appreciated.
I am quite bad myself when it comes to Makefile and related foo, but I
believe you should add -lrdmacm (assuming lib's name is librdmacm) to
src/Makefile.am, under 'libcommon_la_LDFLAGS':
libcommon_la_CXXFLAGS= ${AM_CXXFLAGS}
- libcommon_la_LDFLAGS = -lrt
+ libcommon_la_LDFLAGS = -lrt -lrdmacm
noinst_LTLIBRARIES += libcommon.la
Given the messenger classes are being packed into libcommon (which is
then linked to by the daemons), I'm fairly certain this would be the
right place.
When this works, it should also find its way into the configure.* files;
how it should be done however is beyond me.
Hope this helps!
-Joao
I've put the code up at - https://github.com/funkBuild/ceph-rsockets
Thanks again
-Matt
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
