On Wed, 2010-10-20 at 10:04 -0600, Sage Weil wrote: > On Wed, 20 Oct 2010, Jim Schutt wrote: > > Hi, > > > > On Wed, 2010-10-20 at 09:34 -0600, Sage Weil wrote: > > > > > > > I think I can propose a patch which solves this problem without > > > rados > > > > client-specific data, threads, tids, maps, thread-local data, or > > > > changes to SimpleMessenger. > > > > > > SimpleMessenger will change no matter what, because the core problem > > > that we're trying to solve is that it allocates a new buffer when a > > > message is being read off the wire. See > > > SimpleMessenger::read_message(). > > > > FWIW, this buffer allocation is also an issue for trying to > > implement an RDMA-based messenger. It would be nice to have > > a pool of buffers to use that are already registered for RDMA, > > to reduce the memory registration overhead. > > That much is easy enough if it doesn't matter which buffer you use for a > particular message. If the result of a specific read needs to be read > into a specific buffer, then something like the mechanism we're talking > about will be needed. Do you know if RDMA will require that? No, I don't think that will be required. > > That's what the kernel client does, incidentally, except the callback is > to find/allocate the whole message struct, not just the data payload > portion. We could do that here as well, although the memory reservation > requirements on the userspace side aren't as critical as in the kernel. I think it will be OK to have the data payload buffer separate: for RDMA we'd need to receive the header information into a buffer pre-posted on the receive queue. It will contain a remote key we'd use when queuing the RDMA read operation for the payload. -- Jim > > sage > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
