Rafi, it totally makes sense to me that you need to pre-allocate i/o buffers that will be used by RDMA, and you don't want to constantly change (i.e. allocate and deallocate) these buffers. Since a remote RDMA controller can be reading and writing to them, we have to be very careful about deallocating in particular. So an "arena" of pre-registered RDMA buffers makes perfect sense. Am I understanding you correctly that io-cache translator is soaking up all the RDMA-related buffers? How important is io-cache translator to Gluster performance at this point? Given that FUSE caching is now enabled, it seems to me that io-cache translator would accomplish very little. Should we have it disabled by default? If so, would that solve your problem? So how do read-ahead translator and write-behind translator interact with RDMA buffering? -ben ----- Original Message ----- > From: "Mohammed Rafi K C" <rkavunga@xxxxxxxxxx> > To: gluster-devel@xxxxxxxxxxx > Sent: Tuesday, January 13, 2015 9:29:56 AM > Subject: Suggestion needed to make use of iobuf_pool as rdma buffer. > > Hi All, > > When using RDMA protocol, we need to register the buffer which is going > to send through rdma with rdma device. In fact, it is a costly > operation, and a performance killer if it happened in I/O path. So our > current plan is to register pre-allocated iobuf_arenas from iobuf_pool > with rdma when rdma is getting initialized. The problem comes when all > the iobufs are exhausted, then we need to dynamically allocate new > arenas from libglusterfs module. Since it is created in libglusterfs, we > can't make a call to rdma from libglusterfs. So we will force to > register each of the iobufs from the newly created arenas with rdma in > I/O path. If io-cache is turned on in client stack, then all the > pre-registred arenas will use by io-cache as cache buffer. so we have to > do the registration in rdma for each i/o call for every iobufs, > eventually we cannot make use of pre registered arenas. > > To address the issue, we have two approaches in mind, > > 1) Register each dynamically created buffers in iobuf by bringing > transport layer together with libglusterfs. > > 2) create a separate buffer for caching and offload the data from the > read response to the cache buffer in background. > > If we could make use of preregister memory for every rdma call, then we > will have approximately 20% increment for write and 25% of increment for > read. > > Please give your thoughts to address the issue. > > Thanks & Regards > Rafi KC > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-devel > _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
