On 2/5/2016 11:57 AM, ira.weiny@xxxxxxxxx wrote: > From: Ira Weiny <ira.weiny@xxxxxxxxx> > > Doug, > > This was resubmitted to you as v2 and is the first series which all of our > submissions are based on. We detected that we had improperly used the BIT > macro in the exported hfi1_user.h header.[1] This resulted in us needing to > drop a patch and rework the next patch which adds a bit to the header. This > rework is self contained within this series. None of the other series' are > affected by this change. > > Thanks, > Ira > > [1] I'll blame checkpatch.pl --strict but it's still my fault... ;-) > > > > Expected receives work by user-space libraries (PSM) calling into the driver > with information about the user's receive buffer and have the driver DMA-map > that buffer and program the HFI to receive data directly into it. > > This is an expensive operation as it requires the driver to pin the pages which > the user's buffer maps to, DMA-map them, and then program the HFI. > > When the receive is complete, user-space libraries have to call into the driver > again so the buffer is removed from the HFI, un-mapped, and the pages unpinned. > > All of these operations are expensive, considering that a lot of applications > (especially micro-benchmarks) use the same buffer over and over. > > In order to get better performance for user-space applications, it is highly > beneficial that they don't continuously call into the driver to register and > unregister the same buffer. Rather, they can register the buffer and cache it > for future work. The buffer can be unregistered when it is freed by the user. > > This change implements such buffer caching by making use of the kernel's MMU > notifier API. User-space libraries call into the driver only when they need to > register a new buffer. > > Once a buffer is registered, it stays programmed into the HFI until the kernel > notifies the driver that the buffer has been freed by the user. At that time, > the user-space library is notified and it can do the necessary work to remove > the buffer from its cache. > > Buffers which have been invalidated by the kernel are not automatically removed > from the HFI and do not have their pages unpinned. Buffers are only completely > removed when the user-space libraries call into the driver to free them. This > is done to ensure that any ongoing transfers into that buffer are complete. > This is important when a buffer is not completely freed but rather it is > shrunk. The user-space library could still have uncompleted transfers into the > remaining buffer. > > With this feature, it is important that systems are setup with reasonable > limits for the amount of lockable memory. Keeping the limit at "unlimited" (as > we've done up to this point), may result in jobs being killed by the kernel's > OOM due to them taking up excessive amounts of memory. > > > --- > Changes from V2: > Remove patch (series is now 13): > uapi/rdma/hfi/hfi1_user.h: Convert definitions to use BIT() macro > because exported user headers can't use the BIT macro > Rework patch: > uapi/rdma/hfi/hfi1_user.h: Add command and event for TID caching > to not use the BIT macro > > > Mitko Haralanov (13): > staging/rdma/hfi1: Add function stubs for TID caching > uapi/rdma/hfi/hfi1_user.h: Correct comment for capability bit > uapi/rdma/hfi/hfi1_user.h: Add command and event for TID caching > staging/rdma/hfi1: Add definitions needed for TID caching support > staging/rdma/hfi1: Remove un-needed variable > staging/rdma/hfi1: Add definitions and support functions for TID > groups > staging/rdma/hfi1: Start adding building blocks for TID caching > staging/rdma/hfi1: Convert lock to mutex > staging/rdma/hfi1: Add Expected receive init and free functions > staging/rdma/hfi1: Add MMU notifier callback function > staging/rdma/hfi1: Add TID free/clear function bodies > staging/rdma/hfi1: Add TID entry program function body > staging/rdma/hfi1: Enable TID caching feature > > drivers/staging/rdma/hfi1/Kconfig | 1 + > drivers/staging/rdma/hfi1/Makefile | 2 +- > drivers/staging/rdma/hfi1/file_ops.c | 458 +---------- > drivers/staging/rdma/hfi1/hfi.h | 40 +- > drivers/staging/rdma/hfi1/init.c | 5 +- > drivers/staging/rdma/hfi1/trace.h | 132 ++-- > drivers/staging/rdma/hfi1/user_exp_rcv.c | 1208 ++++++++++++++++++++++++++++++ > drivers/staging/rdma/hfi1/user_exp_rcv.h | 8 + > drivers/staging/rdma/hfi1/user_pages.c | 14 - > include/uapi/rdma/hfi/hfi1_user.h | 14 +- > 10 files changed, 1373 insertions(+), 509 deletions(-) > create mode 100644 drivers/staging/rdma/hfi1/user_exp_rcv.c > Most of this looked fine. I had to retouch all the commit log subjects (too long), and fix a spelling issue in one, but otherwise I've pulled this into a branch for hfi1.
Attachment:
signature.asc
Description: OpenPGP digital signature
