Hi Bart, +AD4- -----Original Message----- +AD4- From: Bart Van Assche +AD4- Sent: Friday, March 1, 2019 12:17 PM +AD4- To: Parav Pandit +ADs- linux-rdma+AEA-vger.kernel.org +AD4- Subject: Re: +AFs-EXPERIMENTAL v1 2/4+AF0- RDMA/loopback: Add helper lib for +AD4- resources and cqe fifo +AD4- +AD4- On Wed, 2019-02-27 at 00:27 -0600, Parav Pandit wrote: +AD4- +AD4- +-+ACM-include +ADw-linux/completion.h+AD4- +AD4- +AD4- +-struct loopback+AF8-resource +AHs- +AD4- +AD4- +- struct completion completion+ADs- +AD4- +AD4- +- /+ACo- wait for all datapath references to drop, we don't want to do +AD4- +AD4- +- +ACo- large memcpy while holding rcu, refcount doesn't hurt the +AD4- +AD4- +- +ACo- performance. +AD4- +AD4- +- +ACo-/ +AD4- +AD4- +- refcount+AF8-t refcount+ADs- +AD4- +AD4- Please consider struct kref instead of open-coding the kref implementation. +AD4- In the current approach QP's refcount drops to zero, when in modify+AF8-qp state so if incoming request arrives for this QP under state change, get+AF8-table+AF8-entry+AF8-for+AF8-id() returns an error in datapath, this error results into retry+AF8-exceeded+AF8-error. We don't want to free the QP struct at this point in modify+AF8-qp(). Do not wanted to hold the QP's spinlock either while doing memcpy(). Also there are two QPs involved whose state can change, and both QPs can be working on send/recv. So spin locking QP on each side will results into A-B, B-A locking sequence into deadlock. So current design does with refcount of QP and modify+AF8-qp waits until post+AF8-sends() are done. So refcount is not really keeping the life of an object (for which kref are better choice) from memory perspective. Any alternative lock less ideas would you suggest? +AD4- +AD4- +-/+ACo- TODO: replace with kernel pfifo, which consumes more memory +ACo-/ +AD4- +AD4- +-struct loopback+AF8-fifo +AHs- +AD4- +AD4- +- /+ACo- Protect insert, remove to list +ACo-/ +AD4- +AD4- +- spinlock+AF8-t lock+ADs- +AD4- +AD4- +- /+ACo- head of entries, FIFO order +ACo-/ +AD4- +AD4- +- struct list+AF8-head list+ADs- +AD4- +AD4- +- u32 entries+ADs- +AD4- +AD4- +-+AH0AOw- +AD4- +AD4- How do pfifo and kfifo compare and which of the two is the most +AD4- appropriate for this driver? +AD4- I didn't recall at time of writing the TODO, what was the actual API, but knew something I used in past that is available for this exact need. include/linux/kfifo.h should be used with write lock due to sender and reciver and srq sharing the cq.
