On 2020/5/18 2:58, Tom Talpey wrote: > On 5/17/2020 9:14 AM, Leon Romanovsky wrote: >> On Fri, May 15, 2020 at 09:40:26AM +0000, liweihang wrote: >>> On 2020/5/12 19:35, Leon Romanovsky wrote: >>>> On Mon, May 11, 2020 at 01:54:48PM +0000, liweihang wrote: >>>>> Hi All, >>>>> >>>>> I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd): >>>>> >>>>> 1. The kernel now supports masked atomic, but the it does not support atomic >>>>> operation. Is the masked atomic valid in kernel currently? >>>> >>>> Yes, it is valid, but probably has a very little real value for the kernel ULPs. >>>> I see code in the RDS that uses atomics, but it says nothing to me, because >>>> upstream RDS and version in-real-use are completely different. >>>> >>>>> 2. In the userspace, ofed does not have the corresponding opcode for the masked >>>>> atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD), >>>>> and ibv_send_wr also has no related data segment for it. How to support it in >>>>> userspace? >>>> >>>> ibv_send_wr is not extensible, so the real solution will need to extend ibv_wr_post() [1] >>>> with specific and new post builders. >>>> >>>> Thanks >>>> >>>> [1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_wr_post.3.md >>>> >>> >>> Hi Leon, >>> >>> Thanks for your response. May I ask another question: >>> >>> Why it's not encouraged to use atomic/extended atomic/masked atomic operations in kernel? >>> Jason said that there seems no kernel users of extended atomic, is there any other reasons? >> >> I don't think that "it is not encouraged", the more accurate will be >> "the IBTA atomics will give nothing to the kernel ULPs". >> >> The atomic data is not necessary stored in the host memory, while ULPs >> need it in the memory. It means that they anyway will need to do some >> synchronization in the host and "cancel" any advantage of atomics if >> they exist. > > Indeed, it is a common misconception by upper layer implementers that > the atomicity is available to the responder CPU. In fact, atomics work > only from the HCA that executes them, and the result is flushed to > memory, non-atomically, at some later time. These limitations greatly > reduce the motivation to use them at all, much less the exotic masked > ones. > > I believe another reason they're not surfaced for kernel consumers is > that there aren't any. Primarily, the kernel consumers are storage, and > storage protocols stay far away from atomics. > > Tom. > Hi Tom and Leon, Thank you for the explanation, it helps me lot. Weihang
