On 1/22/2020 3:54 AM, Weihang Li wrote:
On 2020/1/17 3:51, Jason Gunthorpe wrote:
What happens to your userspace if it runs on an old kernel and tries
to use extended atomic?
Jason
Hi Jason,
If the hns userspace runs with old kernel, the hardware will report a asynchronous
event for the extended atomic operation and modify the qp to error state because
the enable bit in this qp's context hasn't been set.
The driver will print like this:
[ 1252.240921] hns3 0000:7d:00.0: Invalid request local work queue 0x9 error.
[ 1252.247772] hns3 0000:7d:00.0: no hr_qp can be found!
Ideally the provider will not set
IBV_PCI_ATOMIC_OPERATION_4_BYTE_SIZE_SUP and related without kernel
support..
I've applied this patch, but I feel like you may need a followup to
fix the capability reporting?
Jason
Hi Jason,
Thank for your suggestions.
But I'm confuse about the relationship between "PCI ATOMIC" in this macro
and atomic operations in RDMA.
PCI Atomics are optonal and are a much more recent facility.
RDMA Atomics do not require PCI Atomics, because they have
different semantics with respect to memory atomicity. Read
carefully and you'll see that they operate atomically only
within the adapter, and are not atomic all the way to the
underlying memory. It's a long and somewhat historical story.
Now that PCIe Atomics are becoming more widely supported by
processor complexes, there is the possibility these may be
more tightly embraced by RDMA implementations. There is
discussion in IBTA and IETF around this currently, in fact,
for the new RDMA Atomic Write operation.
Be aware that PCI Atomics are relatively expensive operations.
The existing ones perform a read-modify-write cycle on both
PCI and memory busses. This overhead is not to be taken lightly.
Tom.
I found the related series on patchwork:
https://patchwork.kernel.org/cover/10782873/
And I found the description about atomic operations in PCIe specification
v4.0:
"An Atomic Operation (AtomicOp) is a single PCI Express transaction that
targets a location in Memory Space, reads the location’s value, potentially
writes a new value back to the location, and returns the original value. This
"read-modify-write" sequence to the location is performed atomically."
It seems that the atomic for PCIe and RDMA is different concepts, and the macro
IBV_PCI_ATOMIC_OPERATION_4_BYTE_SIZE_SUP is for PCIe atomic.
Could you please give me more suggestions about them?
Thanks
Weihang
