On 2022/9/24 6:22, Jason Gunthorpe wrote:
On Fri, Jul 08, 2022 at 04:02:36AM +0000, yangx.jy@xxxxxxxxxxx wrote:
+static enum resp_states atomic_write_reply(struct rxe_qp *qp,
+ struct rxe_pkt_info *pkt)
+{
+ u64 src, *dst;
+ struct resp_res *res = qp->resp.res;
+ struct rxe_mr *mr = qp->resp.mr;
+ int payload = payload_size(pkt);
+
+ if (!res) {
+ res = rxe_prepare_res(qp, pkt, RXE_ATOMIC_WRITE_MASK);
+ qp->resp.res = res;
+ }
+
+ if (!res->replay) {
+#ifdef CONFIG_64BIT
+ memcpy(&src, payload_addr(pkt), payload);
+
+ dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, payload);
+ /* check vaddr is 8 bytes aligned. */
+ if (!dst || (uintptr_t)dst & 7)
+ return RESPST_ERR_MISALIGNED_ATOMIC;
+
+ /* Do atomic write after all prior operations have completed */
+ smp_store_release(dst, src);
Someone needs to fix iova_to_vaddr to do the missing kmap, we can't
just assume you can cast a u64 pfn to a vaddr like this.
Hi Jason,
Cc Ira,
When using PMEM with DAX mode(devdax or fsdax), we cannot ensure that
iova_to_vaddr() can cast a u64 pfn to a vaddr, right? so we have to
replace page_address() with kmap_local_page().
Without Ira's PKS patch set, I didn't see any failure when accessing
remote PMEM with devdax mode by RDMA based on RXE. I don't know why I
cannot trigger any failure in the condition.
+ /* decrease resp.resid to zero */
+ qp->resp.resid -= sizeof(payload);
+
+ qp->resp.msn++;
+
+ /* next expected psn, read handles this separately */
+ qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
+ qp->resp.ack_psn = qp->resp.psn;
+
+ qp->resp.opcode = pkt->opcode;
+ qp->resp.status = IB_WC_SUCCESS;
+
+ return RESPST_ACKNOWLEDGE;
+#else
+ pr_err("32-bit arch doesn't support 8-byte atomic write\n");
+ return RESPST_ERR_UNSUPPORTED_OPCODE;
No print on receiving a remote packet
OK
Best Regards,
Xiao Yang
Jason
