Hello All,
I can successfully send RDMA READ/WRITE,
but I can't get RDMA atomic operations to work.
I get an error when calling ibv_post_send function in the client,
and the errno will be set to "Invalid Arguments."
Below I pasted important parts of my code.
Could you please check my code and let me know if I'm missing anything?
*********** client side *****************:
-- Registering the memory regions --
mr = ibv_reg_mr(pd, buff, size, IBV_ACCESS_LOCAL_WRITE);
// and the size is 8
if (!mr){
fprintf(stderr, "Error, memory registration failed\n");
return -1;
}
-- Preparing RDMA ATOMIC FETCH AND
struct ibv_send_wr wr, *bad_wr = NULL;
struct ibv_sge sge;
memset(&sge, 0, sizeof(sge));
sge.addr = buff;
sge.length = 8;
sge.lkey = mr->lkey;
memset(&wr, 0, sizeof(wr));
wr.wr_id = 0;
wr.opcode = IBV_WR_ATOMIC_FETCH_AND_ADD;
wr.sg_list = &sge;
wr.num_sge = 1;
wr.send_flags = IBV_SEND_SIGNALED;
wr.wr.atomic.remote_addr = remote_buffer;
wr.wr.atomic.rkey = peer_mr->rkey;
wr.wr.atomic.compare_add = 1ULL;
if (ibv_post_send(qp, &wr, &bad_wr)) {
fprintf(stderr, "Error, ibv_post_send() failed\n");
return -1;
}
********* End of Client side *******
****** Server side ****************
-- Registering the memory regions --
mr = ibv_reg_mr(pd, rdma_region_timestamp_oracle,
sizeof(TimestampOracle),
IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE
| IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC ));
if (!mr){
fprintf(stderr, "Error, memory registration() failed\n");
return -1;
}
NOTE: TimestampOracle is a class with two int members,
so its size is 8 bytes (satisfies 64-bit condition for RDMA ATOMIC operations)
Thank you for your helps
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
