On 1/12/2016 12:32 PM, Sagi Grimberg wrote:
mlx5 keeps a lot of internal accounting for wr processing.
mlx5_ib_wq consists of multiple arrays:
struct mlx5_ib_wq {
u64 *wrid;
u32 *wr_data;
struct wr_list *w_list;
unsigned *wqe_head;
...
}
Each time we access each of these arrays, even for a single index
we fetch a cacheline. Reduce cacheline bounces by fitting these members
in a cacheline aligned struct (swr_ctx) and allocate an array. Accessing
this array will fetch all of these members in a single shot.
Since the receive queue needs only the wrid we use a nameless union
where in the rwr_ctx we only have wrid member.
Have some performance numbers before/after this patch to support the
proposed change?
Also, I have asked you the same question re the iser remote invalidation
series [1], this datais needed there too.
Or.
[1] http://marc.info/?l=linux-rdma&m=145188949807652&w=2
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -129,11 +129,24 @@ struct wr_list {
u16 next;
};
+/* Please don't let this exceed a single cacheline */
+struct swr_ctx {
+ u64 wrid;
+ u32 wr_data;
+ struct wr_list w_list;
+ u32 wqe_head;
+ u8 rsvd[12];
+}__packed;
what the role of the reserved field, is that for alignment purposes? if
yes, maybe better namewould be "align"
Nit, (same) checkpatch error here and below
ERROR: space required after that close brace '}'
#111: FILE: drivers/infiniband/hw/mlx5/mlx5_ib.h:139:
+}__packed;
+
+struct rwr_ctx {
+ u64 wrid;
+}__packed;
+
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
