To follow up on the previous discussion,
we changed the RFC proposal to focus on this single type for vendor QP.
The purpose of this RFC is to show a method by which a vendor can add a software
interface support for a QP type that is not described in the InfiniBand spec.
Generally speaking, a QP is an software object used by abstract software interface
to send and receive data to and from the hardware. The InfiniBand spec describes
several types of QPs, different from each other by the service that they provide.
Vendors that implement network services that are not described in the InfiniBand
spec may want to use the verbs interface to create, modify, query and destroy a QP
to maintain a software object that keeps the context of the hardware QP. However,
we don't want to taint the IB/core layer with code that is vendor specific and
leave all vendor logic to be done in the vendor driver. To do this we suggest a new
QP type - IB_QPT_VENDOR which is almost all the addition to IB/core. The following
describes the flow to create a vendor QP but it can be easily applied to modify query
and destroy operations.
A vendor that implements a new QP type will provide an interface to send a command to
the kernel to create a QP of this type. Of course that this is probably not enough
data to create the QP so the vendor will also provide in that interface a way to pass
extra data that is required to create the QP. This extra data will be passed through the
vendor channel which is not described here. The IB/core layer in the kernel may process
the command with respect to the new QP type but without assuming anything that is not
general to all vendor QPs. For example the IB/core layer may not validate the that the
receive CQ is different from the send CQ for vendor QP but it may assume that that zero
length send queue means that send CQ is not present.
This message will be followed by patches that implement the above and use the mlx5 driver
as an example vendor.
The following patches to kernel show the changes in code that are required to
- IB/core: [RFC] Introduce vendor QP type
- IB/mlx5: [RFC] Example use of vendor QP
The following patches to userspace show the changes in code
- verbs: [RFC] Intorduce QP type IBV_QPT_VENDOR
- mlx5: [RFC] Introduce mlx5dv_create_qp()
and an application that want to create a vendor QP (in this case the send part of the DC
transport service) will look like this
{
struct ibv_context *context;
struct ibv_pd *pd;
struct ibv_cq *cq;
struct ibv_qp *qp;
int qp_depth = 128;
struct ibv_qp_attr attr;
struct ibv_qp_init_attr_ex init_attr;
struct mlx5dv_qp_init_attr dv_init_attr;
context = ibv_open_device(ib_dev);
if (!context) {
fprintf(stderr, "Couldn't get context for %s\n",
ibv_get_device_name(ib_dev));
return -1;
}
pd = ibv_alloc_pd(context);
if (!pd) {
fprintf(stderr, "Couldn't allocate PD\n");
return -1;
}
cq = ibv_create_cq(context, qp_depth + 1, NULL, NULL, 0);
if (cq) {
fprintf(stderr, "Couldn't create CQ\n");
return -1;
}
init_attr.send_cq = cq;
init_attr.recv_cq = cq;
init_attr.cap.max_send_wr = 100;
init_attr.cap.max_send_sge = 1;
init_attr.qp_type = IBV_QPT_VENDOR;
init_attr.pd = pd;
dv_init_attr.vendor_qp_type = MLX5_VENDOR_QPT_DCI;
qp = mlx5dv_create_qp(context, &init_attr, &dv_init_attr);
if (qp) {
fprintf(stderr, "Couldn't create QP\n");
return -1;
}
fprintf(stdout, "Success: Create DC send QP\n");
}
Attachment:
0001-IB-core-RFC-Introduce-vendor-QP-type.patch
Description: 0001-IB-core-RFC-Introduce-vendor-QP-type.patch
Attachment:
0001-verbs-RFC-Intorduce-QP-type-IBV_QPT_VENDOR.patch
Description: 0001-verbs-RFC-Intorduce-QP-type-IBV_QPT_VENDOR.patch
Attachment:
0002-IB-mlx5-RFC-Example-use-of-vendor-QP.patch
Description: 0002-IB-mlx5-RFC-Example-use-of-vendor-QP.patch
Attachment:
0002-mlx5-RFC-Introduce-mlx5dv_create_qp.patch
Description: 0002-mlx5-RFC-Introduce-mlx5dv_create_qp.patch
Attachment:
0003-mlx5-examples-Create-a-DCI-QP.patch
Description: 0003-mlx5-examples-Create-a-DCI-QP.patch
Attachment:
0004-Export-mlx5dv_create_qp-from-libmlx5.patch
Description: 0004-Export-mlx5dv_create_qp-from-libmlx5.patch
