On 2/2/2017 10:52 AM, Sagi Grimberg wrote:
This patchset from Leon adds direct access to mlx5 devices.
The libibverbs API is an abstract API. It is agnostic to any underlying
provider specific implementation. While this abstraction has the
advantage of
user applications portability it has a performance penalty. For some
applications optimizing performance is more important than portability.
The mlx5 direct verbs API introduced in this patchset is intended for
such
applications. It exposes mlx5 specific low level data path
(send/receive/completion) operations, allowing the application to
bypass the
libibverbs data path API.
Yishai and Leon this is good stuff !!
Thanks Sagi for pointing on, it's really appreciated.
Couple of comments:
- The user of this is pretty intimate with the driver, which makes a
somewhat strange combo of having both ibv_ and mlx5dv_ API. I
understand that it can inflate the the implementation, but perhaps
for completeness we can have also the control plane API as well (even
as a trampoline to the generic equivalent where it's useless).
The rational here is that extended features that needs allocation
flags can be contained to mlx5 and not pollute/inflate the generic
verbs implementation.
This series opens the door for optimizing the data path flows by
exposing the internal driver stuff that is required in the general data
path flow. It enables an application to optimize its own specific flow
without considering the general case which involves branches that are
not relevant to its use case.
Re control path:
Extending control path for driver specific configuration is really
something that exists in our plans. It may be achieved by extending the
above direct channel in a second phase with some extra APIs or by
introducing some vendor channel mechanism to enable that on top of
current control verbs. I believe that triggering that will be some
feature request or a specific need that will come from the
field/customers that will require this functionality.
- This brings me to the next step of this. What is still missing is
how we allow the user to have more control and information about how
his db records (namely UARs) are allocated
Are you referring to some specific information that exists in mlx5
driver which was not exposed by the direct API ? please have a look in
dv.h for mlx5dv_init_obj and in mlx5 driver code and let us know if we
missed something useful, we can add it.
Please consider that the above API was introduced with some extension
mechanism (i.e. comp_mask) so that we can extend it in the future
without breaking compatibility with applications that will use it.
. How does the user know if he needs to protect the DB write?
Once an application uses the direct flow the locking scheme is fully
managed by itself. We don't expect mixed flows in the data path for both
direct/non-direct access by same application.
> Is it BF?
This information is based on mlx5dv_qp->bf->size, in case of 0 it's not
a BF. Please note that as part of V1 we plan to introduce a man page for
mlx5 direct verbs which may point on that, in addition to other notes
that I pointed on above.
How can the user ask for more UARs than originally allocated to it by the kernel?
The user of this can really use an API that allows more control
of this area.
This is some functionality that currently doesn't exist even in the
*non* direct mode. It should be considered as some feature request
regardless the above series. Please note that moving forward with this
series and concept opens the door to give such an API which is some
vendor specific more quickly and easily.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
