On 5/21/2015 9:56 AM, Or Gerlitz wrote:
Hi Doug,
This patchset adds completion timestamping supports for verbs consumers.
Timestamping is used by applications in order to know when a WQE was
received/transmitted by the HW. The value is given is HCA hardware cycles,
but could be easily converted as the hardware's core clock frequecny is
available through extension of query device.
Moreover, we add an ability to read the HCA's current clock. This could be
useful on order to synchronize events to the wall clock.
This functionality is achieved by adding/extending the following verbs:
create_cq - create_cq is extended in order to allow passing creation flags
to the CQ creation function. We change IB/core --> vendors API
to be easily extendible by passing a struct which contains
comp_vectors, cqe and the new flags parameter. In order to create
CQ which supports timestamping, IB_CQ_FLAGS_TIMESTAMP should be given.
query_device - We extend query_device uverb further by giving the hardware's
clock frequency and the timestamp mask (the number of timestamp
bits which are supported). If timestamp isn't supported, 0 is returned.
In order to read the timestamp in the WQE, the user needs to query the device
for support, create an appropriate CQ (using the extanded uverb with
IB_CQ_FLAGS_TIMESTAMP) and poll the CQ with an extended poll_cq verb (currently,
only implemented in user-space).
In mlx4, allowing the user to read the core clock efficiently involves mapping
this area of the hardware to user-space (being done by using a mmap command)
and reading the clock from the correct offset of the page.
This offset is returned in the vendor's specific data from mlx4's kernel driver
to the mlx4's user-space driver. query_device is modified in order to support
passing this vendor specific data. A user-space application could use a new
verb in order to read the hardware's clock.
Translating the hardware's clock into ms could be done by dividing this
value by hca_core_clock (which is returned by the extended version of
query_device uverb).
A user-space application could get the current HW's clock by executing
ibv_query_values_ex(struct ibv_context *context, uint32_t q_values,
struct ibv_values_ex *values)
The function gets a mask of the values to query and return their values.
Vendors could either implement this as a uverb command or use their
user-space driver to return those values directly from the HW (the mlx4 way).
I'm just reviewing this now, and I haven't looked at the user side, but
it appears the CQE timestamp is available for all devices to support or
not in a generic manner. IE it is part of the extended CQ UAPI. But
the task of fetching the current timestamp value/mask seems to be
device-specific. Shouldn't that also be a standard operation that
devices can choose to support or not? IE an application can
generically setup a CQ and get timestamps on CQEs, but it seems the
application would have to have device-specific code to get the current
timestamp/mask. Perhaps I'm not understanding the design?
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
