On 5/4/2018 12:46 AM, Rohit Zambre wrote:
On Thu, May 3, 2018 at 3:15 PM, Alex Rosenbaum <rosenbaumalex@xxxxxxxxx> wrote:
On Thu, May 3, 2018 at 6:19 PM, Rohit Zambre <rzambre@xxxxxxx> wrote:
An independent communication path is one that shares no hardware resources
with other communication paths. From a Verbs perspective, an independent
path is the one obtained by the first QP in a context. The next QPs of the
context may or may not share hardware resources amongst themselves; the
mapping of the resources to the QPs is provider-specific. Sharing resources
can hurt throughput in certain cases. When only one thread uses the
independent path, we term it an uncontended independent path.
Today, the user has no way to request for an independent path for an
arbitrary QP within a context. To create multiple independent paths, the
Verbs user must create mulitple contexts with 1 QP per context. However,
this translates to significant hardware-resource wastage: 89% in the case
of the ConnectX-4 mlx5 device.
This RFC patch allows the user to request for uncontended independent
communication paths in Verbs through an "independent" flag during Thread
Domain (TD) creation. The patch also provides a first-draft implementation
of uncontended independent paths in the mlx5 provider.
In mlx5, every even-odd pair of TDs share the same UAR page, which is not
case when the user creates multiple contexts with one TD per context. When
the user requests for an independent TD, the driver will dynamically
allocate a new UAR page and map bfreg_0 of that UAR to the TD. bfreg_1 of
the UAR belonging to an independent TD is never used and is essentially
wasted. Hence, there must be a maximum number of independent paths allowed
within a context since the hardware resources are limited. This would be
half of the maximum number of dynamic UARs allowed per context.
I'm not sure I follow what you're trying to achieve here on the mlx5 HW level.
Are you assuming that two threads with seperate 'indep-comm-paths'
using separate bfreg on the same UAR page causes some contention and
performance hit in the mlx5 HW?
We should first prove that's true, and then design a solution to solve it.
Do you have benchmark results of any kind?
Yes, there is a ~20% drop in message rates when there are concurrent
BlueFlame writes to separate bfregs on the same UAR page.
Can you please share your test code to help us make sure that you are
really referring to the above case with the below analysis ?
The graph attached reports message rates using rdma-core for 2-byte
RDMA-writes using 16 threads. Each thread is driving its own QP. Each
thread has its own CQ. Thread Domains are not used in this benchmark.
Can you try to use in your test TDs and see if you get the same results
before your initial patch ? this mode cleanly guarantees the 1<->1 UAR
bfreg to a QP.
The x-axis is the ratio of #QPs:#CTXs. For example, 2-way CTX-sharing
means there are 8 CTXs with 2 QPs each. "wBF" means "with BlueFlame"
and "woBF" means without (by setting MLX5_SHUT_UP_BF=1). "wPostlist2"
means the size of the linked-list of WQEs is 2.
Looking at your graph, the best results are wPostlist2-wBF, correct ?
but in that case we don't expect BF at all but DB as you wrote below.
Can you please clarify the test and the results that are represented here ?
"woPostlist" means
each thread is posting only 1 WQE per ibv_post_send. These numbers are
on a ConnectX-4 mlx5 device (on the Gomez machine of JLSE). The
numbers are the same on the ConnectX-4 device on the Thor cluster of
the HPC Advisory Council. The behavior with MOFED is the same with
slight differences in absolute numbers; the drop is ~15%.
The first drop in the green line is due to concurrent BlueFlame writes
on the same UAR page. The second drop is due to bfreg lock contention
between the 5th and the 16th QP. With a postlist size greater than 1,
rdma-core does only 64-bit DoorBells. Concurrent Doorbells don't hurt.
Concurrent BlueFlame writes do. What is exactly causing this, I am not
sure. But from some more experimenting, I think the answer lies in how
the NIC finds out whether to fetch the WQE from the BlueFlame buffer
or DMA-read it from memory. I wasn't able to find a "bit" that was set
during WQE preparation that tells the NIC where to read from. But it
could be something else entirely..
We are addressing the green line with this patch.
When you create two seperate ibv_context you will separate a lot more
then just the UAR pages on which the bfreg are mapped. Ehe entier
software locking scheme is separated.
Right. In the description, I wanted to emphasize the independent path
aspect of different contexts since that is most important to the MPI
library. The locking can be controlled through Thread Domains.
The ibv_td object allows the user to separate resources so that locks
could be managed in a smarter way in the provider lib data fast path.
For that we allocate a bfreg for each ibv_td obj. Using a dedicated
bfreg allows lower latency sends, as the doorbell does not need a lock
to write the even/odd entries.
At the time we did not extend the work to cover additional locks in
mlx5. but it seems your series is targeting something else.
If you are referring to [1], then that patch is targeting just to
disable QP-lock if a Thread Domain is specified. To create an
independent software path, the MPI library will use the Thread Domain.
[1] https://patchwork.kernel.org/patch/10367419/
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
