On Mon, May 7, 2018 at 11:52 AM, Rohit Zambre <rzambre@xxxxxxx> wrote: > On Mon, May 7, 2018 at 10:24 AM, Yishai Hadas > <yishaih@xxxxxxxxxxxxxxxxxx> wrote: >> On 5/7/2018 7:26 AM, Rohit Zambre wrote: >>> >>> On Sun, May 6, 2018 at 7:47 AM, Yishai Hadas <yishaih@xxxxxxxxxxxxxxxxxx> >>> wrote: >>>> >>>> On 5/4/2018 12:46 AM, Rohit Zambre wrote: >>>>> >>>>> >>>>> On Thu, May 3, 2018 at 3:15 PM, Alex Rosenbaum <rosenbaumalex@xxxxxxxxx> >>>>> wrote: >>>>>> >>>>>> >>>>>> On Thu, May 3, 2018 at 6:19 PM, Rohit Zambre <rzambre@xxxxxxx> wrote: >>>>>>> >>>>>>> >>>>>>> An independent communication path is one that shares no hardware >>>>>>> resources >>>>>>> with other communication paths. From a Verbs perspective, an >>>>>>> independent >>>>>>> path is the one obtained by the first QP in a context. The next QPs of >>>>>>> the >>>>>>> context may or may not share hardware resources amongst themselves; >>>>>>> the >>>>>>> mapping of the resources to the QPs is provider-specific. Sharing >>>>>>> resources >>>>>>> can hurt throughput in certain cases. When only one thread uses the >>>>>>> independent path, we term it an uncontended independent path. >>>>>>> >>>>>>> Today, the user has no way to request for an independent path for an >>>>>>> arbitrary QP within a context. To create multiple independent paths, >>>>>>> the >>>>>>> Verbs user must create mulitple contexts with 1 QP per context. >>>>>>> However, >>>>>>> this translates to significant hardware-resource wastage: 89% in the >>>>>>> case >>>>>>> of the ConnectX-4 mlx5 device. >>>>>>> >>>>>>> This RFC patch allows the user to request for uncontended independent >>>>>>> communication paths in Verbs through an "independent" flag during >>>>>>> Thread >>>>>>> Domain (TD) creation. The patch also provides a first-draft >>>>>>> implementation >>>>>>> of uncontended independent paths in the mlx5 provider. >>>>>>> >>>>>>> In mlx5, every even-odd pair of TDs share the same UAR page, which is >>>>>>> not >>>>>>> case when the user creates multiple contexts with one TD per context. >>>>>>> When >>>>>>> the user requests for an independent TD, the driver will dynamically >>>>>>> allocate a new UAR page and map bfreg_0 of that UAR to the TD. bfreg_1 >>>>>>> of >>>>>>> the UAR belonging to an independent TD is never used and is >>>>>>> essentially >>>>>>> wasted. Hence, there must be a maximum number of independent paths >>>>>>> allowed >>>>>>> within a context since the hardware resources are limited. This would >>>>>>> be >>>>>>> half of the maximum number of dynamic UARs allowed per context. >>>>>> >>>>>> >>>>>> >>>>>> I'm not sure I follow what you're trying to achieve here on the mlx5 HW >>>>>> level. >>>>>> Are you assuming that two threads with seperate 'indep-comm-paths' >>>>>> using separate bfreg on the same UAR page causes some contention and >>>>>> performance hit in the mlx5 HW? >>>>>> We should first prove that's true, and then design a solution to solve >>>>>> it. >>>>>> Do you have benchmark results of any kind? >>>>> >>>>> >>>>> >>>>> Yes, there is a ~20% drop in message rates when there are concurrent >>>>> BlueFlame writes to separate bfregs on the same UAR page. >>>> >>>> >>>> >>>> Can you please share your test code to help us make sure that you are >>>> really >>>> referring to the above case with the below analysis ? >>> >>> >>> I have attached my benchmark code. The critical path of interest is >>> lines 554-598. In the README, I have included an example of how to run >>> the benchmark. Let me know if you have any questions/concerns >>> regarding the benchmark code. >>> >>>>> The graph attached reports message rates using rdma-core for 2-byte >>>>> RDMA-writes using 16 threads. Each thread is driving its own QP. Each >>>>> thread has its own CQ. Thread Domains are not used in this benchmark. >>>> >>>> >>>> >>>> Can you try to use in your test TDs and see if you get the same results >>>> before your initial patch ? this mode cleanly guarantees the 1<->1 UAR >>>> bfreg >>>> to a QP. >>> >>> >>> I will most likely have the numbers with TDs in the second half of the >>> week. I will report them here then. >>> >> >> Yes, please share your results with the TDs usage with and without your >> patch, this may help clarification the issue. I have attached the graph which shows sender-receiver, 2-byte RDMA_WRITE message rates with Thread Domains and the effects of my patches. All the lines in the graph are with 16 threads using BlueFlame writes i.e. no postlist. The lines represent the following: woTD: base without using Thread Domains i.e. using the statically allocated UARs wTD: base using Thread Domains; 1 TD for each QP wTD-nolocks: base using TD + [1] wTD-independent: base using TD + this RFC patch wTD-nolocks-independent: base using TD + [1] + this RFC patch where, base = rdma-core@1eee3c837e0290f1ac7e5ac453ed69e8fd927aab [1] = https://patchwork.kernel.org/patch/10374625/ This is from the Gomez machines of the JLSE cluster which host a ConnectX-4 card. The dedicated nodes are running the 4.16 elrepo kernel (http://elrepo.org/linux/kernel/el7/SPECS/kernel-ml-4.16.spec) I had initially attributed the 8-way to 16-way drop to the contention between the 5th and the 16th QP in the original case of using just the statically allocated UARs. However, it seems like something more is happening since the drop exists even in the "wTD-nolocks-independent" line where we have no locks, no UAR-sharing. From perf, I see there is 36% increase in L1-dcache-load-misses going from 8-way to 16-way in "wTD-nolocks-independent". I'm still investigating but let me know if you have any ideas/thoughts. >>>>> The x-axis is the ratio of #QPs:#CTXs. For example, 2-way CTX-sharing >>>>> means there are 8 CTXs with 2 QPs each. "wBF" means "with BlueFlame" >>>>> and "woBF" means without (by setting MLX5_SHUT_UP_BF=1). "wPostlist2" >>>>> means the size of the linked-list of WQEs is 2. >>>> >>>> >>>> >>>> Looking at your graph, the best results are wPostlist2-wBF, correct ? but >>>> in >>>> that case we don't expect BF at all but DB as you wrote below. Can you >>>> please clarify the test and the results that are represented here ? >>> >>> >>> Yes, correct. With postlist, rdma-core doesn't use BF, just DB. I >>> included the various lines to show differences in behavior. The >>> semantics of Verbs-users may or may not allow the use of features such >>> as postlist. >>> >> >> So what in the graph referred to your initial patch improvements ? the green >> line was a DB test and not a BF results. > > The green line is without Postlist. So, the number of WQEs per > ibv_post_send is 1. In this case, rdma-core uses BF, not DB. The graph > doesn't show improvements from my patches; I'm just showing the > current behavior under different scenarios: "wPostlist2-wBF" means DB > is used on WC pages; "woPostlistwBF" means BF is used on WC pages; > "woPostlistwoBF" means sending 1 WQE per ibv_post_send on UC pages. > Hope this is clearer. Please let me know if you have more > clarification questions. > > -Rohit
Attachment:
rdma_core_tds_patches.png
Description: PNG image
