Re: Understanding UUAR calculations in mlx5 userspace driver

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, Feb 14, 2018 at 4:08 PM, Anuj Kalia <anujkaliaiitd@xxxxxxxxx> wrote:
>
> 1. FP stands for fast path
> 2. Each UAR page has four BlueFlame registers. This is specific to the
> NIC and firmware and is mentioned in the PRM. ConnectX-5 InfiniBand
> has four, ConnectX-4 in Ethernet mode seems to have two according to
> the online PRM.
> 3. No idea
>
> I'm curious: are you seeing any performance impact of
> "latency_sensitve" or "low latency UARs"?

Sorry for the delayed response. Getting back to this topic only now.
I do see some performance impact on the QPs assigned to low_lat_uuars
versur other uUARs. But I see them when I take off some opitmizations
such as postlist (so there are a lot more doorbells). Also, if you set
MLX5_SINGLE_THREADED=1 while running your bench, the locks are never
taken and also won't see any difference in performance between the
different uUARs.


 I've looked at this briefly
> in the past and couldn't find perf impact. IIRC, the driver uses "fast
> path" for only the first few QPs, but in my experiment even later QPs
> got similar latency. But I might have missed something. If there's an
> observable perf impact, you can probably figure out the answer to (3).
>
> --Anuj

I studied the code a little further and here are some questions below:

(1) It seems like only BFREG_0 and BFREG_1 of a UAR are being used to
ring the doorbell. BFREG_2 and BFREG_3, while being allocated in
`ibv_open_device`, are not assigned to the QPs belonging to the same
context.The PRM says that BFREG_2 and BFREG_3 of a UAR are used for
"fast path posting" (page 65). What do these "fast path" operations
refer to: are they control operations?

(2) From my understanding of `allocate_uars` and `alloc_bfreg` kernel
functions, there is no difference (in terms of hardware uUAR
functionality) between the `low_lat_uuars` and the other uUARs that
are being allocated during context creation. The only reason why
`low_lat_uuars` are called so is that there is no lock taken for the
QP's assigned to the `low_lat_uuars`, hence "low latency". The QPs
assigned to other uUARs will take a lock and might contend for it if
multiple QPs are assigned the same uUAR, hence not low latency. Can
someone (maybe, Yishai Hadas) confirm this?

-- Rohit

>
>
>
>
>
> On Wed, Feb 14, 2018 at 10:35 AM, Rohit Zambre <rzambre@xxxxxxx> wrote:
> > Hi,
> >
> > I am trying to understand the uuar-related calculations of the
> > userspace mlx5 driver (in mlx5_alloc_context) that use the various
> > constants introduced in
> > https://github.com/linux-rdma/rdma-core/commit/166d34841dd6a835e4f4f0196143880a73ce71bd.
> >
> > My understanding is that a uar is comprised of uuars. bfregs mean the
> > same thing as uuars. A doorbell is written to the bfreg/uuar assigned
> > to the QP.
> >
> > (1) W.r.t. MLX5_NUM_NON_FP_BFREGS_PER_UAR, what is a NON_FP_BFREG in a
> > UAR? I am curious to know what FP stands for.
> >
> > (2) Is the value of NUM_BFREGS_PER_UAR set to 4 because the driver
> > will have at least 4 low_lat_uuars for a context? This constant is
> > used only in the calculation of gross_uuars. My assumption is that the
> > NUM_BFREGS_PER_UAR value is not specific to the mlx5 device because of
> > the missing MLX5_ prefix. Please correct me if I am wrong.
> >
> > (3) Given that the value of tot_uuars and low_lat_uuars can be
> > configured through the MLX5_TOTAL_UUARS and MLX5_NUM_LOW_LAT_UUARS
> > environment variables respectively, it is possible for me to have, in
> > 1 context, 512 tot_uuars (=MLX5_MAX_BFREGS) and set all 511 of them as
> > low_lat_uuars. Is this correct?
> >
> > Thanks,
> >
> > Rohit Zambre
> > Ph.D. Student, Computer Engineering
> > University of California, Irvine
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux