On 6/28/24 21:14, David Laight wrote:
...
The way Mellanox ConnectX5 driver handles 'release of allocated pages
from HCA' or 'allocation of pages to HCA', is by sending an event to the
host. This event will have number of pages in it. If the number is
positive, that indicates HCA is requesting that number of pages to be
allocated. And if that number is negative, it is the HCA indicating that
that number of pages can be reclaimed by the host.
A one line comment would do.
Possibly even negating the be32toh() result?
In this patch we are restricting the maximum number of pages that can be
reclaimed to be 50000 (effectively this would be -50000 as it is
reclaim). This limit is based on the capability of the firmware as it
cannot release more than 50000 back to the host in one go.
Hang on, why are you soft limiting it to the hard limit?
I thought the problem was that releasing a lot of pages took a long
time and 'stuffed' other time-critical tasks.
The only way to resolve that would seem to be to defer the actual freeing
to a low (or at least normal user) priority thread.
You would definitely want to get out of 'softint' context.
(Which is out of napi unless forced to be threaded - and that only really
works if you force the threads under the RT scheduler.)
David
Hi David,
The issue here is, when Mellanox device sends a huge number of pages
back to the host to reclaim, the host allocates a certain number of
mailbox messages mlx5_cmd_mailbox to accommodate the DMA addresses of
the memory to be reclaimed. The freeing of these mailbox messages is
time consuming (not the freeing of actual pages).
Now, the limit of the FW is that presently, it frees upto 50000 pages.
This limit can increase in future firmware versions. We are limiting
this in the driver because we see optimal results with this limit during
our tests. The results indicated that the time consumed while freeing of
mailbox messages stayed 2 usec on average - which is tolerable and would
not need running this thread in a different (low priority) context.
Thanks,
Anand
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
