On 4/14/21 4:58 AM, wzab@xxxxxxxxxxxxx wrote:
Hi,
I'm working both on DMA engines implementations in FPGA and their
Linux drivers.
Now I need to create an engine that takes the hugepages-backed buffer
allocated
by the user-space application and passes it to the device.
My current solution:
https://forums.xilinx.com/t5/Embedded-Linux/How-to-pass-efficiently-the-hugepages-backed-buffer-to-the-BM/m-p/1229340/highlight/true#M49777
or https://stackoverflow.com/a/67065962/1735409
uses the get_user_pages_fast function, to create the kernel mapping,
and then uses sg_alloc_table_from_pages to build sg_table for it.
I have verified that the created sg_table has one entry for each
individual
hugepage (so I can efficiently map it for my SG-capable DMA device).
The disadvantage of that solution is that I need to create and keep a
huge set
of standard-size pages. Because the "struct page" occupies between 56
and 80
bytes, I get the overhead up to 80/4096 which is approx. 2%.
The open question is if I can free those pages as soon as the sg_table
is created? (As far as I know, the hugepages are locked automatically).
Of course it is unclear what happens if the application crashes and
its mmaped
hugepage-based buffer gets freed. Will the driver be notified about
closing the
file so that it can disable DMA before that memory can be taken for other
purposes?
To be sure that it doesn't happen maybe it is good to keep those pages
locked
in the kernel as well.
The big improvement would be if we could have the get_user_hugepages_fast
function. The user should be allowed to create a smaller number of
page structs.
The function should check if the requested region really consists of
hugepages
and return an error if it doesn't.
Another important fact is that we don't need a kernel mapping for
those pages
at all. So it would be good to have yet another function:
sg_alloc_table_from_user_pages
which should take an additional "flag" argument enabling the user to
decide
if the area used consists of normal pages or of hugepages.
The function should return an error in case if the flag does not
match the
properties of the region. Of course the function should also lock the
pages,
and sg_free_table should unlock them (protecting against the danger
of application crash, that I described above).
As a temporary workaround, is it possible to "manually" walk the pages
creating the application-delivered buffer, verify that they are
hugepages,
lock them and create the sg_table?
What functions/macros should be used for that to ensure that the
implemented
solution be portable and keeps working in a reasonable number of
future versions
of the Linux kernel?
I'll appreciate comments, suggestions and considering of the above
proposal.
With best regards,
Wojtek
Do you have trial patch you could RFC ?
Are you aware of the xilinx qdma patchset ?
https://www.xilinx.com/support/answers/71453.html
https://github.com/Xilinx/dma_ip_drivers
Maybe its kernel or dpdk driver has what you need.
Tom
