Hi,
I'm working both on DMA engines implementations in FPGA and their Linux
drivers.
Now I need to create an engine that takes the hugepages-backed buffer
allocated
by the user-space application and passes it to the device.
My current solution:
https://forums.xilinx.com/t5/Embedded-Linux/How-to-pass-efficiently-the-hugepages-backed-buffer-to-the-BM/m-p/1229340/highlight/true#M49777
or https://stackoverflow.com/a/67065962/1735409
uses the get_user_pages_fast function, to create the kernel mapping,
and then uses sg_alloc_table_from_pages to build sg_table for it.
I have verified that the created sg_table has one entry for each individual
hugepage (so I can efficiently map it for my SG-capable DMA device).
The disadvantage of that solution is that I need to create and keep a
huge set
of standard-size pages. Because the "struct page" occupies between 56 and 80
bytes, I get the overhead up to 80/4096 which is approx. 2%.
The open question is if I can free those pages as soon as the sg_table
is created? (As far as I know, the hugepages are locked automatically).
Of course it is unclear what happens if the application crashes and its
mmaped
hugepage-based buffer gets freed. Will the driver be notified about
closing the
file so that it can disable DMA before that memory can be taken for other
purposes?
To be sure that it doesn't happen maybe it is good to keep those pages
locked
in the kernel as well.
The big improvement would be if we could have the get_user_hugepages_fast
function. The user should be allowed to create a smaller number of page
structs.
The function should check if the requested region really consists of
hugepages
and return an error if it doesn't.
Another important fact is that we don't need a kernel mapping for those
pages
at all. So it would be good to have yet another function:
sg_alloc_table_from_user_pages
which should take an additional "flag" argument enabling the user to decide
if the area used consists of normal pages or of hugepages.
The function should return an error in case if the flag does not match the
properties of the region. Of course the function should also lock the pages,
and sg_free_table should unlock them (protecting against the danger
of application crash, that I described above).
As a temporary workaround, is it possible to "manually" walk the pages
creating the application-delivered buffer, verify that they are hugepages,
lock them and create the sg_table?
What functions/macros should be used for that to ensure that the implemented
solution be portable and keeps working in a reasonable number of future
versions
of the Linux kernel?
I'll appreciate comments, suggestions and considering of the above proposal.
With best regards,
Wojtek
--
Wojciech M Zabolotny, PhD, DSc
Institute of Electronic Systems
Faculty of Electronics and Information Technology
Warsaw University of Technology
