Hi Michael and Christoph:
I just sent out V5 patchset. I use alloc_pages() to allocate rx/tx
ring buffer in Isolation VM and use vmap() to map rx/tx buffer first
because the vmbus_establish_gpadl() still needs to va of low end memory
to initialize gpadl buffer. After calling vmbus_establish_gpadl(), the
va returned by vmap will be unmapped to release virtual address space
which will not be used in the following code and then map these pages in
the extra address space above shared_gpa_boundary via vmap_pfn(). Please
have a look.
https://lkml.org/lkml/2021/9/14/672
Thanks.
On 9/2/2021 11:57 PM, Michael Kelley wrote:
From: Christoph Hellwig <hch@xxxxxx> Sent: Thursday, September 2, 2021 1:00 AM
On Tue, Aug 31, 2021 at 05:16:19PM +0000, Michael Kelley wrote:
As a quick overview, I think there are four places where the
shared_gpa_boundary must be applied to adjust the guest physical
address that is used. Each requires mapping a corresponding
virtual address range. Here are the four places:
1) The so-called "monitor pages" that are a core communication
mechanism between the guest and Hyper-V. These are two single
pages, and the mapping is handled by calling memremap() for
each of the two pages. See Patch 7 of Tianyu's series.
Ah, interesting.
3) The network driver send and receive buffers. vmap_phys_range()
should work here.
Actually it won't. The problem with these buffers is that they are
physically non-contiguous allocations.
Indeed you are right. These buffers are allocated with vzalloc().
We really have two sensible options:
1) use vmap_pfn as in the current series. But in that case I think
we should get rid of the other mapping created by vmalloc. I
though a bit about finding a way to apply the offset in vmalloc
itself, but I think it would be too invasive to the normal fast
path. So the other sub-option would be to allocate the pages
manually (maybe even using high order allocations to reduce TLB
pressure) and then remap them
What's the benefit of getting rid of the other mapping created by
vmalloc if it isn't referenced? Just page table space? The default sizes
are a 16 Meg receive buffer and a 1 Meg send buffer for each VMbus
channel used by netvsc, and usually the max number of channels
is 8. So there's 128 Meg of virtual space to be saved on the receive
buffers, which could be worth it.
Allocating the pages manually is also an option, but we have to
be careful about high order allocations. While typically these buffers
are allocated during system boot, these synthetic NICs can be hot
added and removed while the VM is running. The channel count
can also be changed while the VM is running. So multiple 16 Meg
receive buffer allocations may need to be done after the system has
been running a long time.
2) do away with the contiguous kernel mapping entirely. This means
the simple memcpy calls become loops over kmap_local_pfn. As
I just found out for the send side that would be pretty easy,
but the receive side would be more work. We'd also need to check
the performance implications.
Doing away with the contiguous kernel mapping entirely seems like
it would result in fairly messy code to access the buffer. What's the
benefit of doing away with the mapping? I'm not an expert on the
netvsc driver, but decoding the incoming packets is already fraught
with complexities because of the nature of the protocol with Hyper-V.
The contiguous kernel mapping at least keeps the basics sane.
4) The swiotlb memory used for bounce buffers. vmap_phys_range()
should work here as well.
Or memremap if it works for 1.
Case #2 above does unusual mapping. The ring buffer consists of a ring
buffer header page, followed by one or more pages that are the actual
ring buffer. The pages making up the actual ring buffer are mapped
twice in succession. For example, if the ring buffer has 4 pages
(one header page and three ring buffer pages), the contiguous
virtual mapping must cover these seven pages: 0, 1, 2, 3, 1, 2, 3.
The duplicate contiguous mapping allows the code that is reading
or writing the actual ring buffer to not be concerned about wrap-around
because writing off the end of the ring buffer is automatically
wrapped-around by the mapping. The amount of data read or
written in one batch never exceeds the size of the ring buffer, and
after a batch is read or written, the read or write indices are adjusted
to put them back into the range of the first mapping of the actual
ring buffer pages. So there's method to the madness, and the
technique works pretty well. But this kind of mapping is not
amenable to using vmap_phys_range().
Hmm. Can you point me to where this is mapped? Especially for the
classic non-isolated case where no vmap/vmalloc mapping is involved
at all?
The existing code is in hv_ringbuffer_init() in drivers/hv/ring_buffer.c.
The code hasn't changed in a while, so any recent upstream code tree
is valid to look at. The memory pages are typically allocated
in vmbus_alloc_ring() in drivers/hv/channel.c.
Michael
