Hi Geoff, On Wed, 16 May 2018 20:45:22 +1000 geoff@xxxxxxxxxxxxxxx wrote: > Hi All, > > I have been working on making LookingGlass (LG) > (https://github.com/gnif/LookingGlass) viable to use inside a Linux > guest for Windows -> Linux frame streaming. Currently LG works very well > when running native on the host streaming frames from a Windows guest > via an IVSHMEM device, however when the client is running inside a Linux > guest we seem to be hitting a memory performance wall. To check my understanding, a Windows VM with assigned GPU is scraping frames and writing them into the shared memory BAR of an ivshmem device. If the client is the host Linux, we can read frames out of the shared memory space fast enough, but if the client is Linux in another VM then we hit a performance issue in reading the frames out of the shared memory BAR. In the former case, the client is reading from the shared memory area directly, while in the latter it's through the ivshmem device. > Before I continue here is my hardware configuration: > > ThreadRipper 1950X in NUMA mode. > GeForce 1080Ti passed through to a Windows guest > AMD Vega 56 passed through to a Linux guest > > Both Windows and Linux guests are bound to the same node as are their > memory allocations. Memory copy performance in the Linux guest matches > native memory copy performance at ~40GB/s. Windows copy performance is > slower seeming to hit a wall at ~14GB/s, slower then I would have > expected but thats is a separate issue, suffice to say it's plenty fast > enough for what I am trying to accomplish here. Copy performance here is between two buffers in VM RAM though, right? The performance issue is copies from an MMIO BAR, which should have some sort of uncacheable mapping. I guess we assume write performance to the MMIO BAR is not the issue as that's static between the test cases. > Windows is feeding captured frames at 1920x1200 @ 32bpp into the IVSHMEM > virtual device, which the Linux guest is using as it's input. The data > transfer rate is matching that above of ~14GB/s, allowing for in theory > over 1,600 frames per second. But when I take this buffer and try to > feed it to the AMD Vega, I see an abysmal transfer rate of ~131MB/s (~ > 15fps). Copying the shared memory into an intermediate buffer before > feeding the data to the GPU doesn't make a difference. What does "feeding" from ivshmem to GPU look like here? Is this a user process in the Linux VM mmap'ing both the ivshmem MMIO BAR and the GPU BAR and doing a memcpy() directly between them? Is there a driver in the Linux guest for the ivshmem device or are we just enabling MMIO in the command register and mmap'ing the resource file via sysfs? > Now one might thing the rendering code is at fault here (and it might > be), however if I instead dynamically create the buffer for each frame I > do not see any performance issues, for example, the below will generate > a vertical repeating gradient that is horizontally scrolling. > > static int offset = 0; > char * data = malloc(frameSize); > for(int i = 0; i < frameSize; ++i) > data[i] = i + offset; > ++offset; > render(data, frameSize); > free(data); This is trying to prove that the copy to the GPU in the Linux guest is not at fault, right? It seems though that that path is the same regardless of whether the client is in the host or another guest, the GPU is assigned in the latter case, but there's plenty of evidence that we can use a GPU at near native performance in a guest already. So, despite the subject, I'm not sure how the passthrough/assigned device plays a factor here, I think the question is around Linux reading data out of the ivshmem MMIO BAR. Is that actually affected by the presence of an assigned device in the client VM? We should be able to test reading out of the ivshmem device regardless of an assigned device. > Is there something I am overlooking with regards to this buffer transfer > Could DMA be rejecting the buffer because it's not in system ram but a > memory mapped BAR? and if so, why doesn't copying it to an intermediate > buffer first help? If DMA were the issue, I'd expect "work" vs "not work", not "slow" vs "fast". I'm a bit confused how an intermediate buffer could improve the situation here, your evidence suggests that we can write data out to the client GPU at sufficient speed, it's getting data out of the ivshmem BAR that's the problem. We still need to fill an intermediate buffer from the ivshmem BAR and we should be able to measure the performance of that independent of ultimately writing the data into the GPU on the client. In fact, I'm not really sure where any DMA is occurring, afaict we're not issuing a command to ivshmem asking it to write some range of shared memory to system RAM, we're reading it directly from the MMIO space of the device. That's just regular programmed I/O from the CPU governed by the CPUs memory attributes for the given address space. Of course it would be really bad if these PIOs trapped out to QEMU, have you verified we're not hitting memory_region_dispatch_read() while the client is accessing the MMIO BAR? Thanks, Alex
