Hi Frediano, > Il giorno ven 31 mar 2023 alle ore 08:52 Kasireddy, Vivek > <vivek.kasireddy@xxxxxxxxx> ha scritto: > > > > Hi Frediano, > > > > > > > > Il giorno lun 27 mar 2023 alle ore 07:53 Kasireddy, Vivek > > > <vivek.kasireddy@xxxxxxxxx> ha scritto: > > > > > > > > Hi Frediano, > > > > > > > > > I was trying the patch series but the client keeps crashing. I > > > > > tried different versions of remote-viewer (from Fedora and from > > > > > Ubuntu) and they both crashed. > > > > > On host I installed Intel VA drivers , Gstreamer VAAPIs and forced > > > > > gstreamer:h264 as encoder. > > > > [Kasireddy, Vivek] I don't think any of these packages are needed if we > are > > > testing > > > > with a software (i.e, CPU-based) encoder/decoder (i.e, using > x264enc/x264dec > > > plugins). > > > > > > They should allow hardware encoding. > > [Kasireddy, Vivek] IIUC, currently, Spice only uses x264enc/x264dec plugins > and > > with these only software encoding is possible. We need to use different > plugins to > > enable hardware encoding. For instance we need to use either > vaapih264enc or > > msdkh264enc on Intel hardware. As I mentioned earlier, we are planning to > > add this support. > > > > > > > > > On my Fedora 37, these relevant plugins are provided by: > > > > # rpm -q --whatprovides /usr/lib64/gstreamer-1.0/libgstx264.so > > > > gstreamer1-plugins-ugly-1.20.5-1.fc37.x86_64 > > > > # rpm -q --whatprovides /usr/lib64/gstreamer-1.0/libgstlibav.so > > > > gstreamer1-plugin-libav-1.20.5-1.fc37.x86_64 > > > > > > > > The former provides x264enc and the latter provides x264dec. You can > either > > > choose > > > > the ones provided by your distro or build Gstreamer with -Dgst-plugins- > > > ugly:x264=enabled > > > > and -Dlibav=enabled. However, note that one of our eventual goals is to > cleanly > > > add a > > > > new Gstreamer pipeline to Spice to provide users an option for > hardware-based > > > > (i.e, GPU assisted) H.264 encoding/decoding using > msdkh264enc/msdkh264dec > > > plugins: > > > > > https://gstreamer.freedesktop.org/documentation/msdk/msdkh264enc.htm > l?gi- > > > language=c > > > > > > > > > Anything else you need to do? > > > > [Kasireddy, Vivek] Sorry, I was not expecting anyone would test the > Spice server > > > > patches and therefore did not provide all the required information to > test them. > > > > Anyway, I suspect the reason for your remote-viewer crash is because > there is no > > > > (primary) surface created. Here is the tentative Qemu patch that does > just that: > > > > > > > > > > Do you have a branch for Qemu somewhere? > > [Kasireddy, Vivek] Yes, I created one here: > > https://gitlab.freedesktop.org/Vivek/qemu/-/commits/pref_codec1 > > https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf1 > > > > And, FYI, here is my exact qemu launch cmd (on Fedora 37): > > ./qemu-system-x86_64 -m 4096m -enable-kvm -cpu host -smp cores=8 - > drive file=./fed30kvm.img,format=qcow2,cache=none -vga none -device > e1000,netdev=net0,mac=DE:AD:BE:EF:BF:FA -netdev > tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-gpu- > pci,max_outputs=1,blob=true,xres=1920,yres=1080 -spice > port=3001,gl=on,disable-ticketing=on,preferred-codec=gstreamer:h264 - > object memory-backend-memfd,id=mem1,size=4096M -machine memory- > backend=mem1 -usb -device usb-tablet -serial stdio > > > > > > > > > fd = egl_get_fd_for_texture(ssd->ds->texture, > > > > > > > > I did not yet post this patch to the Qemu mailing list because I still need > to > > > implement > > > > feature negotiation between Qemu and Spice to auto-detect this > capability. And, > > > > in addition to this patch, you also need the patch that adds the > "preferred-codec" > > > > option to Qemu, unless you are hardcoding it in Spice: > > > > https://lists.nongnu.org/archive/html/qemu-devel/2023- > 01/msg04999.html > > > > > > > > > > Yes, I hardcoded that in SPICE. Note that I posted a comment in that > > > commit, I think it's valid as it is. > > > Also I think it should not change the SPICE default. > > [Kasireddy, Vivek] Yes, I noted your comment; I'll make the change in the > > next version. > > > > Thanks, > > Vivek > > > > > > > > > Lastly, not sure if it makes a difference, but here are the relevant > options I am > > > > using to launch Qemu: > > > > -device virtio-gpu-pci,max_outputs=1,blob=true,xres=1920,yres=1080 > > > > -spice port=3001,gl=on,disable-ticketing=on,preferred- > codec=gstreamer:h264 > > > > -object memory-backend-memfd,id=mem1,size=4096M > > > > -machine memory-backend=mem1 -usb -device usb-tablet -serial stdio > > > > > > > > Thanks, > > > > Vivek > > > > > > > > > > I will try again. > > > > > > Thanks, > > > Frediano > > > > > > > > > > > > > On the logs I find: > > > > > > > > > > 2023-03-25T19:31:36.034007Z qemu-system-x86_64: warning: Spice: > > > > > ../server/dcc-send.cpp:1786:red_marshall_gl_draw_stream: bad > return > > > > > value (0) from VideoEncoder::encode_dmabuf > > > > > 2023-03-25T19:31:37.064219Z qemu-system-x86_64: warning: spice: > no > > > > > gl-draw-done within one second > > > > > 2023-03-25T19:31:58.214387Z qemu-system-x86_64: warning: Spice: > > > > > Connection reset by peer > > > > > 2023-03-25T19:31:58.214482Z qemu-system-x86_64: warning: Spice: > > > > > Connection reset by peer > > > > > 2023-03-25T19:31:58.214580Z qemu-system-x86_64: warning: Spice: > > > > > Connection reset by peer > > > > > 2023-03-25T19:31:58.214642Z 2023-03-25T19:31:58.214636Z > > > > > qemu-system-x86_64:qemu-system-x86_64: warning: Spice: > Connection > > > > > reset by peer > > > > > warning: Spice: display:0 (0x55947e76fd10): Connection reset by peer > > > > > 2023-03-25T19:31:58.214721Z qemu-system-x86_64: warning: Spice: > > > > > Connection reset by peer > > > > > 2023-03-25T19:31:58.214841Z qemu-system-x86_64: warning: Spice: > > > > > Connection reset by peer > > > > > 2023-03-25T19:31:58.215057Z qemu-system-x86_64: warning: Spice: > > > > > Connection reset by peer > > > > > > > > > > 0 value from encode_dmabuf should mean > > > VIDEO_ENCODER_FRAME_DROP. > > > > > > > > > > Regards, > > > > > Frediano > > > > > > > Hi, > some updates. I grab your exact versions and compiled them again. > At the beginning I still had some problems with codecs and security > settings (seccomp was enabled causing Qemu to crash on gstreamer code > ,turned off). > However even after solving them the result is not that great. The > video is not fluid and when there are some updates it is almost > "garbage". Which type of GPU are you using? I'm using integrated Intel > and I remember they have a not-linear memory addressing for textures > which lead to these garbaged style images due to trying to interpret > memory as linear. [Vivek] I am using an integrated Intel GPU (Gen 12, codenamed TigerLake/AlderLake) as well. Regardless, IIUC, the memory backing the dmabuf fd is linear as Virtio-gpu does not allow/expose modifier support -- which is needed to allocate non-linear/tiled memory. The video not being fluid issue is probably a symptom of the time it (software encoder/x264enc) takes to encode a frame. In my setup, it takes about ~10-12 ms to encode a 1920x1080 BGRX frame -- which obviously drops the FPS to ~25-30. And, the "garbage" behavior could most likely be environmental. In other words, in this use-case, we are heavily relying on the complex Gstreamer stack which includes 100s of libraries and plugins; so, any of these could be playing a role. Can you please try building the Gstreamer stack yourself and test to see if it helps? Also, can you forcefully use only the avdec_h264 decoder (by possibly hiding/removing libgstvaapi.so, libgstva.so, libgstopenh264.so and other h264 decoders) to see if the garbage goes away? Lastly, could you please share details about your setup such as OS version, which libraries you have built (decoders?) and installed, Qemu/Spice build options, runtime options, etc., so that I can try reproducing this issue in my setup? > I also had to change spice-gtk due to a missing specification in the > encoder. Which client did you use? Which version? Maybe the bug was > introduced later. [Vivek] I am currently using remote-viewer version 11.0-5.fc37 as the client but I have also successfully tested with the remote-viewer version from Fedora 36. In addition, I have also tested with spice-gtk master as well. > Also I have some warnings in Qemu logs due to attempts to create > primary surfaces multiple times. [Vivek] Yeah, I see them as well. I'll eventually fix them as they seem harmless so far. > > By the way, the code looks better than the previous version but as > commented the running behaviour is not that great. [Vivek] As I mentioned above, I think there is not much we can do in Spice to fix the behavior you are seeing given the heavy reliance on Gstreamer in this use-case. However, we are also actively working on cleanly adding Spice options to enable hardware encoders/decoders for Intel GPUs to reduce the encode time to < 5 ms. In addition, we are also trying to identify and fix all the other performance bottlenecks. We'll post patches as soon as we clean everything up and fix all the issues we are seeing. > The style (well, also the style of spice code I have to admit) is a > bit "messy" but better to fix the big problems first than try to solve [Vivek] Ok, I guess we can fix it incrementally if you can first point out the worst style violoations. Thanks, Vivek > everything at once. > > Regards, > Frediano > > > > > > Il giorno gio 16 mar 2023 alle ore 06:05 Vivek Kasireddy > > > > > <vivek.kasireddy@xxxxxxxxx> ha scritto: > > > > > > > > > > > > For clients that cannot accept a dmabuf fd directly (such as those > > > > > > running on a remote system), this patch series provides a way for > > > > > > the Spice server to stream the gl/dmabuf data/buffer instead. This > > > > > > is mostly done by enabling the creation of Gst memory using a > dmabuf > > > > > > fd as the source. This ability is useful given that dmabuf is the > > > > > > standard mechanism for sharing buffers between various drivers > and > > > > > > userspace in many Graphics and Media usecases. Currently, this is > > > > > > only used/tested with Qemu and remote-viewer using the > x264enc/dec > > > > > > codec to stream the Guest/VM desktop but it can be easily extended > > > > > > to other plugins and applications. > > > > > > > > > > > > Here is roughly how things work: > > > > > > - The application (e.g, Qemu) chooses its preferred codec (a > Gstreamer > > > > > > one) and calls gl_scanout (to update the fd) followed by gl_draw. > > > > > > - In response, the Spice server checks to see if the client is capable > > > > > > of accepting a dmabuf fd directly or not. If yes, the fd is forwarded > > > > > > directly to the client; otherwise, a new stream is created. > > > > > > - The Spice server then sends the dmabuf fd to the Gstreamer > encoder > > > > > > which uses it as an input for creating an encoded buffer which is > then > > > > > > sent to the client. > > > > > > - Once the encoding process is done, an async completion cookie is > sent > > > > > > to the application. > > > > > > > > > > > > Here is a link to the previous version that used a drawable to share > > > > > > the dmabuf fd with the Gstreamer encoder: > > > > > > https://lists.freedesktop.org/archives/spice-devel/2023- > January/052948.html > > > > > > > > > > > > Cc: Frediano Ziglio <freddy77@xxxxxxxxx> > > > > > > Cc: Gerd Hoffmann <kraxel@xxxxxxxxxx> > > > > > > Cc: Marc-André Lureau <marcandre.lureau@xxxxxxxxxx> > > > > > > Cc: Dongwon Kim <dongwon.kim@xxxxxxxxx> > > > > > > > > > > > > Vivek Kasireddy (5): > > > > > > dcc: Check to see if the client supports multiple codecs > > > > > > dcc: Create a stream associated with gl_draw for non-gl clients > > > > > > dcc-send: Encode and send gl_draw stream data to the remote > client > > > > > > gstreamer-encoder: Add an encoder function that takes dmabuf fd > as > > > > > > input > > > > > > video-stream: Don't stop a stream if a gl_draw operation is > pending > > > > > > > > > > > > meson.build | 2 +- > > > > > > server/dcc-private.h | 4 ++ > > > > > > server/dcc-send.cpp | 89 ++++++++++++++++++++++- > > > > > > server/dcc.cpp | 36 +++++++--- > > > > > > server/display-channel-private.h | 6 ++ > > > > > > server/gstreamer-encoder.c | 119 > > > ++++++++++++++++++++++++++++++- > > > > > > server/video-encoder.h | 13 ++++ > > > > > > server/video-stream.cpp | 65 ++++++++++++++++- > > > > > > server/video-stream.h | 2 + > > > > > > 9 files changed, 319 insertions(+), 17 deletions(-) > > > > > > > > > > > > -- > > > > > > 2.37.2 > > > > > >