Hi,
On Tue, Apr 16, 2024 at 12:59:50PM GMT, Michael Scherle wrote:
> Hello,
>
> Thanks for your changesets, they definitely reduce the delay significantly
> (to a similar level as our provosoric fixes, but yours are much cleaner).
>
> On the client side (spice-gtk) I looked at the problem with the high
> decoding time (2 frames buffering) and was able to find a simple fix with
> the help of the gstreamer community:
>
> ---
> src/channel-display-priv.h | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/src/channel-display-priv.h b/src/channel-display-priv.h
> index 1a7590a..a2af1a7 100644
> --- a/src/channel-display-priv.h
> +++ b/src/channel-display-priv.h
> @@ -177,7 +177,7 @@ static const struct {
> * (hardcoded in spice-server), let's add it here to avoid the warning.
> */
> { SPICE_DISPLAY_CAP_CODEC_H264, "h264",
> - "h264parse ! avdec_h264", "video/x-h264,stream-format=byte-stream" },
> + "h264parse ! avdec_h264",
> "video/x-h264,stream-format=byte-stream,alignment=au" },
>
> /* SPICE_VIDEO_CODEC_TYPE_VP9 */
> { SPICE_DISPLAY_CAP_CODEC_VP9, "vp9",
> @@ -185,7 +185,7 @@ static const struct {
>
> /* SPICE_DISPLAY_CAP_CODEC_H265 */
> { SPICE_DISPLAY_CAP_CODEC_H265, "h265",
> - "h265parse ! avdec_h265", "video/x-h265,stream-format=byte-stream" },
> + "h265parse ! avdec_h265",
> "video/x-h265,stream-format=byte-stream,alignment=au" },
jfyi, this was discussed in the past. It depends how spice server
was configured too, no? I'm not sure, it has been awhile. What I
mean is, what/who is doing h264 encoding. We had a
spice-streaming-agent that wrapped guest's GPU h264 encoding and
sent to the client, with the same protocol.... depending how it
is configured, the stream-format was important I think. Again,
not 100% sure.
>
> };
>
> --
> 2.40.1
>
> However, this change should probably still be tested on different setups.
> Since I don't know whether they are always au aligned, I should probably
> find out about that.
>
> Also I have made other experiments, such as removing decoding_queue in
> channel-display-gst.c and adding the SpiceGstFrame to the metadata of the
> gstBuffer instead, as well as completely ignoring the display time of a
> frame and instead displaying them immediately. With that i got down to
> 60-80ms delay.
If you send patches about this one, feel free to tag me. This
looks cool.
> Do you know if your changes or similar ones that reduce the
> delay will go upstream soon?
>
> While looking through the source code, I found
> SPICE_KEYPRESS_DELAY, which is not mentioned anywhere. Is this
> the only use to save some network traffic? Is there any reason
> not to always set this to 0 in today's network environments?
> (And maybe set the default to 0?)
Introduced in c03e002152dc0c, commit log says:
> widget: add keypress-delay property
>
> The delay before the press event is sent to the server if the key is
> kept pressed. If the key is released within that time, that delay is
> ignored and a single key-press-release event will be sent.
Introduced in 2012. I'm pretty sure there were reasons for it.
Not sure if worth to remove it.
Cheers,
Victor
> Michael
>
> On 03.04.24 21:22, Frediano Ziglio wrote:
> > Frediano
> >
> > Il giorno mar 2 apr 2024 alle ore 15:27 Michael Scherle
> > <michael.scherle@xxxxxxxxxxxxxxxxxx> ha scritto:
> > >
> > > Hi Frediano,
> > >
> > > thank you very much for your detailed answer.
> > >
> > >
> > > On 02.04.24 14:13, Frediano Ziglio wrote:
> > >
> > > > Really short explanation: Lipsync.
> > > >
> > > > Less cryptic explanation: video streaming was added much time ago when
> > > > desktops used 2D graphic drawings, like lines, fillings, strings and
> > > > so on. At that time networks were more unreliable, latency bigger, and
> > > > with high probability a continuous bitblt on the same big area was a
> > > > video playing. So the idea of detecting the video playing and
> > > > optimizing to sync audio and video was a good idea.
> > >
> > > ok this explains a lot.
> > >
> > > > Now starts my opinionated ideas. The idea of continuous bitblt being
> > > > only a video stream is wrong, nowadays desktops do use large bitblt
> > > > for everything, or better they use 3D cards a lot and compose the
> > > > various windows on the screen which appears to us as just bitblt,
> > > > often contiguous. So the delay should just be removed optimizing for
> > > > real time video streaming. As you realize the algorithm also keeps
> > > > increasing the delay for every glitch found which is not improving the
> > > > user experience. I have different changesets removing entirely all
> > > > these delays (it's possible to get this just by changing the server
> > > > part), the result is much less delay, the audio/video sync (watching a
> > > > movie) is, with nowadays networks, acceptable.
> > >
> > >
> > > Would it be possible to get your changesets, so that I could try them
> > > out? I would be interested to know how this can be implemented with only
> > > server-side changes. A dirty idea I had (and tried) would be to set the
> > > mm_time to the past so that the client displays the image immediately,
> > > but that would not be a good fix in my opinion.
> > >
> >
> > That's the commit
> > https://cgit.freedesktop.org/~fziglio/spice-server/commit/?h=nvidia&id=eaaec7be80a9d402f425f7571bb27a082ebf739a.
> >
> > > I would rather consider it reasonable that the server timestamps the
> > > frames (and perhaps the sound) with the encoding time and that the
> > > client itself calculates when it wants to display them (from the diffs).
> > > So the client could decide if it wants to display the images directly or
> > > add some delay to compensate for network jitter (or lipsync) or maybe
> > > even implement something like v-sync. These would of course be breaking
> > > changes that would require changes to the client and server and would
> > > make them incompatible with older versions. If this could not be done
> > > directly, due to compatibility reasons, maybe this could be implemented
> > > in a separate low latency mode or something like that (which both server
> > > and client needs to support).
> > >
> >
> > I suppose the negative time you though is something like
> > https://cgit.freedesktop.org/~fziglio/spice-server/commit/?h=nvidia&id=4a1a2a20505bc453f30573a0d453a9dfa1d97e7c
> > (which improve the previous).
> >
> > > Even with above ideas applied, for spice-gtk, I have noticed a high
> > > decode delay. The gstreamer pipeline always seems to keep at least 2
> > > frames in the pipeline (regardless of the frame rate) which increases
> > > the delay further. Have you also noticed this? I'm currently looking
> > > into the reason for this.
> > >
> > > When testing stuff out we saw that Sunshine/Moonlight performed very
> > > well in generating a low delay and high QoE. That is kind of our
> > > benchmark for remote access to strive for :)
> > >
> > > Greetings
> > > Michael
> > >
> >
> > Frediano
> >
> > > > >
> > > > > On 15.03.24 14:08, Michael Scherle wrote:
> > > > > > Hello spice developers,
> > > > > >
> > > > > > we are trying to develop an Open Source virtual desktop infrastructure
> > > > > > to be deployed at multiple German universities as described, by my
> > > > > > colleagues, in the paper which I have put in the attachment. The
> > > > > > solution based on openstack, qemu, spice... Our plan is also to have VM
> > > > > > instances with virtual GPUs (SR-IOV). Due to the resulting requirements,
> > > > > > it is necessary to transmit the image data as a video stream.
> > > > > > We have seen Vivek Kasireddy recent work on spice which solves exactly
> > > > > > this problem. However, when we tested it, we noticed a very high input
> > > > > > to display delay (400 ms+ but only if the image data is transferred as
> > > > > > video-stream). However, the problem seems to be a more general spice
> > > > > > problem or is there something wrong with our setup or are there special
> > > > > > parameters that we are missing?
> > > > > >
> > > > > > Our setup:
> > > > > >
> > > > > > QEMU: https://gitlab.freedesktop.org/Vivek/qemu/-/commits/spice_gl_on_v2
> > > > > > Spice:
> > > > > > https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v6
> > > > > > virt-viewer
> > > > > > Intel HW decoder/encoder (but same with sw)
> > > > > >
> > > > > > I have looked into what is causing the delay and have noticed that
> > > > > > encoding only takes about 3-4ms. In general, the image seems to reach
> > > > > > the client in less than 15ms.
> > > > > > The main problem seems to be that gstreamer gets a very high
> > > > > > margin(https://gitlab.freedesktop.org/spice/spice-gtk/-/blob/master/src/channel-display.c?ref_type=heads#L1773) and therefore waits a long time before starting decoding. And the reason for the high margin seems to be the bad mm_time_offset https://gitlab.freedesktop.org/spice/spice-gtk/-/blob/master/src/spice-session.c?ref_type=heads#L2418 which is used to offset the server time to the client time (with some margin). And this variable is set by the spice server to initially 400 ms https://gitlab.freedesktop.org/spice/spice/-/blob/master/server/reds.cpp?ref_type=heads#L3062 and gets updated with the latency https://gitlab.freedesktop.org/spice/spice/-/blob/master/server/reds.cpp?ref_type=heads#L2614 (but only increased). I still need to see how this latency is calculated.
> > > > > >
> > > > > > Am I missing something or is this design not intended for transmitting
> > > > > > interactive content via video stream?
> > > > > > Temporarily overwriting the margin and tweaking parameter settings on
> > > > > > the msdkh264dec brought the delay to about 80-100ms, which is not yet
> > > > > > optimal but usable. To see what is technical possible on my setup, I
> > > > > > made a comparison using moonlight/sunshine which resulted in an delay of
> > > > > > 20-40ms.
> > > > > >
> > > > > > Our goal is to achieve some round trip time similar to the
> > > > > > moonlight/sunshine scenario to achieve a properly usable desktop
> > > > > > experience.
> > > > > >
> > > > > > Greetings
> > > > > > Michael
> > > > >
> > > > > Greetings
> > > > > Michael
>
Attachment:
signature.asc
Description: PGP signature