Re: [PATCH spice-server 00/28] adaptive video streaming

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




----- Original Message -----
> From: "Yonit Halperin" <yhalperi@xxxxxxxxxx>
> To: "Andrew Cathrow" <acathrow@xxxxxxxxxx>
> Cc: spice-devel@xxxxxxxxxxxxxxx
> Sent: Thursday, March 7, 2013 3:52:04 PM
> Subject: Re:  [PATCH spice-server 00/28] adaptive video streaming
> 
> Hi,
> On 03/03/2013 07:50 AM, Andrew Cathrow wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Yonit Halperin" <yhalperi@xxxxxxxxxx>
> >> To: spice-devel@xxxxxxxxxxxxxxx
> >> Sent: Tuesday, February 26, 2013 1:03:46 PM
> >> Subject:  [PATCH spice-server 00/28] adaptive video
> >> streaming
> >>
> >> Hi,
> >>
> >> The following patch series introduces adaptive video streaming to
> >> spice.
> >>
> >> Until now, the mjpeg quality was constant (70), and the frame rate
> >> was modified
> >> according to the rate of frame drops in the server side (a drop
> >> occurs when a new frame reaches the server while
> >> an older frame is still queued in the pipe). In the client side,
> >> the
> >> video playback is synchronized according
> >> to the audio playback (each audio and video frame holds an mm-time
> >> field). The jitter-buffer size in the client
> >> was constant as well - 100 ms. When video frames arrive late to
> >> the
> >> client (i.e., when the audio playback is ahead of them),
> >> they are dropped.
> >>
> >> The adaptive video streaming is implemented by the following
> >> heuristic:
> >> Given a bit rate, we calculate the best combination of mjpeg
> >> quality
> >> and frame rate (henceforth, the stream parameters) for this
> >> bit rate. In order to decide this combination, we evaluate the
> >> encoding size for different jpeg
> >> qualities by applying them on successive frames.
> >> Every new stream is assigned with an initial bit rate. The bit
> >> rate
> >> is re-estimated and
> >> modified during the stream life time. The bit-rate is modified
> >> based
> >> on:
> >> 1) periodic reports from the client:
> >>     The client reports includes information about drops and the
> >>     playback latency.
> >>     In response to drops, or too short playback latency, we
> >>     decrease
> >>     the bit rate.
> >>     In response to reports that suggest that the client playback
> >>     is
> >>     stable with the
> >>     current configuration, we try to increase the bit-rate.
> >> 2) server drops: the bit-rate is decreased when server drops
> >> occur.
> >>
> >> Each time the bit rate changes, the stream parameters are
> >> re-evaluated.
> >> In addition, we monitor the frames' encoding size, and when there
> >> is
> >> a change
> >> that may allow improving the stream parameters, or alternatively,
> >> requires decreasing the
> >> quality, we again re-evaluate them.
> >
> >
> > What kind of metrics do we expose - how can I get the bitrate back
> > on the client and/or server?
> >
> The server side estimates the latency, and an optimal bit rate for
> the
> video playback, which is dynamically modified throughout the
> playback.
> This measurements are kept on the server (they are documented in the
> log
> file if you run with the highest debug level).


I think we should consider exposing these through at least through qemu monitor/libvirt on the host side but also what we want to expose into the client through the guest agent.



> >>
> >> Other changes:
> >> --------------
> >>
> >> Besides the client reports, I also added to the protocol a message
> >> that controls the
> >> audio playback latency, for allowing better synchronization of the
> >> audio and video playback buffering.
> >>
> >> The roundtrip time is used for estimating the required playback
> >> delay. In order to get a more accurate estimation
> >> of the roundtrip time I also added an option to measure it
> >> periodically instead of just on startup, and
> >> take the minimum measurement as estimation.
> >>
> >> Results
> >> -------
> >> I compared the video quality of the current spice master, and of
> >> the
> >> new spice, under different network setups.
> >> Spice master was a bit modified for making the comparison more
> >> fair:
> >> I increased the audio jitter buffer to 200ms (instead of 100),
> >> and also included the patch "red_worker: stream agent - fix
> >> miscounting of frames".
> >> The network setup was emulated using tc.
> >>
> >> You can find the tests details and the results in a following
> >> email.
> >>
> >> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than
> >> 70%
> >> of the frames that are sent to the client are being dropped, and
> >> the
> >> video
> >> is unwatchable. With new spice, while the average frame rate is
> >> about
> >> the same, only about 2% of the frames are being dropped by the
> >> client.
> >> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for
> >> spice-master, and the drops rate reaches 90%. For the new spice,
> >> it
> >> is less then 20%, and
> >> the video is watchable.
> >>
> >> I also tested a setup of 10Mbps with high latency (170ms, Test3).
> >> The
> >> latency affects the initial bit rate estimation in spice (probably
> >> due to the tcp acks overhead).
> >> Thus, the stream is started with a bit-rate estimation of less
> >> then
> >> 1.25Mbps. The adaptive video heuristic gradually converges to a
> >> higher bit rate (the column "end-bit-rate"), and
> >> the next video stream will be started with the improved bit rate
> >> estimation.
> >> In Test5 I tested a real environment with a network setup similar
> >> to
> >> Test3. However, the test are not comparable because in Test5 setup
> >> (different server and guest),
> >> the basic frame rate (i.e., from the guest to the server) is much
> >> smaller (still need to investigate why).
> >>
> >> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited
> >> setup,
> >> i.e., a setup which will allow the best frame rate and
> >> jpeg-quality
> >> for the stream.
> >> With new spice, the capacity of the channel is exploited
> >> efficiently.
> >> With spice-master, the condition for dropping frames according to
> >> the defined fps is too strict,
> >> and the observed frame rate is smaller then the maximum possible.
> >>
> >> Video streaming short-term TODO:
> >> ----------------
> >> - Implement playback-latency adjustments for spice-gtk gstreamer
> >> front-end.
> >> - Add vp8 encoding
> >> - Solve some problems we have with video identification.
> >> - Try to achieve faster convergence to the "right" bit-rate when
> >> we
> >> start with a wrong estimation.
> >>
> >> long-term TODO:
> >> ---------------
> >> video pass-through
> >>
> >> Regards,
> >> Yonit.
> >>
> >> Yonit Halperin (28):
> >>    red_worker: stream agent - fix miscounting of frames
> >>    server/red_worker: streams: moving mjpeg_encoder from Stream to
> >>      StreamAgent
> >>    mjpeg_encoder: configure mjpeg quality and frame rate according
> >>    to
> >>    a
> >>      given bit rate
> >>    mjpeg_encoder: re-configure stream parameters when the frame's
> >>      encoding size changes
> >>    mjpeg_encoder: adjust the stream bit rate based on periodic
> >>    client
> >>      feedback
> >>    mjpeg_encoder: modify stream bit rate based on server side pipe
> >>      congestion
> >>    mjpeg_encoder: update the client with estimations for the
> >>    required
> >>      playback latency
> >>    mjpeg_encoder: move the control over frame drops to
> >>    mjpeg_encoder
> >>    mjpeg_encoder: keep the average observed fps similar to the
> >>    defined
> >>      fps
> >>    mjpeg_encoder: add stream warmup time, in which we avoid server
> >>    and
> >>      client drops
> >>    server: spice_timer_queue
> >>    server/red_worker: assign timer callbacks to worker_core, using
> >>      spice_timer_queue
> >>    red_channel: monitor connection latency using MSG_PING
> >>    red_worker: stream - update periodically the input frame rate
> >>    server/red_worker: enable latency monitoring in the display
> >>    channel
> >>    red_worker: start using mjpeg_encoder rate control capabilities
> >>    red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
> >>    red_worker: notify mjpeg_encoder on server frame drops
> >>    red_worker: ignoring video frame drops that are not due to pipe
> >>      congestion
> >>    dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define
> >>    DISPATCHER_H
> >>    snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
> >>    reds: support mm_time latency adjustments
> >>    red_worker: video streams - adjust client playback latency
> >>    server/red_worker.c: use the bit rate of old streams as a start
> >>    point
> >>      for new streams
> >>    server/red_worker: add an option to supply the bandwidth via
> >>    env
> >>    var
> >>    collect and print video stream statistics
> >>    red_worker: increase the interval limit for stream frames
> >>    red_worker: assign mm_time to vga frames
> >>
> >>   server/Makefile.am         |   2 +
> >>   server/dispatcher.h        |   6 +-
> >>   server/inputs_channel.c    |   1 +
> >>   server/main_channel.c      |   7 +-
> >>   server/main_channel.h      |   1 +
> >>   server/main_dispatcher.c   |  32 ++
> >>   server/main_dispatcher.h   |   2 +
> >>   server/mjpeg_encoder.c     | 981
> >>   ++++++++++++++++++++++++++++++++++++++++++++-
> >>   server/mjpeg_encoder.h     |  70 +++-
> >>   server/red_channel.c       | 228 +++++++++++
> >>   server/red_channel.h       |  18 +
> >>   server/red_dispatcher.c    |   1 +
> >>   server/red_worker.c        | 514 +++++++++++++++++++-----
> >>   server/reds-private.h      |   2 +
> >>   server/reds.c              |  28 +-
> >>   server/reds.h              |   2 +
> >>   server/smartcard.c         |   1 +
> >>   server/snd_worker.c        |  45 +++
> >>   server/snd_worker.h        |   2 +
> >>   server/spice_timer_queue.c | 268 +++++++++++++
> >>   server/spice_timer_queue.h |  43 ++
> >>   server/spicevmc.c          |   1 +
> >>   spice-common               |   2 +-
> >>   23 files changed, 2148 insertions(+), 109 deletions(-)
> >>   create mode 100644 server/spice_timer_queue.c
> >>   create mode 100644 server/spice_timer_queue.h
> >>
> >> --
> >> 1.8.1
> >>
> >> _______________________________________________
> >> Spice-devel mailing list
> >> Spice-devel@xxxxxxxxxxxxxxxxxxxxx
> >> http://lists.freedesktop.org/mailman/listinfo/spice-devel
> >>
> 
> 
_______________________________________________
Spice-devel mailing list
Spice-devel@xxxxxxxxxxxxxxxxxxxxx
http://lists.freedesktop.org/mailman/listinfo/spice-devel


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]     [Monitors]