Hi,
The following patch series introduces adaptive video streaming to spice.
Until now, the mjpeg quality was constant (70), and the frame rate was modified
according to the rate of frame drops in the server side (a drop occurs when a new frame reaches the server while
an older frame is still queued in the pipe). In the client side, the video playback is synchronized according
to the audio playback (each audio and video frame holds an mm-time field). The jitter-buffer size in the client
was constant as well - 100 ms. When video frames arrive late to the client (i.e., when the audio playback is ahead of them),
they are dropped.
The adaptive video streaming is implemented by the following heuristic:
Given a bit rate, we calculate the best combination of mjpeg quality and frame rate (henceforth, the stream parameters) for this
bit rate. In order to decide this combination, we evaluate the encoding size for different jpeg
qualities by applying them on successive frames.
Every new stream is assigned with an initial bit rate. The bit rate is re-estimated and
modified during the stream life time. The bit-rate is modified based on:
1) periodic reports from the client:
The client reports includes information about drops and the playback latency.
In response to drops, or too short playback latency, we decrease the bit rate.
In response to reports that suggest that the client playback is stable with the
current configuration, we try to increase the bit-rate.
2) server drops: the bit-rate is decreased when server drops occur.
Each time the bit rate changes, the stream parameters are re-evaluated.
In addition, we monitor the frames' encoding size, and when there is a change
that may allow improving the stream parameters, or alternatively, requires decreasing the
quality, we again re-evaluate them.
Other changes:
--------------
Besides the client reports, I also added to the protocol a message that controls the
audio playback latency, for allowing better synchronization of the audio and video playback buffering.
The roundtrip time is used for estimating the required playback delay. In order to get a more accurate estimation
of the roundtrip time I also added an option to measure it periodically instead of just on startup, and
take the minimum measurement as estimation.
Results
-------
I compared the video quality of the current spice master, and of the new spice, under different network setups.
Spice master was a bit modified for making the comparison more fair: I increased the audio jitter buffer to 200ms (instead of 100),
and also included the patch "red_worker: stream agent - fix miscounting of frames".
The network setup was emulated using tc.
You can find the tests details and the results in a following email.
For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70% of the frames that are sent to the client are being dropped, and the video
is unwatchable. With new spice, while the average frame rate is about the same, only about 2% of the frames are being dropped by the client.
For 2.5Mbps and 60ms (Test2), as expected, things gets worse for spice-master, and the drops rate reaches 90%. For the new spice, it is less then 20%, and
the video is watchable.
I also tested a setup of 10Mbps with high latency (170ms, Test3). The latency affects the initial bit rate estimation in spice (probably due to the tcp acks overhead).
Thus, the stream is started with a bit-rate estimation of less then 1.25Mbps. The adaptive video heuristic gradually converges to a higher bit rate (the column "end-bit-rate"), and
the next video stream will be started with the improved bit rate estimation.
In Test5 I tested a real environment with a network setup similar to Test3. However, the test are not comparable because in Test5 setup (different server and guest),
the basic frame rate (i.e., from the guest to the server) is much smaller (still need to investigate why).
In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup, i.e., a setup which will allow the best frame rate and jpeg-quality for the stream.
With new spice, the capacity of the channel is exploited efficiently. With spice-master, the condition for dropping frames according to the defined fps is too strict,
and the observed frame rate is smaller then the maximum possible.
Video streaming short-term TODO:
----------------
- Implement playback-latency adjustments for spice-gtk gstreamer front-end.
- Add vp8 encoding
- Solve some problems we have with video identification.
- Try to achieve faster convergence to the "right" bit-rate when we start with a wrong estimation.
long-term TODO:
---------------
video pass-through
Regards,
Yonit.
Yonit Halperin (28):
red_worker: stream agent - fix miscounting of frames
server/red_worker: streams: moving mjpeg_encoder from Stream to
StreamAgent
mjpeg_encoder: configure mjpeg quality and frame rate according to a
given bit rate
mjpeg_encoder: re-configure stream parameters when the frame's
encoding size changes
mjpeg_encoder: adjust the stream bit rate based on periodic client
feedback
mjpeg_encoder: modify stream bit rate based on server side pipe
congestion
mjpeg_encoder: update the client with estimations for the required
playback latency
mjpeg_encoder: move the control over frame drops to mjpeg_encoder
mjpeg_encoder: keep the average observed fps similar to the defined
fps
mjpeg_encoder: add stream warmup time, in which we avoid server and
client drops
server: spice_timer_queue
server/red_worker: assign timer callbacks to worker_core, using
spice_timer_queue
red_channel: monitor connection latency using MSG_PING
red_worker: stream - update periodically the input frame rate
server/red_worker: enable latency monitoring in the display channel
red_worker: start using mjpeg_encoder rate control capabilities
red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
red_worker: notify mjpeg_encoder on server frame drops
red_worker: ignoring video frame drops that are not due to pipe
congestion
dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define DISPATCHER_H
snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
reds: support mm_time latency adjustments
red_worker: video streams - adjust client playback latency
server/red_worker.c: use the bit rate of old streams as a start point
for new streams
server/red_worker: add an option to supply the bandwidth via env var
collect and print video stream statistics
red_worker: increase the interval limit for stream frames
red_worker: assign mm_time to vga frames
server/Makefile.am | 2 +
server/dispatcher.h | 6 +-
server/inputs_channel.c | 1 +
server/main_channel.c | 7 +-
server/main_channel.h | 1 +
server/main_dispatcher.c | 32 ++
server/main_dispatcher.h | 2 +
server/mjpeg_encoder.c | 981 ++++++++++++++++++++++++++++++++++++++++++++-
server/mjpeg_encoder.h | 70 +++-
server/red_channel.c | 228 +++++++++++
server/red_channel.h | 18 +
server/red_dispatcher.c | 1 +
server/red_worker.c | 514 +++++++++++++++++++-----
server/reds-private.h | 2 +
server/reds.c | 28 +-
server/reds.h | 2 +
server/smartcard.c | 1 +
server/snd_worker.c | 45 +++
server/snd_worker.h | 2 +
server/spice_timer_queue.c | 268 +++++++++++++
server/spice_timer_queue.h | 43 ++
server/spicevmc.c | 1 +
spice-common | 2 +-
23 files changed, 2148 insertions(+), 109 deletions(-)
create mode 100644 server/spice_timer_queue.c
create mode 100644 server/spice_timer_queue.h
--
1.8.1
_______________________________________________
Spice-devel mailing list
Spice-devel@xxxxxxxxxxxxxxxxxxxxx
http://lists.freedesktop.org/mailman/listinfo/spice-devel