----- Original Message ----- > From: "Yonit Halperin" <yhalperi@xxxxxxxxxx> > To: "Andrew Cathrow" <acathrow@xxxxxxxxxx> > Cc: spice-devel@xxxxxxxxxxxxxxx > Sent: Thursday, March 7, 2013 3:52:04 PM > Subject: Re: [PATCH spice-server 00/28] adaptive video streaming > > Hi, > On 03/03/2013 07:50 AM, Andrew Cathrow wrote: > > > > > > ----- Original Message ----- > >> From: "Yonit Halperin" <yhalperi@xxxxxxxxxx> > >> To: spice-devel@xxxxxxxxxxxxxxx > >> Sent: Tuesday, February 26, 2013 1:03:46 PM > >> Subject: [PATCH spice-server 00/28] adaptive video > >> streaming > >> > >> Hi, > >> > >> The following patch series introduces adaptive video streaming to > >> spice. > >> > >> Until now, the mjpeg quality was constant (70), and the frame rate > >> was modified > >> according to the rate of frame drops in the server side (a drop > >> occurs when a new frame reaches the server while > >> an older frame is still queued in the pipe). In the client side, > >> the > >> video playback is synchronized according > >> to the audio playback (each audio and video frame holds an mm-time > >> field). The jitter-buffer size in the client > >> was constant as well - 100 ms. When video frames arrive late to > >> the > >> client (i.e., when the audio playback is ahead of them), > >> they are dropped. > >> > >> The adaptive video streaming is implemented by the following > >> heuristic: > >> Given a bit rate, we calculate the best combination of mjpeg > >> quality > >> and frame rate (henceforth, the stream parameters) for this > >> bit rate. In order to decide this combination, we evaluate the > >> encoding size for different jpeg > >> qualities by applying them on successive frames. > >> Every new stream is assigned with an initial bit rate. The bit > >> rate > >> is re-estimated and > >> modified during the stream life time. The bit-rate is modified > >> based > >> on: > >> 1) periodic reports from the client: > >> The client reports includes information about drops and the > >> playback latency. > >> In response to drops, or too short playback latency, we > >> decrease > >> the bit rate. > >> In response to reports that suggest that the client playback > >> is > >> stable with the > >> current configuration, we try to increase the bit-rate. > >> 2) server drops: the bit-rate is decreased when server drops > >> occur. > >> > >> Each time the bit rate changes, the stream parameters are > >> re-evaluated. > >> In addition, we monitor the frames' encoding size, and when there > >> is > >> a change > >> that may allow improving the stream parameters, or alternatively, > >> requires decreasing the > >> quality, we again re-evaluate them. > > > > > > What kind of metrics do we expose - how can I get the bitrate back > > on the client and/or server? > > > The server side estimates the latency, and an optimal bit rate for > the > video playback, which is dynamically modified throughout the > playback. > This measurements are kept on the server (they are documented in the > log > file if you run with the highest debug level). I think we should consider exposing these through at least through qemu monitor/libvirt on the host side but also what we want to expose into the client through the guest agent. > >> > >> Other changes: > >> -------------- > >> > >> Besides the client reports, I also added to the protocol a message > >> that controls the > >> audio playback latency, for allowing better synchronization of the > >> audio and video playback buffering. > >> > >> The roundtrip time is used for estimating the required playback > >> delay. In order to get a more accurate estimation > >> of the roundtrip time I also added an option to measure it > >> periodically instead of just on startup, and > >> take the minimum measurement as estimation. > >> > >> Results > >> ------- > >> I compared the video quality of the current spice master, and of > >> the > >> new spice, under different network setups. > >> Spice master was a bit modified for making the comparison more > >> fair: > >> I increased the audio jitter buffer to 200ms (instead of 100), > >> and also included the patch "red_worker: stream agent - fix > >> miscounting of frames". > >> The network setup was emulated using tc. > >> > >> You can find the tests details and the results in a following > >> email. > >> > >> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than > >> 70% > >> of the frames that are sent to the client are being dropped, and > >> the > >> video > >> is unwatchable. With new spice, while the average frame rate is > >> about > >> the same, only about 2% of the frames are being dropped by the > >> client. > >> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for > >> spice-master, and the drops rate reaches 90%. For the new spice, > >> it > >> is less then 20%, and > >> the video is watchable. > >> > >> I also tested a setup of 10Mbps with high latency (170ms, Test3). > >> The > >> latency affects the initial bit rate estimation in spice (probably > >> due to the tcp acks overhead). > >> Thus, the stream is started with a bit-rate estimation of less > >> then > >> 1.25Mbps. The adaptive video heuristic gradually converges to a > >> higher bit rate (the column "end-bit-rate"), and > >> the next video stream will be started with the improved bit rate > >> estimation. > >> In Test5 I tested a real environment with a network setup similar > >> to > >> Test3. However, the test are not comparable because in Test5 setup > >> (different server and guest), > >> the basic frame rate (i.e., from the guest to the server) is much > >> smaller (still need to investigate why). > >> > >> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited > >> setup, > >> i.e., a setup which will allow the best frame rate and > >> jpeg-quality > >> for the stream. > >> With new spice, the capacity of the channel is exploited > >> efficiently. > >> With spice-master, the condition for dropping frames according to > >> the defined fps is too strict, > >> and the observed frame rate is smaller then the maximum possible. > >> > >> Video streaming short-term TODO: > >> ---------------- > >> - Implement playback-latency adjustments for spice-gtk gstreamer > >> front-end. > >> - Add vp8 encoding > >> - Solve some problems we have with video identification. > >> - Try to achieve faster convergence to the "right" bit-rate when > >> we > >> start with a wrong estimation. > >> > >> long-term TODO: > >> --------------- > >> video pass-through > >> > >> Regards, > >> Yonit. > >> > >> Yonit Halperin (28): > >> red_worker: stream agent - fix miscounting of frames > >> server/red_worker: streams: moving mjpeg_encoder from Stream to > >> StreamAgent > >> mjpeg_encoder: configure mjpeg quality and frame rate according > >> to > >> a > >> given bit rate > >> mjpeg_encoder: re-configure stream parameters when the frame's > >> encoding size changes > >> mjpeg_encoder: adjust the stream bit rate based on periodic > >> client > >> feedback > >> mjpeg_encoder: modify stream bit rate based on server side pipe > >> congestion > >> mjpeg_encoder: update the client with estimations for the > >> required > >> playback latency > >> mjpeg_encoder: move the control over frame drops to > >> mjpeg_encoder > >> mjpeg_encoder: keep the average observed fps similar to the > >> defined > >> fps > >> mjpeg_encoder: add stream warmup time, in which we avoid server > >> and > >> client drops > >> server: spice_timer_queue > >> server/red_worker: assign timer callbacks to worker_core, using > >> spice_timer_queue > >> red_channel: monitor connection latency using MSG_PING > >> red_worker: stream - update periodically the input frame rate > >> server/red_worker: enable latency monitoring in the display > >> channel > >> red_worker: start using mjpeg_encoder rate control capabilities > >> red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT > >> red_worker: notify mjpeg_encoder on server frame drops > >> red_worker: ignoring video frame drops that are not due to pipe > >> congestion > >> dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define > >> DISPATCHER_H > >> snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY > >> reds: support mm_time latency adjustments > >> red_worker: video streams - adjust client playback latency > >> server/red_worker.c: use the bit rate of old streams as a start > >> point > >> for new streams > >> server/red_worker: add an option to supply the bandwidth via > >> env > >> var > >> collect and print video stream statistics > >> red_worker: increase the interval limit for stream frames > >> red_worker: assign mm_time to vga frames > >> > >> server/Makefile.am | 2 + > >> server/dispatcher.h | 6 +- > >> server/inputs_channel.c | 1 + > >> server/main_channel.c | 7 +- > >> server/main_channel.h | 1 + > >> server/main_dispatcher.c | 32 ++ > >> server/main_dispatcher.h | 2 + > >> server/mjpeg_encoder.c | 981 > >> ++++++++++++++++++++++++++++++++++++++++++++- > >> server/mjpeg_encoder.h | 70 +++- > >> server/red_channel.c | 228 +++++++++++ > >> server/red_channel.h | 18 + > >> server/red_dispatcher.c | 1 + > >> server/red_worker.c | 514 +++++++++++++++++++----- > >> server/reds-private.h | 2 + > >> server/reds.c | 28 +- > >> server/reds.h | 2 + > >> server/smartcard.c | 1 + > >> server/snd_worker.c | 45 +++ > >> server/snd_worker.h | 2 + > >> server/spice_timer_queue.c | 268 +++++++++++++ > >> server/spice_timer_queue.h | 43 ++ > >> server/spicevmc.c | 1 + > >> spice-common | 2 +- > >> 23 files changed, 2148 insertions(+), 109 deletions(-) > >> create mode 100644 server/spice_timer_queue.c > >> create mode 100644 server/spice_timer_queue.h > >> > >> -- > >> 1.8.1 > >> > >> _______________________________________________ > >> Spice-devel mailing list > >> Spice-devel@xxxxxxxxxxxxxxxxxxxxx > >> http://lists.freedesktop.org/mailman/listinfo/spice-devel > >> > > _______________________________________________ Spice-devel mailing list Spice-devel@xxxxxxxxxxxxxxxxxxxxx http://lists.freedesktop.org/mailman/listinfo/spice-devel