> On 04/17/2013 10:47 AM, Alon Levy wrote: > > On Mon, 2013-04-15 at 14:54 -0400, Yonit Halperin wrote: > >> Hi, > >> On 04/14/2013 09:37 AM, Alon Levy wrote: > >>> On Tue, Feb 26, 2013 at 01:03:46PM -0500, Yonit Halperin wrote: > >>> > >>> ACK series, sorry for the delay. I have to admit I don't understand the > >>> first patches as well as I should, but seeing as they have been tested > >>> and that I would just be delaying them further, I prefer to let you push > >>> them. I hope to work a bit on testability of streams that doesn't > >>> require creating a vm, but of course that too is just a note and not > >>> related to acking this series. > >>> > >> Thanks for reviewing. > >> Anyone care to also review the spice-gtk,spice-common and spice-protocol > >> patches? > > > > I think it's best I review those too. > > > Cool, Thanks. ACK series including common, protocol & gtk, just sent a couple of small notes but looks good. Pulseaudio corking change (6/8) maybe someone else wants to take a look, I'm not sure about what your comment actually means. > >> > >> Thanks, > >> Yonit. > >> > >>>> Hi, > >>>> > >>>> The following patch series introduces adaptive video streaming to spice. > >>>> > >>>> Until now, the mjpeg quality was constant (70), and the frame rate was > >>>> modified > >>>> according to the rate of frame drops in the server side (a drop occurs > >>>> when a new frame reaches the server while > >>>> an older frame is still queued in the pipe). In the client side, the > >>>> video playback is synchronized according > >>>> to the audio playback (each audio and video frame holds an mm-time > >>>> field). The jitter-buffer size in the client > >>>> was constant as well - 100 ms. When video frames arrive late to the > >>>> client (i.e., when the audio playback is ahead of them), > >>>> they are dropped. > >>>> > >>>> The adaptive video streaming is implemented by the following heuristic: > >>>> Given a bit rate, we calculate the best combination of mjpeg quality and > >>>> frame rate (henceforth, the stream parameters) for this > >>>> bit rate. In order to decide this combination, we evaluate the encoding > >>>> size for different jpeg > >>>> qualities by applying them on successive frames. > >>>> Every new stream is assigned with an initial bit rate. The bit rate is > >>>> re-estimated and > >>>> modified during the stream life time. The bit-rate is modified based on: > >>>> 1) periodic reports from the client: > >>>> The client reports includes information about drops and the > >>>> playback latency. > >>>> In response to drops, or too short playback latency, we decrease > >>>> the bit rate. > >>>> In response to reports that suggest that the client playback is > >>>> stable with the > >>>> current configuration, we try to increase the bit-rate. > >>>> 2) server drops: the bit-rate is decreased when server drops occur. > >>>> > >>>> Each time the bit rate changes, the stream parameters are re-evaluated. > >>>> In addition, we monitor the frames' encoding size, and when there is a > >>>> change > >>>> that may allow improving the stream parameters, or alternatively, > >>>> requires decreasing the > >>>> quality, we again re-evaluate them. > >>>> > >>>> Other changes: > >>>> -------------- > >>>> > >>>> Besides the client reports, I also added to the protocol a message that > >>>> controls the > >>>> audio playback latency, for allowing better synchronization of the audio > >>>> and video playback buffering. > >>>> > >>>> The roundtrip time is used for estimating the required playback delay. > >>>> In order to get a more accurate estimation > >>>> of the roundtrip time I also added an option to measure it periodically > >>>> instead of just on startup, and > >>>> take the minimum measurement as estimation. > >>>> > >>>> Results > >>>> ------- > >>>> I compared the video quality of the current spice master, and of the new > >>>> spice, under different network setups. > >>>> Spice master was a bit modified for making the comparison more fair: I > >>>> increased the audio jitter buffer to 200ms (instead of 100), > >>>> and also included the patch "red_worker: stream agent - fix miscounting > >>>> of frames". > >>>> The network setup was emulated using tc. > >>>> > >>>> You can find the tests details and the results in a following email. > >>>> > >>>> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70% of > >>>> the frames that are sent to the client are being dropped, and the video > >>>> is unwatchable. With new spice, while the average frame rate is about > >>>> the same, only about 2% of the frames are being dropped by the client. > >>>> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for > >>>> spice-master, and the drops rate reaches 90%. For the new spice, it is > >>>> less then 20%, and > >>>> the video is watchable. > >>>> > >>>> I also tested a setup of 10Mbps with high latency (170ms, Test3). The > >>>> latency affects the initial bit rate estimation in spice (probably due > >>>> to the tcp acks overhead). > >>>> Thus, the stream is started with a bit-rate estimation of less then > >>>> 1.25Mbps. The adaptive video heuristic gradually converges to a higher > >>>> bit rate (the column "end-bit-rate"), and > >>>> the next video stream will be started with the improved bit rate > >>>> estimation. > >>>> In Test5 I tested a real environment with a network setup similar to > >>>> Test3. However, the test are not comparable because in Test5 setup > >>>> (different server and guest), > >>>> the basic frame rate (i.e., from the guest to the server) is much > >>>> smaller (still need to investigate why). > >>>> > >>>> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup, > >>>> i.e., a setup which will allow the best frame rate and jpeg-quality for > >>>> the stream. > >>>> With new spice, the capacity of the channel is exploited efficiently. > >>>> With spice-master, the condition for dropping frames according to the > >>>> defined fps is too strict, > >>>> and the observed frame rate is smaller then the maximum possible. > >>>> > >>>> Video streaming short-term TODO: > >>>> ---------------- > >>>> - Implement playback-latency adjustments for spice-gtk gstreamer > >>>> front-end. > >>>> - Add vp8 encoding > >>>> - Solve some problems we have with video identification. > >>>> - Try to achieve faster convergence to the "right" bit-rate when we > >>>> start with a wrong estimation. > >>>> > >>>> long-term TODO: > >>>> --------------- > >>>> video pass-through > >>>> > >>>> Regards, > >>>> Yonit. > >>>> > >>>> Yonit Halperin (28): > >>>> red_worker: stream agent - fix miscounting of frames > >>>> server/red_worker: streams: moving mjpeg_encoder from Stream to > >>>> StreamAgent > >>>> mjpeg_encoder: configure mjpeg quality and frame rate according to a > >>>> given bit rate > >>>> mjpeg_encoder: re-configure stream parameters when the frame's > >>>> encoding size changes > >>>> mjpeg_encoder: adjust the stream bit rate based on periodic client > >>>> feedback > >>>> mjpeg_encoder: modify stream bit rate based on server side pipe > >>>> congestion > >>>> mjpeg_encoder: update the client with estimations for the required > >>>> playback latency > >>>> mjpeg_encoder: move the control over frame drops to mjpeg_encoder > >>>> mjpeg_encoder: keep the average observed fps similar to the defined > >>>> fps > >>>> mjpeg_encoder: add stream warmup time, in which we avoid server and > >>>> client drops > >>>> server: spice_timer_queue > >>>> server/red_worker: assign timer callbacks to worker_core, using > >>>> spice_timer_queue > >>>> red_channel: monitor connection latency using MSG_PING > >>>> red_worker: stream - update periodically the input frame rate > >>>> server/red_worker: enable latency monitoring in the display channel > >>>> red_worker: start using mjpeg_encoder rate control capabilities > >>>> red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT > >>>> red_worker: notify mjpeg_encoder on server frame drops > >>>> red_worker: ignoring video frame drops that are not due to pipe > >>>> congestion > >>>> dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define DISPATCHER_H > >>>> snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY > >>>> reds: support mm_time latency adjustments > >>>> red_worker: video streams - adjust client playback latency > >>>> server/red_worker.c: use the bit rate of old streams as a start > >>>> point > >>>> for new streams > >>>> server/red_worker: add an option to supply the bandwidth via env var > >>>> collect and print video stream statistics > >>>> red_worker: increase the interval limit for stream frames > >>>> red_worker: assign mm_time to vga frames > >>>> > >>>> server/Makefile.am | 2 + > >>>> server/dispatcher.h | 6 +- > >>>> server/inputs_channel.c | 1 + > >>>> server/main_channel.c | 7 +- > >>>> server/main_channel.h | 1 + > >>>> server/main_dispatcher.c | 32 ++ > >>>> server/main_dispatcher.h | 2 + > >>>> server/mjpeg_encoder.c | 981 > >>>> ++++++++++++++++++++++++++++++++++++++++++++- > >>>> server/mjpeg_encoder.h | 70 +++- > >>>> server/red_channel.c | 228 +++++++++++ > >>>> server/red_channel.h | 18 + > >>>> server/red_dispatcher.c | 1 + > >>>> server/red_worker.c | 514 +++++++++++++++++++----- > >>>> server/reds-private.h | 2 + > >>>> server/reds.c | 28 +- > >>>> server/reds.h | 2 + > >>>> server/smartcard.c | 1 + > >>>> server/snd_worker.c | 45 +++ > >>>> server/snd_worker.h | 2 + > >>>> server/spice_timer_queue.c | 268 +++++++++++++ > >>>> server/spice_timer_queue.h | 43 ++ > >>>> server/spicevmc.c | 1 + > >>>> spice-common | 2 +- > >>>> 23 files changed, 2148 insertions(+), 109 deletions(-) > >>>> create mode 100644 server/spice_timer_queue.c > >>>> create mode 100644 server/spice_timer_queue.h > >>>> > >>>> -- > >>>> 1.8.1 > >>>> > >>>> _______________________________________________ > >>>> Spice-devel mailing list > >>>> Spice-devel@xxxxxxxxxxxxxxxxxxxxx > >>>> http://lists.freedesktop.org/mailman/listinfo/spice-devel > >> > > > > > > _______________________________________________ Spice-devel mailing list Spice-devel@xxxxxxxxxxxxxxxxxxxxx http://lists.freedesktop.org/mailman/listinfo/spice-devel