Hi,
On 02/26/2013 01:40 PM, Marc-André Lureau wrote:
Hi
----- Mensaje original -----
The adaptive video streaming is implemented by the following
heuristic:
Given a bit rate, we calculate the best combination of mjpeg quality
and frame rate (henceforth, the stream parameters) for this
bit rate. In order to decide this combination, we evaluate the
encoding size for different jpeg
qualities by applying them on successive frames.
But no downscaling?
Good point. However, the default libjpeg sampling factors values are
already 2 for luminance components and 1 for chrominance components
(both horizontal and vertical), while 4 is the highest value, and larger
factor means higher resolution. It is worth trying decreasing the
luminance to 1.
Every new stream is assigned with an initial bit rate. The bit rate
is re-estimated and
modified during the stream life time. The bit-rate is modified based
on:
1) periodic reports from the client:
The client reports includes information about drops and the
playback latency.
In response to drops, or too short playback latency, we decrease
the bit rate.
In response to reports that suggest that the client playback is
stable with the
current configuration, we try to increase the bit-rate.
How is this going to work with multiple client?
Each client have a separate rate-controller and a different stream. Ages
ago, we (alon an I) talked about clustering clients according to their
network setup, and encode each message only once per cluster. I think
that when we have such mechanism, we can use one rate controller per
cluster, and then we can base our estimations on the "weakest" client in
the cluster.
2) server drops: the bit-rate is decreased when server drops occur.
Each time the bit rate changes, the stream parameters are
re-evaluated.
In addition, we monitor the frames' encoding size, and when there is
a change
that may allow improving the stream parameters, or alternatively,
requires decreasing the
quality, we again re-evaluate them.
Results
-------
I compared the video quality of the current spice master, and of the
new spice, under different network setups.
Spice master was a bit modified for making the comparison more fair:
I increased the audio jitter buffer to 200ms (instead of 100),
and also included the patch "red_worker: stream agent - fix
miscounting of frames".
The network setup was emulated using tc.
You can find the tests details and the results in a following email.
For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70%
of the frames that are sent to the client are being dropped, and the
video
is unwatchable. With new spice, while the average frame rate is about
the same, only about 2% of the frames are being dropped by the
client.
For 2.5Mbps and 60ms (Test2), as expected, things gets worse for
spice-master, and the drops rate reaches 90%. For the new spice, it
is less then 20%, and
the video is watchable.
Really nice!
I also tested a setup of 10Mbps with high latency (170ms, Test3). The
latency affects the initial bit rate estimation in spice (probably
due to the tcp acks overhead).
Thus, the stream is started with a bit-rate estimation of less then
1.25Mbps. The adaptive video heuristic gradually converges to a
higher bit rate (the column "end-bit-rate"), and
the next video stream will be started with the improved bit rate
estimation.
In Test5 I tested a real environment with a network setup similar to
Test3. However, the test are not comparable because in Test5 setup
(different server and guest),
the basic frame rate (i.e., from the guest to the server) is much
smaller (still need to investigate why).
In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup,
i.e., a setup which will allow the best frame rate and jpeg-quality
for the stream.
With new spice, the capacity of the channel is exploited efficiently.
With spice-master, the condition for dropping frames according to
the defined fps is too strict,
and the observed frame rate is smaller then the maximum possible.
Video streaming short-term TODO:
----------------
- Implement playback-latency adjustments for spice-gtk gstreamer
front-end.
Shouldn't be too hard. Is it working fine with pulse backend already? Perhaps that will be answer with another patch series :)
I tried using setting the pulse prebuf and tlength, but it didn't work
so well, so instead, I corked and uncorked the playback explicitly. Once
this series is complete and pushed, I'll work on gstreamer as well.
- Add vp8 encoding
Video codec is really bad, it will really take a lot of CPU, unless it can be done on GPU?
I haven't tried it yet, is it even worse than the libjpeg-turbo
performance? For high resolution videos, the cpu usage for libjpeg-turbo
is very high.
About hardware acceleration (at least for decoding):
http://en.wikipedia.org/wiki/WebM#Hardware
- Solve some problems we have with video identification.
Do you mean detecting video regions?
Yes, it's not as good as it used to be, mainly for flash and html5.
Still need to track down all the reasons.
- Try to achieve faster convergence to the "right" bit-rate when we
start with a wrong estimation.
Tricky question, voip application have similar problems, as well as adaptive bandwidth clients (SS/HLS/dash). RTP uses TFRC for smoothing bandwidth, could be an idea to discuss with people on #farstream for example.
Thanks for the references!
long-term TODO:
---------------
video pass-through
This one is hard to prioritize, on the one hand the benefits for server and quality is huge, otoh it won't probably just work in many cases, especially if the final composition isn't done on client side.
Yes, it is not that trivial: several guest applications to support;
dependency on the client platform; proprietary codecs; etc.
Cheers,
Yonit.
_______________________________________________
Spice-devel mailing list
Spice-devel@xxxxxxxxxxxxxxxxxxxxx
http://lists.freedesktop.org/mailman/listinfo/spice-devel