This patch series adds support for using GStreamer to encode and decode the video streams, adding support for VP8 and h264 codecs. As before the patches can also be grabbed from the gst branch of the repositories below: spice: https://github.com/fgouget/spice spice-gtk: https://github.com/fgouget/spice-gtk xf86-video-qxl: https://github.com/fgouget/xf86-video-qxl spice-protocol: https://github.com/fgouget/spice-protocol See also gst-sync for the old spice-gtk code. (there's also 'extras' branches with more experimental/future patches for the curious) Note also that I added gst-v10, gst-v11 and gst-v12 branches to help comparing these revisions. Changes from v12: This round focused on performance and reliability in various special cases such as low bandwidth operation. * The server no longer rebuilds the pipeline from scratch every time the frame size or bitrate changes. The trick to it is to transition to the NULL state (instead of the PAUSED one as before). * I also unified configure_pipeline() and reconfigure_pipeline() by moving all the one-time configuration operations to create_pipeline() and tracking the configuration changes to apply. * The server can now also apply bitrate changes directly if the codec advertises it can be changed in the PLAYING state. This is the case for the x264enc codec. * The server can now detect when the GStreamer pipeline error occurs instead of getting stuck waiting for the compressed buffer. * I fixed the value for the error-resilient vp8enc option. * I removed the resize-allowed=true vp8enc setting as tests showed the decoder was sometimes not restoring a full-size frame. * I removed the 'frame' SpiceGstVideoBuffer field as it was not used. * push_raw_frame() now avoids allocating memory for chunk_copy() if zero_copy() was able to take care of the whole frame. * The client no longer rebuilds or even reconfigure the pipeline from scratch every time the frame size changes. It turns out that the decoders don't care (unlike the encoders). * I have protected the PTS calculation against underflows. This fixes freezes that could occasionally happen with negative latencies near the start of a stream (i.e. in low bandwidth cases). * I renamed SpiceFrameMeta to just SpiceFrame since it also contains the sample once the frame has been decoded, and is thus not just metadata. I also cleaned up the naming of the corresponding queues. * The GStreamer pipeline sometimes returns multiple buffers with the same PTS. The first buffer would remove its SpiceFrame from the decoding_queue so that the next one would not find a match, thus clearing the decoding_queue in the process. As a result the next buffer would not find a match either, again clearing the decoding_queue in the process, etc. So buffers that have an unknown PTS no longer clear the decoding_queue. * When frames are late for display the client now tries to display the least out of date one in the display_queue so there are at least some screen updates. * I removed the qos=true appsink option. It's not clear that it was helping so I decided to remove it until there is more evidence that it is needed. * I renamed stream_dropped_frame() to stream_dropped_frame_on_playback() to make it clearer that it's unrelated to the stream report data (which is based on arrival time). Besides rebases, the changes from v11 are limited to spice-gtk. Specifically they fix two compatibility issues: * v11 used a GAsyncQueue and relied on g_async_queue_push_front() which is only available in GLib >= 2.46 which is unfortunately not available in Debian 8 and presumably on many other Linux distributions. * It also associated the original network message with the corresponding GStreamer buffer using the GstMeta API. This is necessary in order to know where to display the decoded frame as this could change from one frame to the next. However some GStreamer elements have (refcounting) bugs that caused the metadata to be lost in the pipeline. I had a workaround in place that worked on Debian Testing but did not on Debian 8. https://bugzilla.gnome.org/show_bug.cgi?id=757254 So in the end I reworked both by putting the metadata in a regular GQueue and using the PTS timestamp to match elements in that queue to GStreamer's buffers. Patches and changes from v10: server: Check the client video codec capabilities Fixed the case where the server and client cannot agree on the video codec to use. server: Let the administrator pick the video encoder and codec The video_codecs GArray is now considered immutable which avoids copying around and avoids having to make the RED_WORKER_MESSAGE_SET_VIDEO_CODECS RPC synchronous. server: Use the optimal number of threads for VP8 encoding Using the optimal number of cores for the VP8 encoder is now in a separate patch (though skipping it will cause conflicts with patches 14 and 16). server: Avoid copying the input frame in the GStreamer encoder Changed the zero-copy approach to not require ref/unref() of RedDrawable objects to be thread-safe. Thread-safety aspects are instead handled in gstreamer-encoder.c now. server: Adjust the frame rate based on the GStreamer encoding time This patch limits the frame rate (i.e. drops frames) when the server has trouble keeping up with the encoding. This code seems to only be needed in SpiceDeferredFPS mode (so maybe it's in the SpiceDeferredFPS code that something should be modified). In any case this patch can be skipped without any impact on the rest of the series. server: Respect the GStreamer encoder's valid bit rate range set_gstenc_bitrate() should now pass the right integer type to g_object_set(). server: Add support for GStreamer 0.10 Reduced the number of #ifdefs for GStreamer 0.10 using Christophe's suggestions. spice-gtk: Add a GStreamer video decoder for MJPEG, VP8 and h264 - Tweaked some function names in the client's GStreamer decoder to avoid potential name conflicts. - Fixed a race condition in the client which would cause it to freeze when hovering madly over the seek barin YouTube videos. - As a side effect the new code now schedules the frames rending itself so it can adapt when the mm-time get yanked around. - It also does not queue the frames in the pipeline and thus is less likely to lose them when size changes force us to rebuild it. spice-gtk: Avoid GAsyncQueue for compatibility with GLib < 2.46 This patch is optional and avoids the dependency on g_async_queue_push_front() which is missing in GLib < 2.46 (e.g. for Debian 8). -- 2.8.0.rc3 _______________________________________________ Spice-devel mailing list Spice-devel@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/spice-devel