On Thu, 4 Aug 2016, Frediano Ziglio wrote: [...] > In the Perfect World (TM) guest will send the video stream directly to > the client without server using much CPU. Detecting this would require to > have a driver in the client which support streaming (like VAAPI or DirectX VA) > to get the byte stream from the guest. Possibly the guest could read back > the decoded video so in the real world most likely client AND spice-server > would have to do the decoding but this would avoid > decoding (guest) -> encoding (server) -> decoding (client) > for a > byte stream (guest) -> decoding (server) -> decoding (client) Implementing a passthrough streaming mechanism that works well would actually be pretty hard. * First I can confirm that the most popular multimedia applications (Firefox & Chrome's HTML5 video player, Flash, mplayer) use hardware acceleration to decode the video but *not* to display it (if you look for an exception there's Totem). So say you intercept the video stream at the decoder stage and send it to the client. Where should the client display it? * In fact the guest applications could very well not be displaying the decoded video. That's the case when you transcode a video. In that case streaming the video to the client would make no sense. * But let's say the guest application does display the video. Now you must prevent the spice server from sending the corresponding screen updates either through streaming or regular updates. Otherwise you'll end up doubling the bandwidth usage! (and wasting CPU cycles if it is streaming them) * However the video detected by the server may not have the same size as the original stream. This is almost always the case when you display a video fullscreen for instance. Furthermore the spice server often detects a video as two videos (e.g. one with the first 100 lines or so, and another with the bottom 140, causing banding in YouTube for instance). So it seems that matching the video being displayed with the intercepted video stream would require solving that issue. * But it's also possible that parts of the video are simply clipped. Think scrolling a YouTube web page down until the video is only partially visible. Or a dialog popping up in front of the video. Such clipping must not only be detected on the Spice server when attempting to match screen updates with the original video stream, it must also be replicated when the client displays the frames it decoded. Otherwise the video will obscure areas it's not supposed to (e.g. the browser's title bar and tabs). * A variant is that a lot of applications apply overlays to the video, some of them are even translucent: think YouTube ads/comments or movie subtitles. Now you need to send those to the client so it can display them on top the video it decoded. But how do you detect these overlays? It seems like you'll have to decode the video on the server, scale it just right, diff it with the video displayed by the guest application, and send the diff to the client. All that efficiently both from a server CPU usage point of view, and from a network bandwidth one. (otherwise what's the point?) * One way to avoid the Spice server-side issues would be for the decoder to return fake content that's constant, like a green screen. Then the Spice server no longer needs to match screen updates since there are none. But you still have all the issues with transcoding applications, figuring out where to display the video on the client, clipped videos, overlay detection and handling, sending translucency information, etc. * Then there's all the niggly details like expanded codec support, what happens when a passthrough-capable client connects or disconnects while a video is already playing, or what happens when a passthrough-capable client connects at the same time as a non-passthrough-capable one, etc. So unfortunately intercepting the original video stream is the easy part. The hard part is all the rest :-( -- Francois Gouget <fgouget@xxxxxxxxxxxxxxx> _______________________________________________ Spice-devel mailing list Spice-devel@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/spice-devel