[PATCH v2 1/1] virtio-video: Add virtio video device specification

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



From: Dmitry Sepp <dmitry.sepp@xxxxxxxxxxxxxxx>

The virtio video encoder device and decoder device provide functionalities to
encode and decode video stream respectively.
Though video encoder and decoder are provided as different devices, they use a
same protocol.

Signed-off-by: Dmitry Sepp <dmitry.sepp@xxxxxxxxxxxxxxx>
Signed-off-by: Keiichi Watanabe <keiichiw@xxxxxxxxxxxx>
---
 content.tex      |   1 +
 virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 580 insertions(+)
 create mode 100644 virtio-video.tex

diff --git a/content.tex b/content.tex
index 556b373..9e56839 100644
--- a/content.tex
+++ b/content.tex
@@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-vsock.tex}
 \input{virtio-fs.tex}
 \input{virtio-rpmb.tex}
+\input{virtio-video.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
diff --git a/virtio-video.tex b/virtio-video.tex
new file mode 100644
index 0000000..30e728d
--- /dev/null
+++ b/virtio-video.tex
@@ -0,0 +1,579 @@
+\section{Video Device}\label{sec:Device Types / Video Device}
+
+The virtio video encoder device and decoder device are virtual devices that
+supports encoding and decoding respectively. Though the encoder and the decoder
+are different devices, they use the same protocol.
+
+\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
+
+\begin{description}
+\item[30] encoder device
+\item[31] decoder device
+\end{description}
+
+\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
+
+\begin{description}
+\item[0] controlq - queue for sending control commands.
+\item[1] eventq - queue for sending events happened in the device.
+\end{description}
+
+\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature bits}
+
+\begin{description}
+\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for video
+  buffers.
+\end{description}
+
+\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits}
+
+The device MUST offer at least one of feature bits.
+
+\subsection{Device configuration layout}\label{sec:Device Types / Video Device / Device configuration layout}
+
+Video device configuration uses the following layout structure:
+
+\begin{lstlisting}
+struct virtio_video_config {
+        le32 max_cap_len;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{max_cap_len}] defines the maximum length of a descriptor
+  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
+  MUST set this value.
+\end{description}
+
+\subsection{Device Initialization}\label{sec:Device Types / Video Device / Device Initialization}
+
+\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video Device / Device Initialization}
+
+The driver SHOULD query device capability by using the
+VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
+setup.
+
+\subsection{Device Operation}\label{sec:Device Types / Video Device / Device Operation}
+
+The driver allocates input and output buffers and queues the buffers
+to the device. The device performs operations on the buffers according
+to the function in question.
+
+\subsubsection{Device Operation: Create stream}
+
+To process buffers, the device needs to associate them with a certain
+video stream (essentially, a context). Streams are created by
+VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
+determined by the device.
+
+\subsubsection{Device Operation: Create buffers}
+
+Buffers are used to store the actual data as well as the relevant
+metadata. Scatter lists are supported, so the buffer doesn't need to
+be contiguous in guest physical memory.
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
+  resource that is backed by a buffer allocated from the driver's
+  memory.
+\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
+  is no longer needed.
+\end{itemize*}
+
+\subsubsection{Device Operation: Stream parameter control}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
+  input and output streams from the device.
+\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
+  device.
+\item After setting stream parameters, the driver may issue
+  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can be
+  changed implicitly by the device during the set operation.
+\end{itemize*}
+
+\subsubsection{Device Operation: Process buffers}
+
+\begin{itemize*}
+\item If the function and the buffer type require so, write data to
+the buffer memory.
+\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
+processing in the device.
+\item The request completes asynchronously when the device has
+finished with the buffer.
+\end{itemize*}
+
+\subsubsection{Device Operation: Buffer processing control}
+
+\begin{itemize*}
+\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
+  return all of the already queued buffers.
+\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
+  already queued buffers from the input or the output queue. This also
+  includes input or output buffers that can be currently owned by the
+  device's processing pipeline.
+\end{itemize*}
+
+\subsubsection{Device Operation: Asynchronous events}
+
+While processing buffers, the device can send asynchronous event
+notifications to the driver. The behaviour depends on the exact
+stream. For example, the decoder device sends a resolution change
+event when it encounters new resolution metadata in the stream.
+
+\subsubsection{Device Operation: Request header}
+
+All requests and responses on the control virt queue have a fixed
+header using the following layout structure and definitions:
+
+\begin{lstlisting}
+enum virtio_video_ctrl_type {
+        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
+
+        /* request */
+        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
+        VIRTIO_VIDEO_T_STREAM_CREATE,
+        VIRTIO_VIDEO_T_STREAM_DESTROY,
+        VIRTIO_VIDEO_T_STREAM_DRAIN,
+        VIRTIO_VIDEO_T_RESOURCE_CREATE,
+        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
+        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
+        VIRTIO_VIDEO_T_QUEUE_CLEAR,
+        VIRTIO_VIDEO_T_SET_PARAMS,
+        VIRTIO_VIDEO_T_GET_PARAMS,
+
+        /* response */
+        VIRTIO_VIDEO_S_OK = 0x0200,
+        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
+        VIRTIO_VIDEO_S_OK_GET_PARAMS,
+
+        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
+        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
+        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
+        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
+        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
+};
+
+struct virtio_video_ctrl_hdr {
+        le32 type;
+        le32 stream_id;
+        le32 len; /* Length of the structure in bytes. */
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{type}] is the type of the driver request or the device
+response.
+\item[\field{stream_id}] specifies a target stream.
+\item[\field{len}] is the length of data in bytes, which includes
+length of the header.
+\end{description}
+
+\subsubsection{Device Operation: controlq}
+
+\begin{description}
+
+\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
+supported formats.
+
+The driver uses \field{struct virtio_video_get_capability} to send a
+query request.
+
+\begin{lstlisting}
+enum virtio_video_buf_type {
+        VIRTIO_VIDEO_BUF_TYPE_INPUT,
+        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
+};
+
+struct virtio_video_get_capability {
+        struct virtio_video_ctrl_hdr hdr;
+        enum virtio_video_buf_type buf_type;
+};
+\end{lstlisting}
+\begin{description}
+\item[\field{buf_type}] is the buffer type that the driver asks
+information about. The driver MUST set either
+\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
+\end{description}
+
+The device responds a capability by using \field{struct
+virtio_video_get_capability_resp}.
+\begin{lstlisting}
+enum virtio_video_format {
+        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
+        /* Raw formats */
+        VIRTIO_VIDEO_FORMAT_NV12 = 1,
+        VIRTIO_VIDEO_FORMAT_YUV420,
+        VIRTIO_VIDEO_FORMAT_YVU420,
+
+        /* Compressed formats */
+        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
+        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
+        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
+};
+
+enum virtio_video_profile {
+        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
+
+        /* H.264 */
+        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
+        VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_BASELINE,
+        VIRTIO_VIDEO_PROFILE_H264_MAIN,
+        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
+        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
+        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+        VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+
+        /* VP8 */
+        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
+        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
+        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
+
+        /* VP9 */
+        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
+        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+};
+
+struct virtio_video_format_range {
+        le32 min;
+        le32 max;
+        le32 step;
+        u8 paddings[4];
+};
+
+struct virtio_video_format_desc {
+        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
+        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
+        le64 mask;
+        struct virtio_video_format_range width;
+        struct virtio_video_format_range height;
+        le32 num_rates;
+        u8 padding[4];
+        /* Followed by struct virtio_video_frame_rate frame_rates[] */
+};
+
+struct virtio_video_get_capability_resp {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 num_descs;
+        /* Followed by struct virtio_video_format_desc desc[] */
+};
+\end{lstlisting}
+
+The format description \field{struct virtio_video_format_desc}
+includes the following fields:
+\begin{description}
+\item[\field{format}] specifies an image format. The device MUST set one
+  of \field{enum virtio_video_format}.
+\item[\field{profile}] specifies a profile of the compressed image format
+  specified in \field{format}. The driver SHOULD ignore this value if
+  \field{format} is a raw format.
+\item[\field{mask}] is a bitset that represents the supported
+  combination of input and output format. If \textit{i}-th bit is set
+  in \field{mask} of \textit{j}-th \field{struct
+  virtio_video_format_desc} for input, the device supports encoding or
+  decoding from the \textit{j}-th input format to \textit{i}-th output
+  format.
+\item[\field{width, height}] represents a range of resolutions
+  supported by the device. If its \field{step} is not applicable, its
+  \field{min} is equal to its \field{max}.
+\item[\field{num_rates}] is the length of an array \field{frame_rates}. In case of decoder, the driver SHOULD ignore this value.
+\item[\field{frame_rates}] is an array of supported frame rates.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
+  within the device.
+
+\begin{lstlisting}
+enum virtio_video_mem_type {
+        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
+};
+
+struct virtio_video_stream_create {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+        char debug_name[64];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{in_mem_type}] is a type of buffer management for input
+buffers. The driver MUST set a value in \field{enum
+virtio_video_mem_type}.
+\item[\field{out_mem_type}] is a type of buffer management for output
+buffers. The driver MUST set a value in \field{enum
+virtio_video_mem_type}.
+\item[\field{debug_name}] is a text string for a debug purpose.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
+  within the device.
+
+\begin{lstlisting}
+struct virtio_video_stream_destroy {
+        struct virtio_video_ctrl_hdr hdr;
+};
+\end{lstlisting}
+
+\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
+  queued buffers through the pipeline.
+
+\begin{lstlisting}
+struct virtio_video_stream_drain {
+        struct virtio_video_ctrl_hdr hdr;
+};
+\end{lstlisting}
+
+\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
+  within the device.
+
+\begin{lstlisting}
+struct virtio_video_mem_entry {
+        le64 addr;
+        le32 length;
+        u8 padding[4];
+};
+
+struct virtio_video_resource_create {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 resource_id;
+        le32 nr_entries;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{resource_id}] internal id of the resource.
+\item[\field{nr_entries}] number of \field{struct
+  virtio_video_mem_entry} memory entries.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
+  within the device.
+
+\begin{lstlisting}
+struct virtio_video_resource_destroy {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 resource_id;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{resource_id}] internal id of the resource.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
+queue.
+
+\begin{lstlisting}
+#define VIRTIO_VIDEO_MAX_PLANES 8
+
+struct virtio_video_resource_queue {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 buf_type;
+        le32 resource_id;
+        le64 timestamp;
+        le32 nr_data_size;
+        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{buf_type}] buf_type of the .
+\item[\field{resource_id}] internal id of the resource.
+\item[\field{timestamp}] an abstract sequence counter that can be used
+  for synchronisation.
+\item[\field{nr_data_size}] number of \field{data_size} entries.
+\item[\field{data_size}] number of data bytes within a plane.
+\end{description}
+
+\begin{lstlisting}
+enum virtio_video_buffer_flag {
+        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
+        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
+        /* Encoder only */
+        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
+        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
+        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
+};
+
+struct virtio_video_resource_queue_resp {
+        struct virtio_video_ctrl_hdr hdr;
+        le64 timestamp;
+        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
+        le32 size;  /* Encoded size */
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{timestamp}] an abstract sequence counter that can be used
+  for synchronisation.
+\item[\field{flags}] mark specific buffers in the sequence.
+\item[\field{size}] data size in the buffer (encoder only).
+\end{description}
+
+The device sends a response to the queue request asynchronously when
+it has finished processing the buffer.
+
+The device SHOULD mark a buffer that triggered a processing error with
+the VIRTIO_VIDEO_BUFFER_F_ERR flag.
+
+The device MUST mark the last buffer with the
+VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
+sequence.
+
+In case of encoder, to denote a particular frame type the devie MUST
+mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
+VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
+
+\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
+  buffers back from the input or the output queue of the device. The
+  device SHOULD return all of the buffers from the respective queue as
+  soon as possible without pushing the buffers through the processing
+  pipeline.
+
+\begin{lstlisting}
+struct virtio_video_queue_clear {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 buf_type;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{buf_type}] buffer type.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
+  output of a stream.
+
+\begin{lstlisting}
+struct virtio_video_plane_format {
+        le32 plane_size;
+        le32 stride;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{plane_size}] size of the plane in bytes.
+\item[\field{stride}] stride used for the plane in bytes.
+\end{description}
+
+\begin{lstlisting}
+struct virtio_video_params {
+        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
+        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
+        le32 frame_width;
+        le32 frame_height;
+        le32 min_buffers;
+        le32 max_buffers;
+        le32 frame_rate;
+        struct virtio_video_crop {
+                le32 left;
+                le32 top;
+                le32 width;
+                le32 height;
+        } crop;
+        le32 num_planes;
+        struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{frame_width}] the value to get/set.
+\item[\field{frame_height}] the value to get/set.
+\item[\field{pixel_format}] the value to get/set.
+\item[\field{min_buffers}] minimum buffers required to handle the
+  format (r/o).
+\item[\field{max_buffers}] maximum buffers required to handle the
+  format (r/o).
+\item[\field{frame_rate}] the value to get/set.
+\item[\field{crop}] cropping (composing) rectangle.
+\item[\field{num_planes}] number of planes used to store pixel data
+(r/o).
+\item[\field{plane_formats}] description of each plane.
+\end{description}
+
+\begin{lstlisting}
+struct virtio_video_get_params {
+        struct virtio_video_ctrl_hdr hdr;
+        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
+};
+
+struct virtio_video_get_params_resp {
+        struct virtio_video_ctrl_hdr hdr;
+        struct virtio_video_params params;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{buf_type}] buffer type.
+\item[\field{params}] parameter values.
+\end{description}
+
+\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
+
+
+\begin{lstlisting}
+struct virtio_video_set_params {
+        struct virtio_video_ctrl_hdr hdr;
+        struct virtio_video_params params;
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{params}] parameters to set.
+\end{description}
+
+Setting stream parameters might have side effects within the device.
+For example, the device MAY perform alignment of width and height,
+change the number of planes it uses for the format, or do whatever
+changes that are required to continue normal operation using the
+updated parameters. It is up to the driver to check the parameter set
+after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
+
+\end{description}
+
+\subsubsection{Device Operation: eventq}
+
+The device can report events on the event queue. The driver initially
+populates the queue with device-writeable buffers. When the device
+needs to report an event, it fills a buffer and notifies the driver.
+The driver consumes the report and adds a new buffer to the virtqueue.
+
+\begin{lstlisting}
+enum virtio_video_event_type {
+        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
+        /* For all functions */
+        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
+        /* For decoder only */
+        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
+};
+
+struct virtio_video_event {
+        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
+        le32 stream_id;
+        u8 padding[4];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{event_type}] type of the triggered event .
+\item[\field{stream_id}] id of the source stream.
+\end{description}
+
+The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
+whenever it encounters new resolution data in the stream. This
+includes the case of the initial device configuration after metadata
+has been parsed and the case of dynamic resolution change.
-- 
2.24.1.735.g03f4e72817-goog




[Index of Archives]     [Linux Input]     [Video for Linux]     [Gstreamer Embedded]     [Mplayer Users]     [Linux USB Devel]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [Yosemite Backpacking]

  Powered by Linux