Hi, Update 05.06.2024: I'm resending the patch because now the virtio mailing lists seem to be functional again. This time without the RFC mark. I think it is already good enough for merging. I'm working on updating the implementation. If there are any minor changes necessary, we still have a lot of time before the next release to fix them. This is the 9th version of virtio-video device patch. There are some major changes and now I think it is good enough conceptually, but may need a few minor changes, when the driver gets updated. Sorry, it took way longer, than I expected. There are a lot of details to care about and also a lot of distractions. Please review. Changelog v8 -> v9: 1. Chenged representation of device capabilities/parameters to Type-Length-Value (TLV). TLV representation is already used in the kernel, see [1][2]. It is mentioned in the virtio sound device specification. It is also compatible with struct v4l2_ext_control, that is used to set V4L2 controls in batches. Here I'm trying to extend the approach onto many other things: device capabilities, formats, buffers/resources, crop settings. This way it is possible to get all the device capabilities in one call, then set all the parameters/controls in one call. For us this is important due to significant hypercall delays. I hope this might be applicable in the V4L2 core as well in the future, so I'm very interested in comments from linux-media community. 2. Changed back the representation of capabilities to pre v7 model. Video decoders and encoders are basically converters between multidimensional shapes of all supported encoded formats and all supported raw formats. This was roughly the way the capabilities were represented in v1-v6 (excluding the controls), then in v7 and v8 I tried to do it closer to the V4L2 discovery process. Now I think this was a mistake, the previously used method may look more complex, but it fits better. This approach is similar to the way the capabilities are described in gstreamer (they don't seem to have explicit links between the sets though). 3. Replaced the internal device's per stream resource queues with virtqueues. The idea briefly: we had to avoid blocking the descriptors for a long time, so I implemented delayed responses in v8. Then there is no natural limit for the number of elements in the resource queues anymore. What to do if the queue becomes full? So there are a bunch of things to define in the spec. I decided to replace these queues with virtqueues, because the virtqueues are already defined well enough. There are of course some upsides and downsides. I think upsides are overwhelming and I'd be happy to discuss this if anyone is interested. 4. Renamed commandq to mainq since commands can be queued to per stream queues as well. 5. Renamed STREAM_CREATE -> STREAM_OPEN, STREAM_DESTROY -> STREAM_CLOSE. Now the per stream virtqueues are opened rather than created. 6. Enabled RESET on the output queue too. I didn't know how to handle a VIDIOC_STREAMOFF on the capture queue coming from guest user-space otherwise. 7. Merged ATTACH_BACKING command into SET_PARAMS. I felt, that it really belongs to the parameters alongside the num_resources, it allows the driver to see if the resources are detached after a SET_PARAMS or not, also might be helpful during debugging. Still there is an issue: SG lists can be quite large. If it is large and the results are returned on the eventq, then this significantly increases the eventq's descriptor sizes. Avoided this by adding a requirement for the device to cap the TLV length to 8. 8. Added DRM format modifiers. 9. Added FWHT codec. 10. Addressed review comments. 11. The spec size went down from 24 pages to 19 pages. Full PDF: https://drive.google.com/file/d/1gj-kOBp1uvCB5IzrtOTjf-zunP3f-EFq/view?usp=sharing PDF with the video section only: https://drive.google.com/file/d/1fH8XaSYkwkb2beB9thS7FOvFysUM22Q5/view?usp=sharing Please note, that external links are broken in the short version. [1] https://docs.kernel.org/userspace-api/netlink/intro.html?highlight=tlv [2] https://docs.kernel.org/sound/kernel-api/writing-an-alsa-driver.html?highlight=tlv [3] https://lore.kernel.org/linux-media/20230922100303.GF19112@xxxxxxxxxxxxxxxxxxxxxxxxxx/ Alexander Gordeev (1): virtio-video: Add virtio video device specification conformance.tex | 4 + content.tex | 1 + device-types/video/description.tex | 1607 +++++++++++++++++++++ device-types/video/device-conformance.tex | 22 + device-types/video/driver-conformance.tex | 20 + introduction.tex | 21 + 6 files changed, 1675 insertions(+) create mode 100644 device-types/video/description.tex create mode 100644 device-types/video/device-conformance.tex create mode 100644 device-types/video/driver-conformance.tex -- 2.40.1
