Re: [virtio-dev] [FYI] virtio-media published

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Alexandre,

On 08.12.23 07:46, Alexandre Courbot wrote:
Hello everyone,

This is just a heads-up to announce the initial publication of virtio-media:

https://github.com/Gnurou/virtio-media

The name has changed slightly, but this project is essentially the
refinement and continuation of my virtio-v4l2 proposal. The reason for
moving forward with this is, to be candid, the unclear direction and
lack of progress regarding the future of virtio-video, and the
non-consensual way in which its development has been, let's say
"redirected".

Great! I like the new name. The old one was confusing IMO. I'm going to have a closer look later.

I'd be happy to clarify the current progress and the future direction of the virtio-video. I'd like to make these major changes in the next spec draft:

1. Represent the capabilities and parameters with TLVs. Basically I'd like to take the data format from VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS, flatten it by replacing the pointers with the actual data (this would be then in fact a sequence of TLVs). Then use this data format when getting capabilities (all of the VIDIOC_ENUM* and VIDIOC_QUERY* in a single call), getting and setting parameters (parameters + controls in V4L2 terms). Here I'd like to reuse the control ids and value definitions as much as possible. Maybe this data presentation could be used in V4L2 later. I think it would be digestible, as it is an extension of the extended controls. Indeed it was not clear, I even thought device trees would be a great solution at some point (see [1] for the rationale), but several people convinced me, that's not going to be merged in the kernel.

BTW this activity was influenced by your comment in [2]:

Beware of not repeating the same mistake as v5 (all controls under one
big structure).

2. Make the input and output resource queues explicit by turning them into virtio queues. It became clear, that after making the commands non blocking, some of the queue control moves to these internal resource queues. Then I thought: why make another queue implementation, when we already have one? So I think it makes sense to turn them into virtio queues. I still would like to keep the delayed responses over the event queue because I like to have this ordering of the events. Otherwise the DRAIN has to be done in some other way. Also I remember the comment by Bartłomiej Grzesik in [3]:

I remember having a few thoughts on how it could be solved and I
think that removing the need to block those descriptors is the best
approach in my opinion. One would argue that preallocating descriptors
for this purpose or splitting the command queue to input, output and
control might be a viable solution or per stream. However it would
only delay the issue in time or could cause other streams to "starve".

So I'd like to try both approaches at once.

I hope to finish the next draft in 3-4 weeks and publish in January. Hopefully the virtio release will be done and folks will have some time for a review.

Also I've been busy updating the virtio-video driver. First I decided to merge all the existing patches. We have a lot of not yet published changes internally, also you and your team made a lot of changes to the driver. I went through all of these, merged everything carefully, learned about some of the problems, that you encountered. This was a big chunk of work and now it is finished, see [4]. Still working on other driver updates.

The first problem was about making a distinction between guest pages and exported buffers. The final solution in your patches AFAIU is to map VB2_MEMORY_USERPTR to guest pages and VB2_MEMORY_DMABUF to the exported objects because this is the only hint, that we have from REQBUFS. I think this is not good enough, because dmabufs can be of different types: it could be an exported object or, for example, one created with udmabuf. I don't think there is a way to provide this information to the device until the first buffer comes, right? So the spec should be changed in this regard. I think it would be enough to add a requirement, that all the buffers belonging to a queue must be of the same type.

Another problem is setting the DMA masks and other such stuff. Our current approach is to register the parent device, which has all the related fields properly set as it is done in virtio-gpu, see [5].

Overall I'd like to highlight the fact, that you and your team already have a lot of influence on the virtio-video direction. I was very busy with investigations trying to select the right solution and merging the patches, but I think this only makes the whole thing better. :) So your comments are very welcome.

The repo above contains the specification in the README file
(presented in a more informal way than the virtio specification), as
well as a guest V4L2 driver able to pass v4l2-compliance when proxying
the vivid/vicodec virtual devices or an actual UVC camera using the
crosvm V4L2 proxy device [1]. As of now it is basically
feature-complete and offers everything that virtio-video is supposed
to eventually do. I am considering adding support for multiple devices
and requests to allow more complex camera setups.

Since the initial proposal has been rejected I have no intent to push
this forward for merging in the virtio specification, so this will
live out of the official spec. However, it is likely that we will soon
switch to this solution for video support in ChromeOS and possibly
other Google projects with a similar need for a stable virtualization
solution for media devices.

Anyone who is interested in using this for their project, or with
specific requests, is welcome to contact me or open issues on the
Github project.

Cheers,
Alex.

[1] https://chromium-review.googlesource.com/c/crosvm/crosvm/+/506532

Kind regards,
Alexander

[1] https://lore.kernel.org/virtio-dev/a9235fe7-7448-fa9f-ea52-fd27f4845bc4@xxxxxxxxxxxxxxx/ [2] https://lore.kernel.org/virtio-dev/CAPBb6MXw5PebCXYBvXOP_b2j+t-1Y_rSJv7kWkLPsa3X+uM9gA@xxxxxxxxxxxxxx/ [3] https://lore.kernel.org/virtio-dev/CAKxn3ec7nDy9B0nyHUg3GfXt_oktrC91kB3QWXM-sGh8ectE5A@xxxxxxxxxxxxxx/
[4] https://github.com/OpenSynergy/linux/tree/virtio-video
[5] https://github.com/torvalds/linux/commit/b5c9ed70d1a94c59dad7b1ecfc928863c0fe6ac0

--
Alexander Gordeev
Senior Software Engineer

OpenSynergy GmbH
Rotherstr. 20, 10245 Berlin
www.opensynergy.com




[Index of Archives]     [Linux Input]     [Video for Linux]     [Gstreamer Embedded]     [Mplayer Users]     [Linux USB Devel]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [Yosemite Backpacking]

  Powered by Linux