Hello Alexandre,
On 08.12.23 07:46, Alexandre Courbot wrote:
Hello everyone,
This is just a heads-up to announce the initial publication of virtio-media:
https://github.com/Gnurou/virtio-media
The name has changed slightly, but this project is essentially the
refinement and continuation of my virtio-v4l2 proposal. The reason for
moving forward with this is, to be candid, the unclear direction and
lack of progress regarding the future of virtio-video, and the
non-consensual way in which its development has been, let's say
"redirected".
Great! I like the new name. The old one was confusing IMO. I'm going to
have a closer look later.
I'd be happy to clarify the current progress and the future direction of
the virtio-video. I'd like to make these major changes in the next spec
draft:
1. Represent the capabilities and parameters with TLVs. Basically I'd
like to take the data format from VIDIOC_G_EXT_CTRLS/VIDIOC_S_EXT_CTRLS,
flatten it by replacing the pointers with the actual data (this would be
then in fact a sequence of TLVs). Then use this data format when getting
capabilities (all of the VIDIOC_ENUM* and VIDIOC_QUERY* in a single
call), getting and setting parameters (parameters + controls in V4L2
terms). Here I'd like to reuse the control ids and value definitions as
much as possible. Maybe this data presentation could be used in V4L2
later. I think it would be digestible, as it is an extension of the
extended controls. Indeed it was not clear, I even thought device trees
would be a great solution at some point (see [1] for the rationale), but
several people convinced me, that's not going to be merged in the kernel.
BTW this activity was influenced by your comment in [2]:
Beware of not repeating the same mistake as v5 (all controls under one
big structure).
2. Make the input and output resource queues explicit by turning them
into virtio queues. It became clear, that after making the commands non
blocking, some of the queue control moves to these internal resource
queues. Then I thought: why make another queue implementation, when we
already have one? So I think it makes sense to turn them into virtio
queues. I still would like to keep the delayed responses over the event
queue because I like to have this ordering of the events. Otherwise the
DRAIN has to be done in some other way. Also I remember the comment by
Bartłomiej Grzesik in [3]:
I remember having a few thoughts on how it could be solved and I
think that removing the need to block those descriptors is the best
approach in my opinion. One would argue that preallocating descriptors
for this purpose or splitting the command queue to input, output and
control might be a viable solution or per stream. However it would
only delay the issue in time or could cause other streams to "starve".
So I'd like to try both approaches at once.
I hope to finish the next draft in 3-4 weeks and publish in January.
Hopefully the virtio release will be done and folks will have some time
for a review.
Also I've been busy updating the virtio-video driver. First I decided to
merge all the existing patches. We have a lot of not yet published
changes internally, also you and your team made a lot of changes to the
driver. I went through all of these, merged everything carefully,
learned about some of the problems, that you encountered. This was a big
chunk of work and now it is finished, see [4]. Still working on other
driver updates.
The first problem was about making a distinction between guest pages and
exported buffers. The final solution in your patches AFAIU is to map
VB2_MEMORY_USERPTR to guest pages and VB2_MEMORY_DMABUF to the exported
objects because this is the only hint, that we have from REQBUFS. I
think this is not good enough, because dmabufs can be of different
types: it could be an exported object or, for example, one created with
udmabuf. I don't think there is a way to provide this information to the
device until the first buffer comes, right? So the spec should be
changed in this regard. I think it would be enough to add a requirement,
that all the buffers belonging to a queue must be of the same type.
Another problem is setting the DMA masks and other such stuff. Our
current approach is to register the parent device, which has all the
related fields properly set as it is done in virtio-gpu, see [5].
Overall I'd like to highlight the fact, that you and your team already
have a lot of influence on the virtio-video direction. I was very busy
with investigations trying to select the right solution and merging the
patches, but I think this only makes the whole thing better. :) So your
comments are very welcome.
The repo above contains the specification in the README file
(presented in a more informal way than the virtio specification), as
well as a guest V4L2 driver able to pass v4l2-compliance when proxying
the vivid/vicodec virtual devices or an actual UVC camera using the
crosvm V4L2 proxy device [1]. As of now it is basically
feature-complete and offers everything that virtio-video is supposed
to eventually do. I am considering adding support for multiple devices
and requests to allow more complex camera setups.
Since the initial proposal has been rejected I have no intent to push
this forward for merging in the virtio specification, so this will
live out of the official spec. However, it is likely that we will soon
switch to this solution for video support in ChromeOS and possibly
other Google projects with a similar need for a stable virtualization
solution for media devices.
Anyone who is interested in using this for their project, or with
specific requests, is welcome to contact me or open issues on the
Github project.
Cheers,
Alex.
[1] https://chromium-review.googlesource.com/c/crosvm/crosvm/+/506532
Kind regards,
Alexander
[1]
https://lore.kernel.org/virtio-dev/a9235fe7-7448-fa9f-ea52-fd27f4845bc4@xxxxxxxxxxxxxxx/
[2]
https://lore.kernel.org/virtio-dev/CAPBb6MXw5PebCXYBvXOP_b2j+t-1Y_rSJv7kWkLPsa3X+uM9gA@xxxxxxxxxxxxxx/
[3]
https://lore.kernel.org/virtio-dev/CAKxn3ec7nDy9B0nyHUg3GfXt_oktrC91kB3QWXM-sGh8ectE5A@xxxxxxxxxxxxxx/
[4] https://github.com/OpenSynergy/linux/tree/virtio-video
[5]
https://github.com/torvalds/linux/commit/b5c9ed70d1a94c59dad7b1ecfc928863c0fe6ac0
--
Alexander Gordeev
Senior Software Engineer
OpenSynergy GmbH
Rotherstr. 20, 10245 Berlin
www.opensynergy.com
