Le mardi 02 mars 2021 à 14:08 -0800, Wren Turkal a écrit : > Okay, so I tried to write up a little bit of an RFC to organize my > thoughts. I would love to get your thoughts: > > RFC: A loopback device for the v4l2 subsystem. > > # What is needed? > > The common denominator for apps that use webcams in linux is the > v4l2 subsystem. The v4l2 subsystem could benefit from a loopback > device that allows one to compose custom video streams and present > them in a normal way from a /dev/video* device. > > # Why is it needed? > > The v4l2 is the framework is the kernel interface used for supporting > video output and video capture. This brings the webcam video to apps > like Cheese or a web browser or OBS. > > However, there is no way to customize the composition of the video > stream before capture on these apps. Of course, apps like Cheese have > filters, but this is not something that is composible with other apps. > This makes it impossible to do something like the following: > > * Composite a video scene with OBS. > * Use that composed video as my webcam for my Zoom call. > > I think this would be cool, and I think that it would allow interesting > things to be done with video. > > # Prior Art > > The v4l2loopback project[1] provides a Linux kernel module to do just > this. This module provides a misc device node used to control the > loopback video devices. In particular, the misc device is used to create > and destroy loopback video devices. It's somewhat similar to the > /dev/loop-control dev node that is used to control block loop devices. > > There is also the vim2m test driver[2] in the kernel that appears to do > some similar things. This module is a module that is used for testing > the v4l2-mem2mem framework. This module has a more modern structure with > a platform device that is used as the context for the other devices. > It also uses the m2m framework that I think could benefit an in-kernel > loopback implementation. It is also important to note that this module > does not generally ship with distribution kernels (like in Fedora), > likely due to the "test" designation. No, since m2m are instance base, you cannot have two process operating on the same node (there is only 1 streaming state). You'd probably want to look at extending the VIMC driver instead. It's a virtual media controller, it's missing few bits to make loopback possible, but would be a good fit. Note that vivid have a loopback capability, but it does not constraint the width/height/format and will produce a test pattern if you request something that differ from the source. > > # Possible paths forward > > ## Integrate v4l2loopback into the kernel > > There is a lot to cleanup in the v4l2loopback code in order to get it > in shape for mainline inclusion. Having said that, this also a somewhat > tested implementation. There are even packages for using this project > from Ubuntu. It is very buggy, it often return a queued buffer back, and in fact, if you try to use the queue too much it fails. Though all this would probably be fine if you port it to VB2. > > ## Start with vim2m > > I think the vim2m.c could provide a really strong base on which to > start a loopback implementation for the kernel. I see two options: > > * Extend vim2m to include more loopback functionality, including > dynamic creation/deletion of video loopback devices. > * Copy the vim2m code to a new module and develop there. > > The extend option is attractive so that a new module mustn't be > developed. > > However, I am not sure that I want to complicate that module with > options that would required for a proper loopback implementation. > > # My thoughts on where to go next > > I think that the vim2m module provides a much nicer starting point for > an implementation of a mainline module. I think the the ideal solution > would start by using the vim2m source as a template and developing a > new module that incorporates the important functionality from > v4l2loopback. > > # Assumptions I have > > I have looked into the code, and the m2m framework appears to provide a > pair of vb2_queues that can be used for capture and output. I am assuming > that the framework would be reasonable to use for a loopback implementation. > > [1] https://github.com/umlaeute/v4l2loopback > [2] > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/media/test-drivers/vim2m.c?h=v5.11.2 > > Thanks, > wt > > On 3/2/21 10:03 AM, Wren Turkal wrote: > > Ezequiel, > > > > Submitting for upstream inclusion is a good idea, and the driver > > is licensed appropriately (gpl v2) for the kernel. I have asked > > the current maintainer for their thoughts. They are supportive > > of the idea, but they are in the middle of implementing an new > > feature that they would like to finish. I will work it up for > > inclusion when that work concludes since it's a pretty major change. > > > > Having said that, does the patch I posted make sense? I think > > that being able to enable the m2m framework without having to > > build in a m2m driver explicitly makes good sense. > > > > Also, thinking about this a little more, it might make more > > sense use use vim2m and v4l2loopback as inspiration and > > make something like loopback that looks more like vim2m. > > > > Things I like about vim2m vs v4l2loopback right now: > > * vim2m uses the m2m framework > > * vim2m is better organized by using a platform device > > to organize the devices that created under the context > > of the platform device. v4l2lb uses a misc device and > > doesn't explicitly place devices under in the context > > of that device. > > * The misc device created by v4l2lb is used to > > dynamically add and remove v4l2lb video devices. It > > It uses new IOCTLs to do that, which I wonder if there > > is a better way. > > * vim2m exposes config via a mechanism that doesn't > > require a new tool. v4l2lb requires a new control > > binary because of the custom ioctls. > > > > So, I have a couple questions: > > * Would it be better to use vim2m as a model to > > implement something like v4l2loopback? > > * If I were to do that should I copy vim2m to a > > new module, or should vim2m be extended. FWIW, > > vim2m appears to have two things against it: > > * vim2m is a test v4l2 driver. It would need > > to be changed to build as a non-test driver. > > This is a pretty easy Kconfig change and maybe > > moving files to a more appropriate place in the > > linux source tree. > > * v4l2loopback does have a number of features that > > vim2m does not have. I will say that I think > > that adding most importantant functionality > > (new formats and dynamic creation of loopback > > video devices in my mind) is probably very > > approachable and may give us a more modern > > linux module code. > > > > Give the above, I am kinda leaning toward extending > > vim2m code. So, do you have an opinion on the following? > > Should it be a new module, or should it literally extend > > and improve vim2m? > > > > wt > > > > > > On 3/1/21 6:11 PM, Ezequiel Garcia wrote: > > > Hi Wren, > > > > > > How about you submit the loopback driver for integration in mainline? > > > > > > We can review the driver, give feedback and help you get it merged. > > > > > > Ezequiel > > > > > > > > > On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <wt@xxxxxxxxxxxxxxxx > > > <mailto:wt@xxxxxxxxxxxxxxxx>> wrote: > > > > > > Okay, so I have found how to enable the option to be manually > > > selected. > > > Would a patch like the following be acceptable? Or is there a better > > > way to do this? > > > > > > diff --git a/drivers/media/v4l2-core/Kconfig > > > b/drivers/media/v4l2-core/Kconfig > > > index bf49f83cb86f..25004137a982 100644 > > > --- a/drivers/media/v4l2-core/Kconfig > > > +++ b/drivers/media/v4l2-core/Kconfig > > > @@ -54,7 +54,7 @@ config V4L2_H264 > > > > > > # Used by drivers that need v4l2-mem2mem.ko > > > config V4L2_MEM2MEM_DEV > > > - tristate > > > + tristate "V4L2 mem2mem framework support" > > > depends on VIDEOBUF2_CORE > > > > > > # Used by LED subsystem flash drivers > > > > > > > > > Thanks, > > > wt > > > > > > On 2/28/21 2:07 PM, Wren Turkal wrote: > > > > Hi there linux-media experts, > > > > > > > > I am working on the out-of-tree v4l2loopback module[1]. My > > > > current goal is to implement the videobuf2 api. However, > > > > it appears that I must use a different vbuf2_queue for the > > > > OUTPUT and CAPTURE types. It appears that the mem2mem > > > > framework has the ability to do just this. I came to this > > > > conclusion while looking at the vim2m test module's code. > > > > While trying to modify the loopback module, I have added > > > > some code to try to call v4l2_m2m_init. You can see my very > > > > WIP attempt at code here[2]. > > > > > > > > I have installed the kernel-devel package on Fedora, which. > > > > Unfortunately, I get he following error messages when trying > > > > to compile: > > > > > > > > <errors> > > > > ERROR: modpost: "v4l2_m2m_release" > > > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > > > ERROR: modpost: "v4l2_m2m_init" > > > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > > > </errors> > > > > > > > > So, I looked into it, and Module.symvers and /proc/kallsyms > > > > don't have those symbols listed, so I think the Fedora kernel > > > > was not compiled with those symbols exported. > > > > > > > > Okay, so next I tried to build a custom kernel that does > > > > export those symbols. I managed to get it done by enabling > > > > the vim2m (VIDEO_VIM2M config option) module. The new > > > > Module.symvers includes the symbols; and the compile succeeds, > > > > which brings me to my final act. > > > > > > > > I need to expose those symbols without enabling the vim2m > > > > module since it is a testing module. I do not see a way > > > > to do that explicitly. The config option in question > > > > appears to be V4L2_MEM2MEM_DEV. And I haven't found a > > > > way to enable it without enabling something that implicitly > > > > selects it (which vim2m does). Do y'all have any suggestions > > > > for how to get these symbols exported properly when vim2m is > > > > not enabled? > > > > > > > > > > > > Thanks, > > > > Wren Turkal (wt) > > > > > > > > [1] https://github.com/umlaeute/v4l2loopback > > > <https://github.com/umlaeute/v4l2loopback> > > > > [2] https://github.com/wt/v4l2loopback/tree/m2m_try > > > <https://github.com/wt/v4l2loopback/tree/m2m_try> > > > > > >
