Hi Todor, On 2018-03-15 11:43:31 +0200, Todor Tomov wrote: > Hello, > > I'm trying to understand what is the current state of the multiple virtual > channel support in V4L2 and Media framework. This is the last activity > on this topic which I was able to find. Is anything new happened since > this RFC, is someone working on this or planing to work on? I'm currently working on this but right now I'm focusing on driver dependencies for my use-case, once that is done I will resume to push more for the multiplexed stream support. I randomly push my latest work to git://git.ragnatech.se/linux v4l2/mux But this is a unstable branch and contains some LOCAL patches to test my work on Renesas platforms. But if you want to checkout my current status that's the branch to check. Out of curiosity what board or use-case are you interested in where multiplexed streams would be useful for you? > > Best regards, > Todor Tomov > > On 11.08.2017 12:56, Niklas Söderlund wrote: > > Hi, > > > > This series is a RFC for how I think one could add CSI-2 virtual channel > > support to the V4L2 framework. The problem is that there is no way to in > > the media framework describe and control links between subdevices which > > carry more then one video stream, for example a CSI-2 bus which can have > > 4 virtual channels carrying different video streams. > > > > This series adds a new pad flag which would indicate that a pad carries > > multiplexed streams, adds a new s_stream() operation to the pad > > operations structure which takes a new argument 'stream'. This new > > s_stream() operation then is both pad and stream aware. It also extends > > struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how > > a CSI-2 link multiplexes virtual channels. I also include one > > implementation based on Renesas R-Car which makes use of these patches > > as I think they help with understanding but they have no impact on the > > RFC feature itself. > > > > The idea is that on both sides of the multiplexed media link there are > > one multiplexer subdevice and one demultiplexer subdevice. These two > > subdevices can't do any format conversions, there sole purpose is to > > (de)multiplex the CSI-2 link. If there is hardware which can do both > > CSI-2 multiplexing and format conversions they can be modeled as two > > subdevices from the same device driver and using the still pending > > incremental async mechanism to connect the external pads. The reason > > there is no format conversion is important as the multiplexed pads can't > > have a format in the current V4L2 model, get/set_fmt are not aware of > > streams. > > > > +------------------+ +------------------+ > > +-------+ subdev 1 | | subdev 2 +-------+ > > +--+ Pad 1 | | | | Pad 3 +---+ > > +--+----+ +---------+---+ +---+---------+ +----+--+ > > | | Muxed pad A +------+ Muxed pad B | | > > +--+----+ +---------+---+ +---+---------+ +----+--+ > > +--+ Pad 2 | | | | Pad 4 +---+ > > +-------+ | | +-------+ > > +------------------+ +------------------+ > > > > In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, > > and the video data for both of them travels the link between pad A and > > B. One shortcoming of this RFC is that there currently are no way to > > express to user-space which pad is routed to which stream of the > > multiplexed link. But inside the kernel this is known and format > > validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 > > to Pad 4 by the V4L2 framework. But it would be nice for the user to > > also be able to get this information while setting up the MC graph in > > user-space. > > > > Obviously there are things that are not perfect in this RFC, one is the > > above mentioned lack of user-space visibility of that Pad 1 is in fact > > routed to Pad 3. Others are lack of Documentation/ work and I'm sure > > there are error path shortcuts which are not fully thought out. One big > > question is also if the s_stream() operation added to ops structure > > should be a compliment to the existing ones in video and audio ops or > > aim to replace the one in video ops. I'm also unsure of the CSI2 flag of > > struct v4l2_mbus_frame_desc_entry don't really belong in struct > > v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's > > why this is a RFC... > > > > A big thanks to Laurent and Sakari for being really nice and taking time > > helping me grasp all the possibilities and issues with this problem, all > > cred to them and all blame to me for misunderstanding there guidance :-) > > > > This series based on the latest R-Car CSI-2 and VIN patches which can be > > found at [1], but that is a dependency only for the driver specific > > implementation which acts as an example of implementation. For the V4L2 > > framework patches the media-tree is the base. > > > > 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12 > > > > Niklas Söderlund (20): > > media.h: add MEDIA_PAD_FL_MUXED flag > > v4l2-subdev.h: add pad and stream aware s_stream > > v4l2-subdev.h: add CSI-2 bus description to struct > > v4l2_mbus_frame_desc_entry > > v4l2-core: check that both pads in a link are muxed if one are > > v4l2-core: verify all streams formats on multiplexed links > > rcar-vin: use the pad and stream aware s_stream > > rcar-csi2: declare sink pad as multiplexed > > rcar-csi2: switch to pad and stream aware s_stream > > rcar-csi2: figure out remote pad and stream which are starting > > rcar-csi2: count usage for each source pad > > rcar-csi2: when starting CSI-2 receiver use frame descriptor > > information > > rcar-csi2: only allow formats on source pads > > rcar-csi2: implement get_frame_desc > > adv748x: add module param for virtual channel > > adv748x: declare source pad as multiplexed > > adv748x: add translation from pixelcode to CSI-2 datatype > > adv748x: implement get_frame_desc > > adv748x: switch to pad and stream aware s_stream > > adv748x: only allow formats on sink pads > > arm64: dts: renesas: salvator: use VC1 for CVBS > > > > arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +- > > drivers/media/i2c/adv748x/adv748x-core.c | 10 + > > drivers/media/i2c/adv748x/adv748x-csi2.c | 78 +++++++- > > drivers/media/i2c/adv748x/adv748x.h | 1 + > > drivers/media/platform/rcar-vin/rcar-csi2.c | 239 ++++++++++++++++------- > > drivers/media/platform/rcar-vin/rcar-dma.c | 6 +- > > drivers/media/v4l2-core/v4l2-subdev.c | 65 ++++++ > > include/media/v4l2-subdev.h | 16 ++ > > include/uapi/linux/media.h | 1 + > > 9 files changed, 341 insertions(+), 77 deletions(-) > > > > -- > Best regards, > Todor Tomov -- Regards, Niklas Söderlund
