Hello, I'm trying to understand what is the current state of the multiple virtual channel support in V4L2 and Media framework. This is the last activity on this topic which I was able to find. Is anything new happened since this RFC, is someone working on this or planing to work on? Best regards, Todor Tomov On 11.08.2017 12:56, Niklas Söderlund wrote: > Hi, > > This series is a RFC for how I think one could add CSI-2 virtual channel > support to the V4L2 framework. The problem is that there is no way to in > the media framework describe and control links between subdevices which > carry more then one video stream, for example a CSI-2 bus which can have > 4 virtual channels carrying different video streams. > > This series adds a new pad flag which would indicate that a pad carries > multiplexed streams, adds a new s_stream() operation to the pad > operations structure which takes a new argument 'stream'. This new > s_stream() operation then is both pad and stream aware. It also extends > struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how > a CSI-2 link multiplexes virtual channels. I also include one > implementation based on Renesas R-Car which makes use of these patches > as I think they help with understanding but they have no impact on the > RFC feature itself. > > The idea is that on both sides of the multiplexed media link there are > one multiplexer subdevice and one demultiplexer subdevice. These two > subdevices can't do any format conversions, there sole purpose is to > (de)multiplex the CSI-2 link. If there is hardware which can do both > CSI-2 multiplexing and format conversions they can be modeled as two > subdevices from the same device driver and using the still pending > incremental async mechanism to connect the external pads. The reason > there is no format conversion is important as the multiplexed pads can't > have a format in the current V4L2 model, get/set_fmt are not aware of > streams. > > +------------------+ +------------------+ > +-------+ subdev 1 | | subdev 2 +-------+ > +--+ Pad 1 | | | | Pad 3 +---+ > +--+----+ +---------+---+ +---+---------+ +----+--+ > | | Muxed pad A +------+ Muxed pad B | | > +--+----+ +---------+---+ +---+---------+ +----+--+ > +--+ Pad 2 | | | | Pad 4 +---+ > +-------+ | | +-------+ > +------------------+ +------------------+ > > In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, > and the video data for both of them travels the link between pad A and > B. One shortcoming of this RFC is that there currently are no way to > express to user-space which pad is routed to which stream of the > multiplexed link. But inside the kernel this is known and format > validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 > to Pad 4 by the V4L2 framework. But it would be nice for the user to > also be able to get this information while setting up the MC graph in > user-space. > > Obviously there are things that are not perfect in this RFC, one is the > above mentioned lack of user-space visibility of that Pad 1 is in fact > routed to Pad 3. Others are lack of Documentation/ work and I'm sure > there are error path shortcuts which are not fully thought out. One big > question is also if the s_stream() operation added to ops structure > should be a compliment to the existing ones in video and audio ops or > aim to replace the one in video ops. I'm also unsure of the CSI2 flag of > struct v4l2_mbus_frame_desc_entry don't really belong in struct > v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's > why this is a RFC... > > A big thanks to Laurent and Sakari for being really nice and taking time > helping me grasp all the possibilities and issues with this problem, all > cred to them and all blame to me for misunderstanding there guidance :-) > > This series based on the latest R-Car CSI-2 and VIN patches which can be > found at [1], but that is a dependency only for the driver specific > implementation which acts as an example of implementation. For the V4L2 > framework patches the media-tree is the base. > > 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12 > > Niklas Söderlund (20): > media.h: add MEDIA_PAD_FL_MUXED flag > v4l2-subdev.h: add pad and stream aware s_stream > v4l2-subdev.h: add CSI-2 bus description to struct > v4l2_mbus_frame_desc_entry > v4l2-core: check that both pads in a link are muxed if one are > v4l2-core: verify all streams formats on multiplexed links > rcar-vin: use the pad and stream aware s_stream > rcar-csi2: declare sink pad as multiplexed > rcar-csi2: switch to pad and stream aware s_stream > rcar-csi2: figure out remote pad and stream which are starting > rcar-csi2: count usage for each source pad > rcar-csi2: when starting CSI-2 receiver use frame descriptor > information > rcar-csi2: only allow formats on source pads > rcar-csi2: implement get_frame_desc > adv748x: add module param for virtual channel > adv748x: declare source pad as multiplexed > adv748x: add translation from pixelcode to CSI-2 datatype > adv748x: implement get_frame_desc > adv748x: switch to pad and stream aware s_stream > adv748x: only allow formats on sink pads > arm64: dts: renesas: salvator: use VC1 for CVBS > > arch/arm64/boot/dts/renesas/salvator-common.dtsi | 2 +- > drivers/media/i2c/adv748x/adv748x-core.c | 10 + > drivers/media/i2c/adv748x/adv748x-csi2.c | 78 +++++++- > drivers/media/i2c/adv748x/adv748x.h | 1 + > drivers/media/platform/rcar-vin/rcar-csi2.c | 239 ++++++++++++++++------- > drivers/media/platform/rcar-vin/rcar-dma.c | 6 +- > drivers/media/v4l2-core/v4l2-subdev.c | 65 ++++++ > include/media/v4l2-subdev.h | 16 ++ > include/uapi/linux/media.h | 1 + > 9 files changed, 341 insertions(+), 77 deletions(-) > -- Best regards, Todor Tomov
