Re: [PATCH 00/20] Add multiplexed media pads to support CSI-2 virtual channels

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Niklas,

On 16.03.2018 01:16, Niklas Söderlund wrote:
> Hi Todor,
> 
> On 2018-03-15 11:43:31 +0200, Todor Tomov wrote:
>> Hello,
>>
>> I'm trying to understand what is the current state of the multiple virtual
>> channel support in V4L2 and Media framework. This is the last activity
>> on this topic which I was able to find. Is anything new happened since
>> this RFC, is someone working on this or planing to work on?
> 
> I'm currently working on this but right now I'm focusing on driver 
> dependencies for my use-case, once that is done I will resume to push 
> more for the multiplexed stream support. I randomly push my latest work 
> to
> 
> git://git.ragnatech.se/linux v4l2/mux
> 
> But this is a unstable branch and contains some LOCAL patches to test my 
> work on Renesas platforms. But if you want to checkout my current status 
> that's the branch to check.
> 
> Out of curiosity what board or use-case are you interested in where 
> multiplexed streams would be useful for you?

Good to hear that you are still working on this :)
I'm working on QComm SoCs and I may need multiple virtual channel
support however the usecases are not clear yet. I'll be keeping an
eye on any activity from you then ;) or when I have more details
about my usecases I'll try to evaluate how they fit with this RFC.

Thank you.
Best regards,
Todor Tomov

> 
>>
>> Best regards,
>> Todor Tomov
>>
>> On 11.08.2017 12:56, Niklas Söderlund wrote:
>>> Hi,
>>>
>>> This series is a RFC for how I think one could add CSI-2 virtual channel 
>>> support to the V4L2 framework. The problem is that there is no way to in 
>>> the media framework describe and control links between subdevices which 
>>> carry more then one video stream, for example a CSI-2 bus which can have 
>>> 4 virtual channels carrying different video streams.
>>>
>>> This series adds a new pad flag which would indicate that a pad carries 
>>> multiplexed streams, adds a new s_stream() operation to the pad 
>>> operations structure which takes a new argument 'stream'. This new 
>>> s_stream() operation then is both pad and stream aware. It also extends 
>>> struct v4l2_mbus_frame_desc_entry with a new sub-struct to describe how 
>>> a CSI-2 link multiplexes virtual channels. I also include one 
>>> implementation based on Renesas R-Car which makes use of these patches 
>>> as I think they help with understanding but they have no impact on the 
>>> RFC feature itself.
>>>
>>> The idea is that on both sides of the multiplexed media link there are 
>>> one multiplexer subdevice and one demultiplexer subdevice. These two 
>>> subdevices can't do any format conversions, there sole purpose is to 
>>> (de)multiplex the CSI-2 link. If there is hardware which can do both 
>>> CSI-2 multiplexing and format conversions they can be modeled as two 
>>> subdevices from the same device driver and using the still pending 
>>> incremental async mechanism to connect the external pads. The reason 
>>> there is no format conversion is important as the multiplexed pads can't 
>>> have a format in the current V4L2 model, get/set_fmt are not aware of 
>>> streams.
>>>
>>>         +------------------+              +------------------+
>>>      +-------+  subdev 1   |              |  subdev 2   +-------+
>>>   +--+ Pad 1 |             |              |             | Pad 3 +---+
>>>      +--+----+   +---------+---+      +---+---------+   +----+--+
>>>         |        | Muxed pad A +------+ Muxed pad B |        |
>>>      +--+----+   +---------+---+      +---+---------+   +----+--+
>>>   +--+ Pad 2 |             |              |             | Pad 4 +---+
>>>      +-------+             |              |             +-------+
>>>         +------------------+              +------------------+
>>>
>>> In the simple example above Pad 1 is routed to Pad 3 and Pad 2 to Pad 4, 
>>> and the video data for both of them travels the link between pad A and 
>>> B. One shortcoming of this RFC is that there currently are no way to 
>>> express to user-space which pad is routed to which stream of the 
>>> multiplexed link. But inside the kernel this is known and format 
>>> validation is done by comparing the format of Pad 1 to Pad 3 and Pad 2 
>>> to Pad 4 by the V4L2 framework. But it would be nice for the user to 
>>> also be able to get this information while setting up the MC graph in 
>>> user-space.
>>>
>>> Obviously there are things that are not perfect in this RFC, one is the 
>>> above mentioned lack of user-space visibility of that Pad 1 is in fact 
>>> routed to Pad 3. Others are lack of Documentation/ work and I'm sure 
>>> there are error path shortcuts which are not fully thought out. One big 
>>> question is also if the s_stream() operation added to ops structure 
>>> should be a compliment to the existing ones in video and audio ops or 
>>> aim to replace the one in video ops. I'm also unsure of the CSI2 flag of 
>>> struct v4l2_mbus_frame_desc_entry don't really belong in struct 
>>> v4l2_mbus_frame_desc. And I'm sure there are lots of other stuff that's 
>>> why this is a RFC...
>>>
>>> A big thanks to Laurent and Sakari for being really nice and taking time 
>>> helping me grasp all the possibilities and issues with this problem, all 
>>> cred to them and all blame to me for misunderstanding there guidance :-)
>>>
>>> This series based on the latest R-Car CSI-2 and VIN patches which can be 
>>> found at [1], but that is a dependency only for the driver specific
>>> implementation which acts as an example of implementation. For the V4L2 
>>> framework patches the media-tree is the base.
>>>
>>> 1. https://git.ragnatech.se/linux#rcar-vin-elinux-v12
>>>
>>> Niklas Söderlund (20):
>>>   media.h: add MEDIA_PAD_FL_MUXED flag
>>>   v4l2-subdev.h: add pad and stream aware s_stream
>>>   v4l2-subdev.h: add CSI-2 bus description to struct
>>>     v4l2_mbus_frame_desc_entry
>>>   v4l2-core: check that both pads in a link are muxed if one are
>>>   v4l2-core: verify all streams formats on multiplexed links
>>>   rcar-vin: use the pad and stream aware s_stream
>>>   rcar-csi2: declare sink pad as multiplexed
>>>   rcar-csi2: switch to pad and stream aware s_stream
>>>   rcar-csi2: figure out remote pad and stream which are starting
>>>   rcar-csi2: count usage for each source pad
>>>   rcar-csi2: when starting CSI-2 receiver use frame descriptor
>>>     information
>>>   rcar-csi2: only allow formats on source pads
>>>   rcar-csi2: implement get_frame_desc
>>>   adv748x: add module param for virtual channel
>>>   adv748x: declare source pad as multiplexed
>>>   adv748x: add translation from pixelcode to CSI-2 datatype
>>>   adv748x: implement get_frame_desc
>>>   adv748x: switch to pad and stream aware s_stream
>>>   adv748x: only allow formats on sink pads
>>>   arm64: dts: renesas: salvator: use VC1 for CVBS
>>>
>>>  arch/arm64/boot/dts/renesas/salvator-common.dtsi |   2 +-
>>>  drivers/media/i2c/adv748x/adv748x-core.c         |  10 +
>>>  drivers/media/i2c/adv748x/adv748x-csi2.c         |  78 +++++++-
>>>  drivers/media/i2c/adv748x/adv748x.h              |   1 +
>>>  drivers/media/platform/rcar-vin/rcar-csi2.c      | 239 ++++++++++++++++-------
>>>  drivers/media/platform/rcar-vin/rcar-dma.c       |   6 +-
>>>  drivers/media/v4l2-core/v4l2-subdev.c            |  65 ++++++
>>>  include/media/v4l2-subdev.h                      |  16 ++
>>>  include/uapi/linux/media.h                       |   1 +
>>>  9 files changed, 341 insertions(+), 77 deletions(-)
>>>
>>
>> -- 
>> Best regards,
>> Todor Tomov
> 

-- 
Best regards,
Todor Tomov



[Index of Archives]     [Linux Input]     [Video for Linux]     [Gstreamer Embedded]     [Mplayer Users]     [Linux USB Devel]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [Yosemite Backpacking]
  Powered by Linux