Hi Sakari and Laurent,
On 16.04.2024 22:32, Sakari Ailus wrote:
Document internal sink pads, pads that have both SINK and INTERNAL flags
set. Use the IMX219 camera sensor as an example.
Signed-off-by: Sakari Ailus <sakari.ailus@xxxxxxxxxxxxxxx>
Reviewed-by Julien Massot <julien.massot@xxxxxxxxxxxxx>
---
.../userspace-api/media/v4l/dev-subdev.rst | 145 ++++++++++++++++++
1 file changed, 145 insertions(+)
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index b76e02e54512..d30dcb9e2537 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the
sub-device and a (pad, stream) pair. For sub-devices that do not support
multiplexed streams the 'stream' field is always 0.
+.. _v4l2-subdev-internal-source-pads:
+
+Internal sink pads and routing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Cases where a single sub-device source pad is traversed by multiple streams, one
+or more of which originate from within the sub-device itself, are special as
+there is no external sink pad for such routes. In those cases, the sources of
+the internally generated streams are represented by internal sink pads, which
+are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`
+pad flag set.
+
+Internal pads have all the properties of an external pad, including formats and
+selections. The format in this case is the source format of the stream. An
+internal pad always has a single stream only (0).
For me, it is not clear what is allowed/expected regarding the formats
on the internal sink pads.
What "The format in this case is the source format of the stream" means?
I suppose the format on the internal pads will never be set by the user,
but be static or propagated?
A glimpse into the future shows that libcamera makes some assumptions in
src/libcamera/sensor/camera_sensor_raw.cpp:
https://lists.libcamera.org/pipermail/libcamera-devel/2024-March/041006.html ;
/*
* Get the native sensor CFA pattern. It is simpler to retrieve it from
* the internal image sink pad as it is guaranteed to expose a single
* format, and is not affected by flips.
*/
I think it would be good to have a more clear documentation in the
kernel, for both the driver developers and user applications to know
what behavior to expect. If a "native" format is always expected on the
internal sink pads, let it be documented, or if the internal sink pads
format should be propagated from the source pad format (presumably set
by the user, and contrary to the general recommendation to propagate the
format from sink to source).
Also, if there is a guarantee to have a single format exposed, as stated
in libcamera, it would be good to have that also documented in the kernel.
I assume some things will change in libcamera after the common raw
sensor model addition in the kernel, I don't fully envision how.
I am working on adding internal pads and streams for os08a20 hdr sensor,
and also looked at the changes made to imx219, to give some context to
my questions.
Thanks,
Mirela
+
+Routes from an internal sink pad to an external source pad are typically not
+modifiable but they can be activated and deactivated using the
+:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
+on driver capabilities.
+
Interaction between routes, streams, formats and selections
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps:
the configurations along the stream towards the receiver, using
:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
stream endpoint in each sub-device.
+
+Internal pads setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- An IMX219 camera sensor source sub-device, with one source pad (0), one
+ internal sink pad (1) as the source of the image data and an internal sink
+ pad (2) as the source of the embedded data. There are two routes, one from the
+ internal sink pad 1 to the source 0 (image data) and another from the internal
+ sink pad 2 to the source pad 0 (embedded data). Both streams are always
+ active, i.e. there is no need to separately enable the embedded data
+ stream. The sensor uses the CSI-2 interface.
+
+- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0),
+ connected to the sensor, and two source pads (pads 1 and 2), to the DMA
+ engine. The receiver demultiplexes the incoming streams to the source pads.
+
+- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a
+ single source pad of the receiver.
+
+The sensor and the receiver are modelled as V4L2 sub-devices, exposed to
+userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as
+V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: enable the links from the sensor to the
+ receiver and from the receiver to the DMA engines. This step does not differ
+ from normal non-multiplexed media controller setup.
+
+2) Configure routing
+
+.. flat-table:: Camera sensor. There are no configurable routes.
+ :header-rows: 1
+
+ * - Sink Pad/Stream
+ - Source Pad/Stream
+ - Routing Flags
+ - Comments
+ * - 1/0
+ - 0/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Pixel data stream from the sink pad
+ * - 2/0
+ - 0/1
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Metadata stream from the internal sink pad
+
+.. flat-table:: Receiver routing table. Typically both routes need to be
+ explicitly set.
+ :header-rows: 1
+
+ * - Sink Pad/Stream
+ - Source Pad/Stream
+ - Routing Flags
+ - Comments
+ * - 0/0
+ - 1/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Pixel data stream from camera sensor
+ * - 0/1
+ - 2/0
+ - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+ - Metadata stream from camera sensor
+
+The options available in sensor's routing configuration are dictated by hardware
+capabilities: typically camera sensors always produce an image data stream while
+it may be possible to enable and disable the embedded data stream.
+
+3) Configure formats and selections
+
+ This example assumes that the formats are propagated from sink pad to the
+ source pad as-is. The tables contain fields of both struct v4l2_subdev_format
+ and struct v4l2_mbus_framefmt.
+
+.. flat-table:: Formats set on the sub-devices. Bold values are set, others are
+ static or propagated. The order is aligned with configured
+ routes.
+ :header-rows: 1
+ :fill-cells:
+
+ * - Sub-device
+ - Pad/Stream
+ - Width
+ - Height
+ - Code
+ * - :rspan:`3` IMX219
+ - 1/0
+ - 3296
+ - 2480
+ - MEDIA_BUS_FMT_SRGGB10
+ * - 0/0
+ - **3296**
+ - **2480**
+ - **MEDIA_BUS_FMT_SRGGB10**
+ * - 2/0
+ - 3296
+ - 2
+ - MEDIA_BUS_FMT_IMX219_EMBEDDED
+ * - 1/1
+ - 3296
+ - 2
+ - MEDIA_BUS_FMT_META_10
+ * - :rspan:`3` CSI-2 receiver
+ - 0/0
+ - **3296**
+ - **2480**
+ - **MEDIA_BUS_FMT_SRGGB10**
+ * - 1/0
+ - 3296
+ - 2480
+ - MEDIA_BUS_FMT_SRGGB10
+ * - 0/1
+ - **3296**
+ - **2**
+ - **MEDIA_BUS_FMT_META_10**
+ * - 2/0
+ - 3296
+ - 2
+ - MEDIA_BUS_FMT_META_10
+
+The embedded data format does not need to be configured as the format is
+dictated by the pixel data format in this case.
