Re: imx8mp mipi csi camera overlay: Unable to retrieve endpoint for port@1

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Jul 11, 2023 at 3:49 PM Kieran Bingham
<kieran.bingham@xxxxxxxxxxxxxxxx> wrote:
>
> Hi Tim,
>
> Quoting Tim Harvey (2023-07-11 23:16:33)
> > On Mon, Jul 10, 2023 at 6:09 PM Adam Ford <aford173@xxxxxxxxx> wrote:
> > >
> > > On Mon, Jul 10, 2023 at 7:59 PM Tim Harvey <tharvey@xxxxxxxxxxxxx> wrote:
> > > >
> > > > On Fri, Jul 7, 2023 at 6:11 PM Adam Ford <aford173@xxxxxxxxx> wrote:
> > > > >
> > > > > On Fri, Jul 7, 2023 at 7:28 PM Tim Harvey <tharvey@xxxxxxxxxxxxx> wrote:
> > > > > >
> > > > > > Greetings,
> > > > > >
> > > > > > I have an imx219 camera connected to an imx8mp-venice-gw74xx via the
> > > > > > following details:
> > > > > > - camera is a RaspberryPi Camera v2 -
> > > > > > https://datasheets.raspberrypi.com/camera/camera-v2-schematics.pdf
> > > > > > - has its own on-board 24MHz osc so no clock is required from the baseboard
> > > > > > - pin 11 on the camera enables 1.8V and 2.8V LDO which is connected to
> > > > > > IMX8MP GPIO1_IO1 so we use that as a gpio regulator
> > > > > > - MIPI_CSI1 lanes 0 and 1 are used
> > > > > >
> > > > > > I'm using Linux 6.4 with Laurent's pending patch to add the MIPI CSI
> > > > > > DT nodes to imx8mp.dtsi [1] as follows:
> > > > > >
> > > > > > #include <dt-bindings/gpio/gpio.h>
> > > > > > #include "imx8mp-pinfunc.h"
> > > > > >
> > > > > > /dts-v1/;
> > > > > > /plugin/;
> > > > > >
> > > > > > &{/} {
> > > > > >         compatible = "gw,imx8mp-gw74xx", "fsl,imx8mp";
> > > > > >
> > > > > >         reg_cam: regulator-cam {
> > > > > >                 pinctrl-names = "default";
> > > > > >                 pinctrl-0 = <&pinctrl_reg_cam>;
> > > > > >                 compatible = "regulator-fixed";
> > > > > >                 regulator-name = "reg_cam";
> > > > > >                 gpio = <&gpio1 1 GPIO_ACTIVE_HIGH>;
> > > > > >                 enable-active-high;
> > > > > >                 regulator-min-microvolt = <1800000>;
> > > > > >                 regulator-max-microvolt = <1800000>;
> > > > > >         };
> > > > > >
> > > > > >         cam24m: cam24m {
> > > > > >                 compatible = "fixed-clock";
> > > > > >                 #clock-cells = <0>;
> > > > > >                 clock-frequency = <24000000>;
> > > > > >                 clock-output-names = "cam24m";
> > > > > >         };
> > > > > > };
> > > > > >
> > > > > > &i2c3 {
> > > > > >         #address-cells = <1>;
> > > > > >         #size-cells = <0>;
> > > > > >
> > > > > >         imx219: sensor@10 {
> > > > > >                 compatible = "sony,imx219";
> > > > > >                 reg = <0x10>;
> > > > > >                 clocks = <&cam24m>;
> > > > > >                 VDIG-supply = <&reg_cam>;
> > > > > >
> > > > > >                 port {
> > > > > >                         /* MIPI CSI-2 bus endpoint */
> > > > > >                         imx219_to_mipi_csi2: endpoint {
> > > > > >                                 remote-endpoint = <&mipi_csi_0_in>;
> > > > > >                                 clock-lanes = <0>;
> > > > > >                                 data-lanes = <1 2>;
> > > > > >                                 link-frequencies = /bits/ 64 <456000000>;
> > > > > >                         };
> > > > > >                 };
> > > > > >         };
> > > > > > };
> > > > > >
> > > > > > &mipi_csi_0 {
> > > > > >         status = "okay";
> > > > > >
> > > > > >         ports {
> > > > > >                 port@0 {
> > > > > >                         mipi_csi_0_in: endpoint {
> > > > > >                                 remote-endpoint = <&imx219_to_mipi_csi2>;
> > > > > >                                 data-lanes = <1 2>;
> > > > > >                         };
> > > > > >                 };
> > > > > >         };
> > > > > > };
> > > > > >
> > > > > > &iomuxc {
> > > > > >         pinctrl_reg_cam: regcamgrp {
> > > > > >                 fsl,pins = <
> > > > > >                         MX8MP_IOMUXC_GPIO1_IO01__GPIO1_IO01     0x41
> > > > > >                 >;
> > > > > >         };
> > > > > > };
> > > > > >
> > > > > > imx-mipi-csis fails to probe due to a missing port1 endpoint and I'm
> > > > > > not clear what to do with that:
> > > > > > imx-mipi-csis 32e40000.csi: Unable to retrieve endpoint for port@1
> > > > > > imx-mipi-csis: probe of 32e40000.csi failed with error -2
> > > > > >
> > > > > > Any suggestions?
> > > > >
> > > > > I think the port needs to point to an ISI node or a ISP node.
> > > > > Linux-next shows port@1 pointing to  isi_in_0.  In the patch you're
> > > > > referencing, it appears that node@1 is missing.
> > > > >
> > > > > Check out:
> > > > >
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/arch/arm64/boot/dts/freescale/imx8mp.dtsi?h=next-20230707&id=9da15c4c850dd53309e07d5611f33655f8f8c05d
> > > > >
> > > > > See if that helps.
> > > > >
> > > > > adam
> > > > >
> > > >
> > > > Hi Adam,
> > > >
> > > > Thanks for the info. I didn't realize those patches were already in
> > > > linux-next and I missed the fact that the ISI was related to the CSI
> > > > (so I was missing the ISI patch and enabling that driver).
> > > >
> > > > I'm now using 6.5.0-rc1 which has the patches that add CSI and ISI
> > > > nodes to the imx8mp.dtsi with the following dt fragment:
> > > > #include <dt-bindings/gpio/gpio.h>
> > > >
> > > > #include "imx8mp-pinfunc.h"
> > > >
> > > > /dts-v1/;
> > > > /plugin/;
> > > >
> > > > &{/} {
> > > >         compatible = "gw,imx8mp-gw74xx", "fsl,imx8mp";
> > > >
> > > >         reg_cam: regulator-cam {
> > > >                 pinctrl-names = "default";
> > > >                 pinctrl-0 = <&pinctrl_reg_cam>;
> > > >                 compatible = "regulator-fixed";
> > > >                 regulator-name = "reg_cam";
> > > >                 gpio = <&gpio1 4 GPIO_ACTIVE_HIGH>;
> > > >                 enable-active-high;
> > > >                 regulator-min-microvolt = <1800000>;
> > > >                 regulator-max-microvolt = <1800000>;
> > > >         };
> > > >
> > > >         cam24m: cam24m {
> > > >                 compatible = "fixed-clock";
> > > >                 #clock-cells = <0>;
> > > >                 clock-frequency = <24000000>;
> > > >                 clock-output-names = "cam24m";
> > > >         };
> > > > };
> > > >
> > > > &i2c4 {
> > > >         #address-cells = <1>;
> > > >         #size-cells = <0>;
> > > >
> > > >         imx219: sensor@10 {
> > > >                 compatible = "sony,imx219";
> > > >                 reg = <0x10>;
> > > >                 clocks = <&cam24m>;
> > > >                 VDIG-supply = <&reg_cam>;
> > > >
> > > >                 port {
> > > >                         /* MIPI CSI-2 bus endpoint */
> > > >                         imx219_to_mipi_csi2: endpoint {
> > > >                                 remote-endpoint = <&mipi_csi_0_in>;
> > > >                                 clock-lanes = <0>;
> > > >                                 data-lanes = <1 2>;
> > > >                                 link-frequencies = /bits/ 64 <456000000>;
> > > >                         };
> > > >                 };
> > > >         };
> > > > };
> > > >
> > > > &isi_0 {
> > > >         status = "okay";
> > > > };
> > > >
> > > > &mipi_csi_0 {
> > > >         status = "okay";
> > > >
> > > >         ports {
> > > >                 port@0 {
> > > >                         mipi_csi_0_in: endpoint {
> > > >                                 remote-endpoint = <&imx219_to_mipi_csi2>;
> > > >                                 data-lanes = <1 2>;
> > > >                         };
> > > >                 };
> > > >         };
> > > > };
> > > >
> > > > &iomuxc {
> > > >         pinctrl_reg_cam: regcamgrp {
> > > >                 fsl,pins = <
> > > >                         MX8MP_IOMUXC_GPIO1_IO04__GPIO1_IO04     0x41
> > > >                 >;
> > > >         };
> > > > };
> > > >
> > > > Now I'm getting:
> > > > # dmesg | grep imx219
> > > > [    1.764111] imx219 2-0010: supply VANA not found, using dummy regulator
> > > > [    1.770953] imx219 2-0010: supply VDDL not found, using dummy regulator
> > > > [    5.520396] imx219 2-0010: Consider updating driver imx219 to match
> > > > on endpoints
> > > > # dmesg | grep mipi
> > > > [    2.080468] imx-mipi-csis 32e40000.csi: lanes: 2, freq: 500000000
> > > > # cat /sys/bus/media/devices/media*/model
> > > > FSL Capture Media Device
> > > > hantro-vpu
> > > > hantro-vpu
> > > > # cat /sys/class/video4linux/video*/name
> > > > mxc_isi.0.capture
> > > > mxc_isi.1.capture
> > > > mxc_isi.m2m
> > > > nxp,imx8mm-vpu-g1-dec
> > > > nxp,imx8mq-vpu-g2-dec
> > > > # media-ctl /dev/media0 -p
> > > > Media controller API version 6.5.0
> > > >
> > > > Media device information
> > > > ------------------------
> > > > driver          mxc-isi
> > > > model           FSL Capture Media Device
> > > > serial
> > > > bus info        platform:32e00000.isi
> > > > hw revision     0x0
> > > > driver version  6.5.0
> > > >
> > > > Device topology
> > > > - entity 1: crossbar (5 pads, 4 links)
> > > >             type V4L2 subdev subtype Unknown flags 0
> > > >             device node name /dev/v4l-subdev0
> > > >         pad0: Sink
> > > >                 [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb
> > > > xfer:srgb ycbcr:601 quantization:lim-range]
> > > >                 <- "csis-32e40000.csi":1 [ENABLED,IMMUTABLE]
> > > >         pad1: Sink
> > > >                 [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb
> > > > xfer:srgb ycbcr:601 quantization:lim-range]
> > > >         pad2: Sink
> > > >                 <- "mxc_isi.output":0 [ENABLED,IMMUTABLE]
> > > >         pad3: Source
> > > >                 [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb
> > > > xfer:srgb ycbcr:601 quantization:lim-range]
> > > >                 -> "mxc_isi.0":0 [ENABLED,IMMUTABLE]
> > > >         pad4: Source
> > > >                 [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb
> > > > xfer:srgb ycbcr:601 quantization:lim-range]
> > > >                 -> "mxc_isi.1":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 7: mxc_isi.0 (2 pads, 2 links)
> > > >             type V4L2 subdev subtype Unknown flags 0
> > > >             device node name /dev/v4l-subdev1
> > > >         pad0: Sink
> > > >                 [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg
> > > > xfer:srgb ycbcr:601 quantization:full-range
> > > >                  compose.bounds:(0,0)/1920x1080
> > > >                  compose:(0,0)/1920x1080]
> > > >                 <- "crossbar":3 [ENABLED,IMMUTABLE]
> > > >         pad1: Source
> > > >                 [fmt:YUV8_1X24/1920x1080 field:none colorspace:jpeg
> > > > xfer:srgb ycbcr:601 quantization:full-range
> > > >                  crop.bounds:(0,0)/1920x1080
> > > >                  crop:(0,0)/1920x1080]
> > > >                 -> "mxc_isi.0.capture":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 10: mxc_isi.0.capture (1 pad, 1 link)
> > > >              type Node subtype V4L flags 0
> > > >              device node name /dev/video0
> > > >         pad0: Sink
> > > >                 <- "mxc_isi.0":1 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 18: mxc_isi.1 (2 pads, 2 links)
> > > >              type V4L2 subdev subtype Unknown flags 0
> > > >              device node name /dev/v4l-subdev2
> > > >         pad0: Sink
> > > >                 [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg
> > > > xfer:srgb ycbcr:601 quantization:full-range
> > > >                  compose.bounds:(0,0)/1920x1080
> > > >                  compose:(0,0)/1920x1080]
> > > >                 <- "crossbar":4 [ENABLED,IMMUTABLE]
> > > >         pad1: Source
> > > >                 [fmt:YUV8_1X24/1920x1080 field:none colorspace:jpeg
> > > > xfer:srgb ycbcr:601 quantization:full-range
> > > >                  crop.bounds:(0,0)/1920x1080
> > > >                  crop:(0,0)/1920x1080]
> > > >                 -> "mxc_isi.1.capture":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 21: mxc_isi.1.capture (1 pad, 1 link)
> > > >              type Node subtype V4L flags 0
> > > >              device node name /dev/video1
> > > >         pad0: Sink
> > > >                 <- "mxc_isi.1":1 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 29: mxc_isi.output (1 pad, 1 link)
> > > >              type Node subtype V4L flags 0
> > > >         pad0: Source
> > > >                 -> "crossbar":2 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 36: csis-32e40000.csi (2 pads, 2 links)
> > > >              type V4L2 subdev subtype Unknown flags 0
> > > >              device node name /dev/v4l-subdev3
> > > >         pad0: Sink
> > > >                 [fmt:UYVY8_1X16/640x480 field:none
> > > > colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
> > > >                 <- "imx219 3-0010":0 []
> > > >         pad1: Source
> > > >                 [fmt:UYVY8_1X16/640x480 field:none
> > > > colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
> > > >                 -> "crossbar":0 [ENABLED,IMMUTABLE]
> > > >
> > > > - entity 41: imx219 3-0010 (1 pad, 1 link)
> > > >              type V4L2 subdev subtype Sensor flags 0
> > > >              device node name /dev/v4l-subdev4
> > > >         pad0: Source
> > > >                 [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
> > > > xfer:srgb ycbcr:601 quantization:full-range
> > > >                  crop.bounds:(8,8)/3280x2464
> > > >                  crop:(8,8)/3280x2464]
> > > >                 -> "csis-32e40000.csi":0 []
> > > >
> > > > # enable imx219 to csi link
> > > > media-ctl --device /dev/media0 --links "'imx219
> > > > 3-0010':0->'csis-32e40000.csi':0[1]"
> > > >
> > > > # v4l2-ctl --device /dev/video0 --all
> > > > Driver Info:
> > > >         Driver name      : mxc-isi
> > > >         Card type        : mxc-isi-cap
> > > >         Bus info         : platform:32e00000.isi
> > > >         Driver version   : 6.5.0
> > > >         Capabilities     : 0xa4201000
> > > >                 Video Capture Multiplanar
> > > >                 Streaming
> > > >                 Extended Pix Format
> > > >                 Device Capabilities
> > > >         Device Caps      : 0x24201000
> > > >                 Video Capture Multiplanar
> > > >                 Streaming
> > > >                 Extended Pix Format
> > > > Media Driver Info:
> > > >         Driver name      : mxc-isi
> > > >         Model            : FSL Capture Media Device
> > > >         Serial           :
> > > >         Bus info         : platform:32e00000.isi
> > > >         Media version    : 6.5.0
> > > >         Hardware revision: 0x00000000 (0)
> > > >         Driver version   : 6.5.0
> > > > Interface Info:
> > > >         ID               : 0x0300000c
> > > >         Type             : V4L Video
> > > > Entity Info:
> > > >         ID               : 0x0000000a (10)
> > > >         Name             : mxc_isi.0.capture
> > > >         Function         : V4L2 I/O
> > > >         Pad 0x0100000b   : 0: Sink
> > > >           Link 0x0200000e: from remote pad 0x1000009 of entity
> > > > 'mxc_isi.0' (Video Pixel Formatter): Data, Enabled, Immutable
> > > > Priority: 2
> > > > Video input : 0 (mxc_isi.0.capture: ok)
> > > > Format Video Capture Multiplanar:
> > > >         Width/Height      : 640/480
> > > >         Pixel Format      : 'RGGB' (8-bit Bayer RGRG/GBGB)
> > > >         Field             : None
> > > >         Number of planes  : 1
> > > >         Flags             :
> > > >         Colorspace        : sRGB
> > > >         Transfer Function : sRGB
> > > >         YCbCr/HSV Encoding: ITU-R 601
> > > >         Quantization      : Limited Range
> > > >         Plane 0           :
> > > >            Bytes per Line : 640
> > > >            Size Image     : 307200
> > > >
> > > > User Controls
> > > >
> > > >                 horizontal_flip 0x00980914 (bool)   : default=0 value=0
> > > >                   vertical_flip 0x00980915 (bool)   : default=0 value=0
> > > >                 alpha_component 0x00980929 (int)    : min=0 max=255
> > > > step=1 default=0 value=0
> > > >
> > > > # try to capture a frame
> > > > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> > > >                 VIDIOC_STREAMON returned -1 (Broken pipe)
> > > > ^^^ fails... not sure why
> > > >
> > > > # try to use gstreamer
> > > > gst-launch-1.0 v4l2src device=/dev/video0 !
> > > > video/x-bayer,format=rggb,width=640,height=480,framerate=10/1 !
> > > > fakesink
> > > > Setting pipeline to PAUSED ...
> > > > Pipeline is live and does not need PREROLL ...
> > > > Pipeline is PREROLLED ...
> > > > Setting pipeline to PLAYING ...
> > > > New clock: GstSystemClock
> > > > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed
> > > > to allocate required memory.
> > > > Additional debug info:
> > > > ../sys/v4l2/gstv4l2src.c(759): gst_v4l2src_decide_allocation ():
> > > > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > > > Buffer pool activation failed
> > > > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > > > Internal data stream error.
> > > > Execution ended after 0:00:00.005517125
> > > > Additional debug info:
> > > > ../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop ():
> > > > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > > > streaming stopped, reason not-negotiated (-4)
> > > > Setting pipeline to NULL ...
> > > > Freeing pipeline ...
> > > > ^^^ not sure what the memory allocation failure is about. It was
> > > > complaining about cma memory until I added cma=256M to the cmdline
> > > >
> > > > I'm now at the point where I likely need help from the video4linux and
> > > > linux-media (cc'd) gurus.
> > >
> > >
> > > The video format from the camera doesn't match the video format
> > > through the ISI and the crossbar, so you're getting the broken pipe.
> > > My experience with the ISI is that they all need to match since the
> > > ISI isn't an ISP.
> > > Laurent would likely suggest libcamera, but I haven't used it yet.
> > > I'm waiting for the 8MP's ISP driver to get finalized.  However, the
> > > ISI on the Nano is similar to the ISI on the Plus, so if you want to
> > > take a look at the patch I did for the Beacon Nano board, it might
> > > give you some ideas on how to set the video formats.  The patch I
> > > submitted uses a different video format, but I would expect it to work
> > > if you change the video format to the RGGB from the IMX219 camera.
> > >
> > > Hopefully this patch [1] can give you some idea on how to set the
> > > video formats for each of the subdevs.
> > >
> > > adam
> > >
> > > [1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/arm64/boot/dts/freescale/imx8mn-beacon-baseboard.dtsi?h=v6.5-rc1&id=6bc3ea47332648a4211926ba7afc1e1fb935c71a
> > >
> >
> > Hi Adam,
> >
> > Thanks, this helped point me in the right direction and proves the
> > video capture device works with v4l2.
> >
> > For the imx219 which can capture 640x480 raw 8bit bayer:
> > # configure media entities for 8-bit raw bayer 640x480
> > media-ctl -v -V "'imx219 3-0010':0 [fmt:SRGGB8/640x480 field:none]"
> > media-ctl -v -V "'crossbar':0 [fmt:SRGGB8/640x480 field:none]"
> > media-ctl -v -V "'mxc_isi.0':0 [fmt:SRGGB8/640x480 field:none]"
> > # configure for RGGB (8-bit bayer) 640x480
> > v4l2-ctl --device /dev/video0
> > --set-fmt-video=width=640,height=480,pixelformat=RGGB --verbose
> > # capture a frame
> > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=frame.raw
> > --stream-count=1
> > convert -size 640x480 -depth 8 gray:frame.raw frame.png # convert to png
> > # stream to display
> > gst-launch-1.0 v4l2src ! \
> >       video/x-bayer,format=rggb,width=640,height=480,framerate=10/1 ! \
> >       bayer2rgb ! fbdevsink
> >
>
> I would expect if that's all working then libcamera with the ISI
> pipeline handler would be able to handle all of the media-ctl
> configuration for you.
>
> You should be able to use this directly (without needing calls to media-ctl):
>
> gst-launch-1.0 libcamerasrc ! \
>         video/x-bayer,format=rggb,width=640,height=480 ! \
>         bayer2rgb ! fbdevsink
>
> in the same way.
>
> If it does work, I'd be interested to hear that, and if it doesn't -
> then lets fix it!
>
> Note I removed the framerate=10/1 as that's the part I would suspect
> might cause issues in the libcamerasrc. It's likely worth trying both
> with and without it.

Hi Kieran,

I've never looked into libcamera before but I love the idea of not
needing to deal with media-ctl pipelines. I don't see a libcamerasrc
in gstreamer 1.20.3 which I'm using. A quick look at the libcamera
docs [1] seems to indicate building from source isn't necessary when
libcamera is installed but I've installed libcamera-tools/libcamera0
and there is still no libcamerasrc shown in gstreamer?

best regards,

Tim
[1] https://libcamera.org/getting-started.html




[Index of Archives]     [Linux Input]     [Video for Linux]     [Gstreamer Embedded]     [Mplayer Users]     [Linux USB Devel]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [Yosemite Backpacking]

  Powered by Linux