On Sat, Feb 2, 2019 at 11:10 AM Steve Longerbeam <slongerbeam@xxxxxxxxx> wrote:
>
>
<snip>
>
> >
> > But when going through the IC we again run into the issue where the
> > output of the IC isn't a suitable colorspace:
> > # 720p@60Hz YUV BT656
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0' # fmt:UYVY8_2X8/1280x720
> > field:none colorspace:srgb
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_2X8/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_2X8/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> > # downscale because the IC can't do >1024
> > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1' #
> > [fmt:AYUV8_1X32/640x360@1/25 field:none colorspace:srgb xfer:srgb
> > ycbcr:601 quantization:lim-range]
> > # ^^^ note we are itu601 again b/c that's the only format the IC can ouput
> > # stream JPEG/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video0 ! \
> > video/x-raw,format=UYVY ! \
> > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> > # ^^^ works... SW JPEG can handle itu601
>
> Ok.
>
> > # stream upscale via mem2mem then JPEG/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video4 ! \
> > v4l2video8convert ! video/x-raw,width=1280,height=720 ! \
> > jpegenc ! rtpjpegpay ! udpsink host=172.24.20.19 port=5000
> > ERROR: from element
> > /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0: Device
> > '/dev/video8' does not support 2:4:7:1 colorimetry
> > # ^^^ fails because mem2mem doesn't support itu601
> > # stream H264/RTP/UDP
> > gst-launch-1.0 v4l2src device=/dev/video4 ! \
> > v4l2h264enc output-io-mode=dmabuf-import ! \
> > rtph264pay ! udpsink host=172.24.20.19 port=5001
> > ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
> > Device '/dev/video9' does not support 2:4:7:1 colorimetry
> > # ^^^ coda has same issue... can't del with itu601
>
> Well, just to see things working, try hacking
> imx_media_fill_default_mbus_fields() to set Rec. 709 encoding:
>
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -571,7 +571,7 @@ void imx_media_fill_default_mbus_fields(struct
> v4l2_mbus_framefmt *tryfmt,
> tryfmt->quantization = is_rgb ?
> V4L2_QUANTIZATION_FULL_RANGE :
> V4L2_QUANTIZATION_LIM_RANGE;
> - tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_601;
> + tryfmt->ycbcr_enc = V4L2_YCBCR_ENC_709;
> }
> }
> EXPORT_SYMBOL_GPL(imx_media_fill_default_mbus_fields);
>
>
> But of course that's not technically correct because the encoding in
> ipu-ic.c is BT.601.
>
> The *real* way to fix this would be to allow programmable encodings in
> ipu-ic.c. But unfortunately the encodings are hardcoded (grep for
> ic_csc_rgb2ycbcr in ipu-ic.c).
>
Ok, I saw that you went ahead and worked on this (thanks!) and that
you have a bt.709.v2 branch... is that one ready for testing?
> The other option would be to change ic_csc_rgb2ycbcr to use the Rec. 709
> coefficients, and then the above patch is no longer a hack. The inverse
> encoding (ic_csc_ycbcr2rgb) would also need to be changed.
>
>
> >
> > Am I perhaps missing a capsfilter to get the mem2mem driver to convert
> > the colorspace properly? If so, they the mem2mem driver could be used
> > to correct the colorspace to get IC output to coda working.
>
> Well, first I don't think the mem2mem driver is using the correct
> encoding. The mem2mem driver is making use of the IC encoding so it
> should be reporting and accepting only BT.601.
>
> Philipp ^^
>
>
> >
> > Also can we connect the mem2mem driver to the unused VDIC input in the
> > media controller so that we can use the VDIC to de-interlace content
> > captured from non IMX sources (ie PCI or USB capture devices)?
>
> Exactly! That's something I have been working on. But it's difficult to
> connect mem2mem to the unused VDIC IDMAC input pad because as of now the
> v4l2 mem2mem internal API's do not allow connecting to *existing*
> processing entities, and there are also issues with how sub-devices are
> to deal with mem2mem contexts.
>
> I do have a WIP branch that creates a video output device that connects
> to the VDIC IDMAC input pad, which doesn't have the above issues. The
> only drawback with that is how gstreamer can make use of such an output
> device.
>
ok, keep me posted. Is it the output-vdic or mem2mem.v4-mc branch?
I also noticed you have a add-fim-to-prpencvf branch... are you
working on adding FIM to ipu_ic_prp/enc still? That would be nice to
have to deal with sync loss/regain in analog decoders going through
VDIC de-interlacing.
>
<snip>
> >>
> > This one (480i60Hz YUV via BT656 sensor->mux->csi->ic_prp->ic_prpenc)
> > still baffles me a bit but I've also found that any bt656 capture that
> > isn't specifically 720x480 (NTSC) or 720x576 (PAL) fails because of
> > the resolution checks in ipu_csi_init_interface() resulting in
> > 'Unsupported interlaced video mode'. I'm not sure if
> > ipu_csi_set_bt_interlaced_codes() can be modified to support other
> > resolutions?
>
> Well, Bt.656 only defines standard definition NTSC and PAL.
>
That is true. Do you know of any other sensors that use higher
resolutions with BT656 SAV/EAV encoding? The BT656 mode does work well
for the progressive modes up to 1080p30 (1080p60 exceeds the IMX6
pixel clock and can't be used).
I could dig into the bt656 spec to try and understand the various
codes that get stuffed into the IPUx_CSI0_CCIR_CODE_1/2/3 registers I
suppose but I'm still not sure I want to push up a device-tree config
that describes the tda1997x to CSI connection as 8bit BT656 as I would
prefer to describe it as 16bit YUV instead (as I may be close to
getting that format working well if we can get the IC able to output
rec709).
<snip>
> >
> > # imx6q-gw54xx tda19971 720p 16bit YUV IPU1_CSI0
> > MODE1:sensor->mux->csi->ic_prp->ic_prpenc
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> > [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
> > # get framerate
> > v4l2-ctl --device /dev/v4l-subdev15 --get-dv-timings
> > DV timings:
> > Active width: 1280
> > Active height: 720
> > Total width: 1650
> > Total height: 750
> > Frame format: progressive
> > Polarities: +vsync +hsync
> > Pixelclock: 74250000 Hz (60.00 frames per second)
> > Horizontal frontporch: 110
> > Horizontal sync: 40
> > Horizontal backporch: 220
> > Vertical frontporch: 5
> > Vertical sync: 5
> > Vertical backporch: 20
> > Standards: CTA-861
> > CTA-861 VIC: 0
> > Flags: framerate can be reduced by 1/1.001, CE-video, has CTA-861 VIC
> >
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
> > media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
> > media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
> > media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
> > media-ctl --get-v4l2 '"ipu1_ic_prpenc":1'
> > # capture device
> > media-ctl -e 'ipu1_ic_prpenc capture'
> > /dev/video0
> > v4l2-ctl --device /dev/video0 --get-fmt-video
> > Format Video Capture:
> > Width/Height : 640/360
> > Pixel Format : 'UYVY' (UYVY 4:2:2)
> > Field : None
> > Bytes per Line : 1280
> > Size Image : 460800
> > Colorspace : Rec. 709
> > Transfer Function : Rec. 709
> > YCbCr/HSV Encoding: ITU-R 601
> > Quantization : Limited Range
> > Flags :
> >
> > # capture 1 frame
> > v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
> > [ 125.966980] ipu1_ic_prpenc: pipeline start failed with -32
> >
> > Do you know what the failure is here?
>
> You are /2 downscaling in the CSI, but did not set the compose window at
> the input pad, e.g:
>
> media-ctl --set-v4l2 '"ipu1_csi0":0[compose:(0,0)/640x360]' # 1/2 scale
>
I don't think this is the issue. Note that I 'was' able to get the
same pipeline with the div-by-2 downscale without compose working with
720p bt656 YUV.
Here is the 720p 16bit YUV bus again with the compose:
# 720p60Hz YUV 16bit YUV bus
# imx6q-gw54xx tda19971 2-0048 IPU1_CSI0
MODE1:sensor->mux->csi->ic_prp->ic_prpenc
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
BT timings set
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0'
[fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':0 [compose:(0,0)/640x360]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x360 ]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x360 ]"
media-ctl --get-v4l2 "'ipu1_csi0':0"
[fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range
crop.bounds:(0,0)/1280x720
crop:(0,0)/1280x720
compose.bounds:(0,0)/1280x720
compose:(0,0)/640x360]
media-ctl --get-v4l2 "'ipu1_csi0':1"
[fmt:UYVY8_1X16/640x360@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:601 quantization:lim-range]
media-ctl --get-v4l2 "'ipu1_ic_prp':1"
[fmt:AYUV8_1X32/640x480@1/30 field:none
colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
media-ctl --get-v4l2 "'ipu1_ic_prpenc':1"
[fmt:AYUV8_1X32/640x360@1001/30000 field:none
colorspace:smpte170m xfer:709 ycbcr:601 quantization:lim-range]
media-ctl -e 'ipu1_ic_prpenc capture'
/dev/video0
v4l2-ctl --device /dev/video0 --get-fmt-video
Format Video Capture:
Width/Height : 640/360
Pixel Format : 'UYVY' (UYVY 4:2:2)
Field : None
Bytes per Line : 1280
Size Image : 460800
Colorspace : SMPTE 170M
Transfer Function : Rec. 709
YCbCr/HSV Encoding: ITU-R 601
Quantization : Limited Range
Flags :
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=x.raw --stream-count=1
[ 1279.017148] ipu1_ic_prpenc: pipeline start failed with -32
>
>
> >
> > And can you explain why I can't colorspace convert the following CSI
> > capture case?:
> >
> > # imx6q-gw54xx tda19971 720p60 16-bit YUV sensor->mux->csi
> > # set sensor output pad to sensor source format
> > v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
> > # sensor format
> > media-ctl --get-v4l2 '"tda19971 2-0048":0'
> > [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
> > # reset all links
> > media-ctl --reset
> > # setup links
> > media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
> > media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
> > media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
> > # configure pads
> > media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
> > media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> > [fmt:UYVY8_1X16/1280x720@1/30 field:none
> > colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
> > ^^^ still itu601
> >
> > I have found in my testing when dealing with a BT656 RGB colorspace
> > input which I need to convert ot rec709 I have to set the 'input' pad
> > of the csi which doesn't make sense and if I do this with a 16-bit RGB
> > colorspace (above) it jacks up the fmt:
> > media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
> > media-ctl --get-v4l2 '"ipu1_csi0":2'
> > [fmt:UYVY8_2X8/1280x720@1/30 field:none
> > colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
> > ^^^ changed fmt because AYUV32 at CSI input is invalid so it defaults
> > fmt to YUVY8_2X8
> >
> > So there is something I'm still doing wrong to setup CSC.
>
> Sorry I guess I don't understand your question here. Correct the CSI
> does not accept AYUV32 at its input pad.
>
What I was asking here is what is the correct way to configure for
ITU601 to REC709 CSC at the CSI? I would expect to set the REC709
format on the output pad of the CSI but it seems to require me to set
it on the input pad.
Consider:
# imx6q-gw54xx tda19971 720p60 16-bit RGB sensor->mux->csi
# set sensor output pad to sensor source format
v4l2-ctl -d /dev/v4l-subdev15 --set-dv-bt-timings=query
# sensor format
media-ctl --get-v4l2 '"tda19971 2-0048":0'
[fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
# reset all links
media-ctl --reset
# setup links
media-ctl -l "'tda19971 2-0048':0 -> 'ipu1_csi0_mux':1[1]"
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]"
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
# configure pads
media-ctl -V "'tda19971 2-0048':0 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY8_1X16/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
[fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
^^^ ITU601, but I want REC709
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
media-ctl --get-v4l2 '"ipu1_csi0":2'
[fmt:UYVY8_1X16/1280x720@1/30 field:none
colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
^^^ No invalid argument but its still ITU601 at CSI output... but what
I've found is if I set it at the CSI input the ycbcr changes but the
fmt gets jacked up (as expected)
media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/1280x720 colorspace:rec709 ycbcr:709]"
[fmt:UYVY8_2X8/1280x720@1/30 field:none
colorspace:rec709 xfer:709 ycbcr:709 quantization:lim-range]
^^^ note we are now rec709 but fmt is UYVY8_2X8 because the CSI
doesn't accept UYVY8_1X16 so it deafults to UYVY8_2X8. However, again
I expect to have set the CSC on the output pad. If I do this on a
BT656 sensor bus it works because UYVY8_2X8 is valid.
Tim
