Patch "media: verisilicon: Fix TRY_FMT on encoder OUTPUT" has been added to the 6.4-stable tree

[Sasha Levin <sashal@xxxxxxxxxx>]

This is a note to let you know that I've just added the patch titled

    media: verisilicon: Fix TRY_FMT on encoder OUTPUT

to the 6.4-stable tree which can be found at:
    http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
     media-verisilicon-fix-try_fmt-on-encoder-output.patch
and it can be found in the queue-6.4 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.



commit 7dbf01d632925e920ed51d57a65bedb2984d3573
Author: Michael Tretter <m.tretter@xxxxxxxxxxxxxx>
Date:   Thu Jul 6 09:15:10 2023 +0200

    media: verisilicon: Fix TRY_FMT on encoder OUTPUT
    
    [ Upstream commit b3b4c9d3cb3bf8725a3ded26f7042b1a37f25333 ]
    
    Commit f100ce3bbd6a ("media: verisilicon: Fix crash when probing
    encoder") removed vpu_fmt from hantro_try_fmt(), since it was
    initialized from vpu_dst_fmt, which may not be initialized, when TRY_FMT
    is called. It was replaced by fmt, which is found using the pixelformat.
    
    For the encoder, this changed the fmt to contain the raw format instead
    of the coded format. The format constraints as of fmt->frmsize are only
    valid for the coded format and are 0 for the raw formats. Therefore, the
    size of a encoder OUTPUT device is constrained to 0 and the
    v4l2-compliance tests for G_FMT, TRY_FMT, and SET_FMT fail.
    
    Bring back vpu_fmt to use the coded format on an encoder OUTPUT device,
    but initialize it using the currently set pixelformat on dst_fmt, which
    is the coded format on an encoder.
    
    Fixes: f100ce3bbd6a ("media: verisilicon: Fix crash when probing encoder")
    Signed-off-by: Michael Tretter <m.tretter@xxxxxxxxxxxxxx>
    Signed-off-by: Hans Verkuil <hverkuil-cisco@xxxxxxxxx>
    Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx>

diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
index 61cfaaf4e927b..e56d58fe28022 100644
--- a/drivers/media/platform/verisilicon/hantro_v4l2.c
+++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
@@ -276,6 +276,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 			  enum v4l2_buf_type type)
 {
 	const struct hantro_fmt *fmt;
+	const struct hantro_fmt *vpu_fmt;
 	bool capture = V4L2_TYPE_IS_CAPTURE(type);
 	bool coded;
 
@@ -295,19 +296,23 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 
 	if (coded) {
 		pix_mp->num_planes = 1;
-	} else if (!ctx->is_encoder) {
+		vpu_fmt = fmt;
+	} else if (ctx->is_encoder) {
+		vpu_fmt = hantro_find_format(ctx, ctx->dst_fmt.pixelformat);
+	} else {
 		/*
 		 * Width/height on the CAPTURE end of a decoder are ignored and
 		 * replaced by the OUTPUT ones.
 		 */
 		pix_mp->width = ctx->src_fmt.width;
 		pix_mp->height = ctx->src_fmt.height;
+		vpu_fmt = fmt;
 	}
 
 	pix_mp->field = V4L2_FIELD_NONE;
 
 	v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
-				       &fmt->frmsize);
+				       &vpu_fmt->frmsize);
 
 	if (!coded) {
 		/* Fill remaining fields */