On 7/14/22 12:18, nanxin.qin@xxxxxxxxxxx wrote:
> Hi, Hans
> thanks for your reply. Currently, our source code has not been submitted to the kernel tree, so you might didn't got them.
>
> /drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_if.c/
> / /
> /Or at least, it does a memcpy to the new buffer and it has some buffer refcounting,/
> /although I didn't go deep into the details. In any case, this might help./
> /
> /
> if (frm_to_show->fb != inst->cur_fb) {
> /* This frame is show exist frame and no decode output
> * copy frame data from frm_to_show to current CAPTURE
> * buffer
> */
> if ((frm_to_show->fb != NULL) &&
> (inst->cur_fb->base_y.size >=
> frm_to_show->fb->base_y.size) &&
> (inst->cur_fb->base_c.size >=
> frm_to_show->fb->base_c.size)) {
> memcpy((void *)inst->cur_fb->base_y.va,
> (void *)frm_to_show->fb->base_y.va,
> frm_to_show->fb->base_y.size);
> memcpy((void *)inst->cur_fb->base_c.va,
> (void *)frm_to_show->fb->base_c.va,
> frm_to_show->fb->base_c.size);
> }
> You should be talking about this code, If the picture buffer is secure and protected, this memory may be prohibited memcpy, so what should we do?
Yeah, that won't work for protected memory. But if it is in protected memory, doesn't that mean that you are using dmabuf for the actual buffer?
We don't have any codec drivers that work with protected memory in the kernel, but the only way to support that would be through dmabuf.
Question: does VP9 indicate that a buffer will be repeated when you receive the first buffer of that sequence? I.e., do you know when you dequeue the first buffer that it will be repeated? Or do you only discover that afterwards?
I'm also assuming that you are talking about a stateful codec driver, not a stateless codec driver, correct?
Regards,
Hans
> Our idea is to dque this buffer to user space multiple times through the videobuf2 framework.
> But v4l2 does not support this behavior, if the buffer is not returned to the driver, it will be output again.
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> nanxin.qin@xxxxxxxxxxx
>
>
> *From:* Hans Verkuil <mailto:hverkuil-cisco@xxxxxxxxx>
> *Date:* 2022-07-14 15:59
> *To:* nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>; xiaohang.cui <mailto:xiaohang.cui@xxxxxxxxxxx>; arch-proaudio <mailto:arch-proaudio@xxxxxxxxxxxxxxxxxxx>
> *CC:* hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; zhi.zhou <mailto:zhi.zhou@xxxxxxxxxxx>
> *Subject:* Re: The problem of VP9 decoder adapter V4L2.
> [ EXTERNAL EMAIL ]
>
> Hmm, it looks I missed the earlier email.
>
> In any case, please post this to the linux-media@xxxxxxxxxxxxxxx mailinglist.
>
> Please mention which driver you are using, is it drivers/staging/media/meson/vdec?
>
> If so, then clearly mention this in the subject line.
>
> If it is an out-of-tree driver, then also mention that, ideally with a link to the code.
>
> By posting to the mailinglist you'll reach the real experts on codecs.
>
> Based on a quick look at the meson driver it looks like it does not support this
> feature. It can detect it, but it doesn't appear to do anything with that
> information. But the mediatek decoder appears to handle this situation properly:
>
> drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_if.c
>
> Or at least, it does a memcpy to the new buffer and it has some buffer refcounting,
> although I didn't go deep into the details. In any case, this might help.
>
> Regards,
>
> Hans
>
> On 7/14/22 09:32, nanxin.qin@xxxxxxxxxxx wrote:
> > Hi, Hans
> >
> > I'm sorry to bother you again. Do you have any good suggestions for this problem? The VP9 spec development team suggested a change to the V4L2 API to allow the vb2 can be deque more times, which seemed difficult.
> >
> >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > nanxin.qin@xxxxxxxxxxx
> >
> >
> > *From:* xiaohang.cui@xxxxxxxxxxx <mailto:xiaohang.cui@xxxxxxxxxxx>
> > *Date:* 2022-06-02 16:26
> > *To:* arch-proaudio <mailto:arch-proaudio@xxxxxxxxxxxxxxxxxxx>
> > *CC:* hverkuil-cisco@xxxxxxxxx <mailto:hverkuil-cisco@xxxxxxxxx>; hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; Nanxin Qin <mailto:Nanxin.Qin@xxxxxxxxxxx>
> > *Subject:* Re: Re: The problem of VP9 decoder adapter V4L2.
> > Dear linux.org multi-media team,
> >
> > I am a multimedia software engineer from AmLogic. We have encountered some decoder problems on V4L2 when supporting VP9.
> >
> > Reading the VP9 specification, show_existing_frame is a feature added in intra-only to solve the problem of video conferencing. If show_existing_frame is equal to 1, the decoder outputs the previously decoded frame. This part is explained in the specification output process.
> >
> > At present, we need to implement VP9 decoder based on V4L2. All decoded frames need to be passed to userspace using VB2 buffer. If show_existing_frame exists, the same frame may be output multiple times. If VB2 buffer was dequed to userspace by APP, and no queue into the videobuf2 driver, then the VP9 decoder driver will fail to output the frame again because the VB2 buffer has not returned to the driver, In other words the buffer's owner is APP not decode driver. So the decoder cannot pass the frame from the VB2 buffer again.
> >
> > According to the above analysis, the show_existing_frame feature seems to conflict with the V4L2 Videobuf2 framework because V4L2 does not allow the same VB2 buffer to be output more than once.
> >
> > I would like to ask, for VP9 decoder adaptation on V4L2, how to should deal with the problem of repeated output frame?
> >
> > I'm looking forward to receiving your reply. Thank you very much.
> >
> > Attachment is the coded stream for reference (big_buck_bunny_720p.mp4).
> >
> >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > 崔晓行
> > SW Department
> >
> > Amlogic Inc. Shenzhen office
> >
> > -------------------------------------------------------------
> >
> > 深圳市南山区科技南四道8号创维半导体设计大厦东座9-10层
> >
> > Tel: 0755-86315800-8169
> >
> >
> > *From:* nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>
> > *Date:* 2022-05-25 16:50
> > *To:* peter.derivaz <mailto:peter.derivaz@xxxxxxxxx>
> > *CC:* agrange <mailto:agrange@xxxxxxxxxx>; tinghai.chen <mailto:tinghai.chen@xxxxxxxxxxx>; hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; xiaohang.cui <mailto:xiaohang.cui@xxxxxxxxxxx>; yixin.peng <mailto:yixin.peng@xxxxxxxxxxx>; Gan Zhang <mailto:Gan.Zhang@xxxxxxxxxxx>; kuan.hu <mailto:Kuan.Hu@xxxxxxxxxxx>; miaohong.chen <mailto:miaohong.chen@xxxxxxxxxxx>; lele.xiang <mailto:lele.xiang@xxxxxxxxxxx>; Shihong.Zheng <mailto:shihong.zheng@xxxxxxxxxxx>
> > *Subject:* Re: Re: The problem of VP9 decoder adapter V4L2.
> > Hi, Peter
> >
> > Thanks for your reply, we will try to feedback this problem with Linux, we plan to implement the second solution according to your proposal, which may be more convenient for us to implement, thanks again.
> >
> >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > nanxin.qin
> > SW Department____
> >
> > Amlogic Inc. Shenzhen office____
> >
> > -------------------------------------------------------------____
> >
> > 深圳市南山区科技南四道8号创维半导体设计大厦东座9-10层____
> >
> > Tel: 0755-86315800-8169
> >
> >
> > *From:* Peter de Rivaz <mailto:peter.derivaz@xxxxxxxxx>
> > *Date:* 2022-05-25 16:33
> > *To:* nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>
> > *CC:* agrange <mailto:agrange@xxxxxxxxxx>; tinghai.chen <mailto:tinghai.chen@xxxxxxxxxxx>; hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; xiaohang.cui <mailto:xiaohang.cui@xxxxxxxxxxx>; yixin.peng <mailto:yixin.peng@xxxxxxxxxxx>; Zhang <mailto:Gan.Zhang@xxxxxxxxxxx>; kuan.hu <mailto:Kuan.Hu@xxxxxxxxxxx>; miaohong.chen <mailto:miaohong.chen@xxxxxxxxxxx>; lele.xiang <mailto:lele.xiang@xxxxxxxxxxx>; Shihong.Zheng <mailto:shihong.zheng@xxxxxxxxxxx>
> > *Subject:* Re: The problem of VP9 decoder adapter V4L2.
> > **[ EXTERNAL EMAIL ]**
> >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > I can see two choices:
> > 1)
> > You could change the V4L2 API to allow the buffer to be returned multiple times (this would involve discussion on the Linux mailing lists as it is not something within our control).
> >
> > 2)
> > You could treat the frame with show_existing_frame=1 as a normal frame with zero motion vectors and no residual.
> > In other words, you just copy the reference frame data from the frame indicated by frame_to_show_map_idx into a new frame and return it.
> >
> > Option 1 would be a bit more efficient, but option 2 is probably simpler to implement.
> >
> > Peter de Rivaz
> >
> > On Wed, May 25, 2022 at 8:55 AM nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx> <nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>> wrote:
> >
> > Dear:
> > agrange, peter.derivaz
> >
> > I am a multimedia software engineer from AmLogic. We have encountered some problems when supporting VP9 decoder + V4L2.
> >
> > Reading the VP9 specification, show_existing_frame is a feature added in intra-only to solve the problem of video conferencing. If show_existing_frame is equal to 1, the decoder outputs the previously decoded frame. This part is explained in the specification output process.
> >
> > At present, we need to implement VP9 decoder based on V4L2. All decoded frames need to be passed to userspace using VB2 buffer. If show_existing_frame exists, the same frame may be output multiple times. If VB2 buffer was dequed to userspace by APP, and no queue into the videobuf2 driver, then the VP9 decoder driver will fail to output the frame again because the VB2 buffer has not returned to the driver, In other words the buffer's onwer is APP not decode driver. So the decoder cannot pass the frame from the VB2 buffer again.
> >
> > According to the above analysis, the show_existing_frame feature seems to conflict with the V4L2 Videobuf2 framework because V4L2 does not allow the same VB2 buffer to be output more than once.
> >
> > I would like to ask, for VP9 decoder adaptation V4L2 should be how to deal with the problem of repeated output frame?
> >
> > Thank you very much for your reply.
> >
> > Attached is the code stream for reference (big_buck_bunny_720p.mp4).
> >
> >
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > nanxin.qin
> > SW Department____
> >
> > Amlogic Inc. Shenzhen office____
> >
> > -------------------------------------------------------------____
> >
> > 深圳市南山区科技南四道8号创维半导体设计大厦东座9-10层____
> >
> > Tel: 0755-86315800-8169
> >
>
>
