Hi,
here's some note of the discussion we had on IRC. First thing to mention, this
is a none-issue for stateless decoders, since the special frame with
show_existing_frame flag set are just never sent to the kernel in that case.
This only affect stateful decoders (which not much design time was put in
really).
The show_existing_frame flag have two uses cases. The most common one is for
video playback, it is used to simulate B-Frame (future references). In the case,
the frames is first decoded, but not outputed. It is part of the same temporal
unit (super frame) as another frame. This frame will use a VB2 buffer, but it
won't be displayed. In that case, it will only be displayed later once, with
show_existing_frame, and for that, you can just delay the buffer_done marking.
The problematic case would be for still picture handling (video with repeated
images and no noise). In this case, the same frames is being displayed over and
over. You could use this while encoding slide deck. There is currently no zero-
copy mechanism in place to allow this in VB2 design. I've made a suggestion to
Hans (which I doubt will be adopted), so I will leave that there for now.
regards,
Nicolas
Le jeudi 14 juillet 2022 à 13:03 +0200, Hans Verkuil a écrit :
>
> On 7/14/22 12:18, nanxin.qin@xxxxxxxxxxx wrote:
> > Hi, Hans
> > thanks for your reply. Currently, our source code has not been submitted to
> > the kernel tree, so you might didn't got them.
> >
> > /drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_if.c/
> > / /
> > /Or at least, it does a memcpy to the new buffer and it has some buffer
> > refcounting,/
> > /although I didn't go deep into the details. In any case, this might help./
> > /
> > /
> > if (frm_to_show->fb != inst->cur_fb) {
> > /* This frame is show exist frame and no decode output
> > * copy frame data from frm_to_show to current CAPTURE
> > * buffer
> > */
> > if ((frm_to_show->fb != NULL) &&
> > (inst->cur_fb->base_y.size >=
> > frm_to_show->fb->base_y.size) &&
> > (inst->cur_fb->base_c.size >=
> > frm_to_show->fb->base_c.size)) {
> > memcpy((void *)inst->cur_fb->base_y.va,
> > (void *)frm_to_show->fb->base_y.va,
> > frm_to_show->fb->base_y.size);
> > memcpy((void *)inst->cur_fb->base_c.va,
> > (void *)frm_to_show->fb->base_c.va,
> > frm_to_show->fb->base_c.size);
> > }
> > You should be talking about this code, If the picture buffer is secure and
> > protected, this memory may be prohibited memcpy, so what should we do?
>
> Yeah, that won't work for protected memory. But if it is in protected memory,
> doesn't that mean that you are using dmabuf for the actual buffer?
Please note that older (stateful) Mediatek drivers will are unlikely to gain
secure memory (this is old HW). But if they do, Mediatek have a firmware running
on a pretty generic co-processor that sits ont the secure side. They could
simply add a function in the firmware to solve this problem.
On the secure note, be aware that another challenge exist with the handling the
the probability context updates. I notice that, similar to most stateless
decoders, Amlogic HW requires the driver to do the update. The probability
tabled filled during decode will inevitably have to be protected. You need a
special firmware feature to copy this table back and fort to allow the
probability updates. Or have that update happen forward (a bit like the VP9
stateless uAPI), but executed in secure space with compressed header being
parsed there* (firmware, OPT-TEE, ...).
* in Widevine L1, there is a small design error regarding VP9 and compressed
headers are encrypted
>
> We don't have any codec drivers that work with protected memory in the kernel,
> but the only way to support that would be through dmabuf.
>
> Question: does VP9 indicate that a buffer will be repeated when you receive
> the first buffer of that sequence? I.e., do you know when you dequeue the
> first buffer that it will be repeated? Or do you only discover that
> afterwards?
>
> I'm also assuming that you are talking about a stateful codec driver, not a
> stateless codec driver, correct?
>
> Regards,
>
> Hans
>
> > Our idea is to dque this buffer to user space multiple times through the videobuf2 framework.
> > But v4l2 does not support this behavior, if the buffer is not returned to the driver, it will be output again.
> >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > nanxin.qin@xxxxxxxxxxx
> >
> >
> > *From:* Hans Verkuil <mailto:hverkuil-cisco@xxxxxxxxx>
> > *Date:* 2022-07-14 15:59
> > *To:* nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>; xiaohang.cui <mailto:xiaohang.cui@xxxxxxxxxxx>; arch-proaudio <mailto:arch-proaudio@xxxxxxxxxxxxxxxxxxx>
> > *CC:* hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; zhi.zhou <mailto:zhi.zhou@xxxxxxxxxxx>
> > *Subject:* Re: The problem of VP9 decoder adapter V4L2.
> > [ EXTERNAL EMAIL ]
> >
> > Hmm, it looks I missed the earlier email.
> >
> > In any case, please post this to the linux-media@xxxxxxxxxxxxxxx mailinglist.
> >
> > Please mention which driver you are using, is it drivers/staging/media/meson/vdec?
> >
> > If so, then clearly mention this in the subject line.
> >
> > If it is an out-of-tree driver, then also mention that, ideally with a link to the code.
> >
> > By posting to the mailinglist you'll reach the real experts on codecs.
> >
> > Based on a quick look at the meson driver it looks like it does not support this
> > feature. It can detect it, but it doesn't appear to do anything with that
> > information. But the mediatek decoder appears to handle this situation properly:
> >
> > drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_if.c
> >
> > Or at least, it does a memcpy to the new buffer and it has some buffer refcounting,
> > although I didn't go deep into the details. In any case, this might help.
> >
> > Regards,
> >
> > Hans
> >
> > On 7/14/22 09:32, nanxin.qin@xxxxxxxxxxx wrote:
> > > Hi, Hans
> > >
> > > I'm sorry to bother you again. Do you have any good suggestions for
> > this problem? The VP9 spec development team suggested a change to the V4L2
> > API to allow the vb2 can be deque more times, which seemed difficult.
> > >
> > >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > nanxin.qin@xxxxxxxxxxx
> > >
> > >
> > > *From:* xiaohang.cui@xxxxxxxxxxx <mailto:xiaohang.cui@xxxxxxxxxxx>
> > > *Date:* 2022-06-02 16:26
> > > *To:* arch-proaudio <mailto:arch-proaudio@xxxxxxxxxxxxxxxxxxx>
> > > *CC:* hverkuil-cisco@xxxxxxxxx <mailto:hverkuil-cisco@xxxxxxxxx>; hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; Nanxin Qin <mailto:Nanxin.Qin@xxxxxxxxxxx>
> > > *Subject:* Re: Re: The problem of VP9 decoder adapter V4L2.
> > > Dear linux.org multi-media team,
> > >
> > > I am a multimedia software engineer from AmLogic. We have
> > encountered some decoder problems on V4L2 when supporting VP9.
> > >
> > > Reading the VP9 specification, show_existing_frame is a feature
> > added in intra-only to solve the problem of video conferencing. If
> > show_existing_frame is equal to 1, the decoder outputs the previously
> > decoded frame. This part is explained in the specification output process.
> > >
> > > At present, we need to implement VP9 decoder based on V4L2. All
> > decoded frames need to be passed to userspace using VB2 buffer. If
> > show_existing_frame exists, the same frame may be output multiple times. If
> > VB2 buffer was dequed to userspace by APP, and no queue into the videobuf2
> > driver, then the VP9 decoder driver will fail to output the frame again
> > because the VB2 buffer has not returned to the driver, In other words the
> > buffer's owner is APP not decode driver. So the decoder cannot pass the
> > frame from the VB2 buffer again.
> > >
> > > According to the above analysis, the show_existing_frame feature
> > seems to conflict with the V4L2 Videobuf2 framework because V4L2 does not
> > allow the same VB2 buffer to be output more than once.
> > >
> > > I would like to ask, for VP9 decoder adaptation on V4L2, how to
> > should deal with the problem of repeated output frame?
> > >
> > > I'm looking forward to receiving your reply. Thank you very much.
> > >
> > > Attachment is the coded stream for reference
> > (big_buck_bunny_720p.mp4).
> > >
> > >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > >
> > > 崔晓行
> > > SW Department
> > >
> > > Amlogic Inc. Shenzhen office
> > >
> > > -------------------------------------------------------------
> > >
> > > 深圳市南山区科技南四道8号创维半导体设计大厦东座9-10层
> > >
> > > Tel: 0755-86315800-8169
> > >
> > >
> > > *From:* nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>
> > > *Date:* 2022-05-25 16:50
> > > *To:* peter.derivaz <mailto:peter.derivaz@xxxxxxxxx>
> > > *CC:* agrange <mailto:agrange@xxxxxxxxxx>; tinghai.chen <mailto:tinghai.chen@xxxxxxxxxxx>; hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; xiaohang.cui <mailto:xiaohang.cui@xxxxxxxxxxx>; yixin.peng <mailto:yixin.peng@xxxxxxxxxxx>; Gan Zhang <mailto:Gan.Zhang@xxxxxxxxxxx>; kuan.hu <mailto:Kuan.Hu@xxxxxxxxxxx>; miaohong.chen <mailto:miaohong.chen@xxxxxxxxxxx>; lele.xiang <mailto:lele.xiang@xxxxxxxxxxx>; Shihong.Zheng <mailto:shihong.zheng@xxxxxxxxxxx>
> > > *Subject:* Re: Re: The problem of VP9 decoder adapter V4L2.
> > > Hi, Peter
> > >
> > > Thanks for your reply, we will try to feedback this problem with Linux, we plan to implement the second solution according to your proposal, which may be more convenient for us to implement, thanks again.
> > >
> > >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > >
> > > nanxin.qin
> > > SW Department____
> > >
> > > Amlogic Inc. Shenzhen office____
> > >
> > > -------------------------------------------------------------____
> > >
> > > 深圳市南山区科技南四道8号创维半导体设计大厦东座9-10层____
> > >
> > > Tel: 0755-86315800-8169
> > >
> > >
> > > *From:* Peter de Rivaz <mailto:peter.derivaz@xxxxxxxxx>
> > > *Date:* 2022-05-25 16:33
> > > *To:* nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>
> > > *CC:* agrange <mailto:agrange@xxxxxxxxxx>; tinghai.chen <mailto:tinghai.chen@xxxxxxxxxxx>; hui.zhang <mailto:hui.zhang@xxxxxxxxxxx>; xiaohang.cui <mailto:xiaohang.cui@xxxxxxxxxxx>; yixin.peng <mailto:yixin.peng@xxxxxxxxxxx>; Zhang <mailto:Gan.Zhang@xxxxxxxxxxx>; kuan.hu <mailto:Kuan.Hu@xxxxxxxxxxx>; miaohong.chen <mailto:miaohong.chen@xxxxxxxxxxx>; lele.xiang <mailto:lele.xiang@xxxxxxxxxxx>; Shihong.Zheng <mailto:shihong.zheng@xxxxxxxxxxx>
> > > *Subject:* Re: The problem of VP9 decoder adapter V4L2.
> > > **[ EXTERNAL EMAIL ]**
> > >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > I can see two choices:
> > > 1)
> > > You could change the V4L2 API to allow the buffer to be returned multiple times (this would involve discussion on the Linux mailing lists as it is not something within our control).
> > >
> > > 2)
> > > You could treat the frame with show_existing_frame=1 as a normal frame with zero motion vectors and no residual.
> > > In other words, you just copy the reference frame data from the frame indicated by frame_to_show_map_idx into a new frame and return it.
> > >
> > > Option 1 would be a bit more efficient, but option 2 is probably simpler to implement.
> > >
> > > Peter de Rivaz
> > >
> > > On Wed, May 25, 2022 at 8:55 AM nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx> <nanxin.qin@xxxxxxxxxxx <mailto:nanxin.qin@xxxxxxxxxxx>> wrote:
> > >
> > > Dear:
> > > agrange, peter.derivaz
> > >
> > > I am a multimedia software engineer from AmLogic. We have encountered some problems when supporting VP9 decoder + V4L2.
> > >
> > > Reading the VP9 specification, show_existing_frame
> > is a feature added in intra-only to solve the problem of video conferencing.
> > If show_existing_frame is equal to 1, the decoder outputs the previously
> > decoded frame. This part is explained in the specification output process.
> > >
> > > At present, we need to implement VP9 decoder based
> > on V4L2. All decoded frames need to be passed to userspace using VB2 buffer.
> > If show_existing_frame exists, the same frame may be output multiple times.
> > If VB2 buffer was dequed to userspace by APP, and no queue into the
> > videobuf2 driver, then the VP9 decoder driver will fail to output the frame
> > again because the VB2 buffer has not returned to the driver, In other words
> > the buffer's onwer is APP not decode driver. So the decoder cannot pass the
> > frame from the VB2 buffer again.
> > >
> > > According to the above analysis, the
> > show_existing_frame feature seems to conflict with the V4L2 Videobuf2
> > framework because V4L2 does not allow the same VB2 buffer to be output more
> > than once.
> > >
> > > I would like to ask, for VP9 decoder adaptation
> > V4L2 should be how to deal with the problem of repeated output frame?
> > >
> > > Thank you very much for your reply.
> > >
> > > Attached is the code stream for reference
> > (big_buck_bunny_720p.mp4).
> > >
> > >
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > >
> > > nanxin.qin
> > > SW Department____
> > >
> > > Amlogic Inc. Shenzhen office____
> > >
> > > -------------------------------------------------------------____
> > >
> > > 深圳市南山区科技南四道8号创维半导体设计大厦东座9-10层____
> > >
> > > Tel: 0755-86315800-8169
> > >
> >
> >
