Hi Alessandro On Fri, 10 May 2024 at 11:42, Alessandro Longobardi <alessandro.longobardi@xxxxxxxxxx> wrote: > > > Hi, > I am using the h264_v4l2m2m decoder with libavcodec on a Raspberry Pi Zero 2 running Bookworm OS. My codec has the AV_CODEC_CAP_DELAY capability set, which means "The decoder has a non-zero delay" (reference: https://ffmpeg.org/doxygen/6.0/group__lavc__core.html#ga3f55f5bcfbb12e06c7cb1195028855e6). I have observed that the decoder stores N encoded frames before outputting a decoded frame, and the value of N varies depending on the resolution, with higher resolution leading to a larger N. It means i have a delay in decoding about N*time_per_frame (e.g., at 25 fps, time_per_frame=40 milliseconds). The bcm2835 stateful decoder driver isn't in mainline (yet), so this is the wrong place to be asking this question. I've also already responded to your email yesterday and your forum post[1]. It looks like it is FFmpeg getting a little overeager in the number of buffers it allocates (20 on both OUTPUT and CAPTURE queues), and ends up trying to fill all buffers on one queue before looking at the other one, so decode becomes quite lumpy. Reduce the number of OUTPUT buffers to 2, and after waiting for the DPB to be full, decode latency is 1 frame (assuming no B-frames). Dave [1] https://forums.raspberrypi.com/viewtopic.php?t=370289 > Is there a way to reduce this N value? > > I am using an h624 1280x960 baseline streaming, I am observing that it stores internally 4 encoded frames before outputting a decoded one (check the log below): > AV_CODEC_CAP_DELAY is set for the codec. > Codec Format: h264_v4l2m2m > [h264_mp4toannexb @ 0x68201020] The input looks like it is Annex B already > [h264_v4l2m2m @ 0x68200490] probing device /dev/video1 > [h264_v4l2m2m @ 0x68200490] driver 'unicam' on card 'unicam' in splane mode > [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported > [h264_v4l2m2m @ 0x68200490] probing device /dev/video0 > [h264_v4l2m2m @ 0x68200490] driver 'unicam' on card 'unicam' in splane mode > [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported > [h264_v4l2m2m @ 0x68200490] probing device /dev/video31 > [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-encode_image' in mplane mode > [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported > [h264_v4l2m2m @ 0x68200490] probing device /dev/video18 > [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-image_fx' in mplane mode > [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported > [h264_v4l2m2m @ 0x68200490] probing device /dev/video12 > [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode > [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported > [h264_v4l2m2m @ 0x68200490] probing device /dev/video11 > [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode > [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported > [h264_v4l2m2m @ 0x68200490] probing device /dev/video10 > [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode > [h264_v4l2m2m @ 0x68200490] Using device /dev/video10 > [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode > [h264_v4l2m2m @ 0x68200490] requesting formats: output=H264 capture=YU12 > [h264_v4l2m2m @ 0x68200490] output: H264 16 buffers initialized: 0000x0000, sizeimage 00524288, bytesperline 00000000 > [h264_v4l2m2m @ 0x68200490] Driver 'bcm2835-codec': Quirks=0 > [h264_v4l2m2m @ 0x68200490] Profile -99 <= 0 - check skipped > [h264_v4l2m2m @ 0x68200490] Format drm_prime chosen by get_format(). > [h264_v4l2m2m @ 0x68200490] avctx requested=-1 ((null)) 0x0; get_format requested=179 (drm_prime) > plane_id: 61 > sending an encoded pkt, tot_pkts sent: 1 > [h264_v4l2m2m @ 0x68200490] output set status ON OK > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000001 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000001 count=1 > Execution time of decode_write: 4.303000 milliseconds > encoded pkt -> decoded frame: 7 millisecs > sending an encoded pkt, tot_pkts sent: 2 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000002 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000002 count=1 > Execution time of decode_write: 0.648000 milliseconds > encoded pkt -> decoded frame: 4 millisecs > sending an encoded pkt, tot_pkts sent: 3 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000003 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000003 count=1 > [h264_v4l2m2m @ 0x68200490] Dq event 5 > [h264_v4l2m2m @ 0x68200490] V4L2 capture changed: alloc=0 (32x32) -> (1280x960) > [h264_v4l2m2m @ 0x68200490] Source change: Fmt: YU12, SAR: 0/0, wxh 1280x960 crop 1280x960 @ 0,0, reinit=1 > [h264_v4l2m2m @ 0x68200490] capture: YU12 20 buffers initialized: 1280x0960, sizeimage 01843200, bytesperline 00001280 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 0, ts=0.000000 count=1 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 1, ts=0.000000 count=2 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 2, ts=0.000000 count=3 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 3, ts=0.000000 count=4 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 4, ts=0.000000 count=5 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 5, ts=0.000000 count=6 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 6, ts=0.000000 count=7 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 7, ts=0.000000 count=8 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 8, ts=0.000000 count=9 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 9, ts=0.000000 count=10 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 10, ts=0.000000 count=11 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 11, ts=0.000000 count=12 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 12, ts=0.000000 count=13 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 13, ts=0.000000 count=14 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 14, ts=0.000000 count=15 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 15, ts=0.000000 count=16 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 16, ts=0.000000 count=17 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 17, ts=0.000000 count=18 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 18, ts=0.000000 count=19 > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 19, ts=0.000000 count=20 > [h264_v4l2m2m @ 0x68200490] capture set status ON OK > Execution time of decode_write: 70.186000 milliseconds > encoded pkt -> decoded frame: 72 millisecs > sending an encoded pkt, tot_pkts sent: 4 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000004 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000004 count=1 > Execution time of decode_write: 0.636000 milliseconds > encoded pkt -> decoded frame: 30 millisecs > sending an encoded pkt, tot_pkts sent: 5 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 1, ts=0.000005 count=1 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 1, ts=0.000005 count=2 > [h264_v4l2m2m @ 0x68200490] ff_v4l2_context_frames_set: HWFramesContext set to yuv420p, 1280x960 > [h264_v4l2m2m @ 0x68200490] Decode running ////////////////////////////////////////////////////////////////////////////////////////////////////////// ←- FROME HERE I STARTS RECEIVING DECODED FRAME, THEN MY STREAMING IS DELAYED OF 4 FRAMES. > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 1 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 0, ts=0.000001 count=20 > Execution time of decode_write: 0.964000 milliseconds > //FROM NOW I AM GETTING EXACTLY ONE DECODED FRAME FOR EACH ENCODED PACKET (AS EXPECTED) > encoded pkt -> decoded frame: 2 millisecs > sending an encoded pkt, tot_pkts sent: 6 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000006 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000006 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 2 > h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 1, ts=0.000002 count=20 > Execution time of decode_write: 1.181000 milliseconds > encoded pkt -> decoded frame: 3 millisecs > sending an encoded pkt, tot_pkts sent: 7 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000007 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000007 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 3 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 2, ts=0.000003 count=20 > Execution time of decode_write: 0.883000 milliseconds > encoded pkt -> decoded frame: 2 millisecs > sending an encoded pkt, tot_pkts sent: 8 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000008 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000008 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 4 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 3, ts=0.000004 count=20 > Execution time of decode_write: 42.602000 milliseconds > encoded pkt -> decoded frame: 44 millisecs > sending an encoded pkt, tot_pkts sent: 9 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000009 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000009 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 5 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 4, ts=0.000005 count=20 > Execution time of decode_write: 0.948000 milliseconds > encoded pkt -> decoded frame: 7 millisecs > sending an encoded pkt, tot_pkts sent: 10 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000010 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000010 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 6 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 5, ts=0.000006 count=20 > Execution time of decode_write: 0.993000 milliseconds > encoded pkt -> decoded frame: 2 millisecs > sending an encoded pkt, tot_pkts sent: 11 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000011 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000011 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 7 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 6, ts=0.000007 count=20 > Execution time of decode_write: 1.356000 milliseconds > encoded pkt -> decoded frame: 4 millisecs > sending an encoded pkt, tot_pkts sent: 12 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000012 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000012 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 8 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 7, ts=0.000008 count=20 > Execution time of decode_write: 15.023000 milliseconds > encoded pkt -> decoded frame: 21 millisecs > sending an encoded pkt, tot_pkts sent: 13 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000013 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000013 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 9 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 8, ts=0.000009 count=20 > Execution time of decode_write: 18.632000 milliseconds > encoded pkt -> decoded frame: 27 millisecs > sending an encoded pkt, tot_pkts sent: 14 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000014 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000014 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 10 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 9, ts=0.000010 count=20 > Execution time of decode_write: 8.249000 milliseconds > encoded pkt -> decoded frame: 12 millisecs > sending an encoded pkt, tot_pkts sent: 15 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000015 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000015 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 11 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 10, ts=0.000011 count=20 > Execution time of decode_write: 9.017000 milliseconds > encoded pkt -> decoded frame: 15 millisecs > sending an encoded pkt, tot_pkts sent: 16 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000016 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000016 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 12 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 11, ts=0.000012 count=20 > Execution time of decode_write: 0.882000 milliseconds > encoded pkt -> decoded frame: 2 millisecs > sending an encoded pkt, tot_pkts sent: 17 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000017 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000017 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 13 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 12, ts=0.000013 count=20 > Execution time of decode_write: 1.041000 milliseconds > encoded pkt -> decoded frame: 5 millisecs > sending an encoded pkt, tot_pkts sent: 18 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000018 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000018 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 14 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 13, ts=0.000014 count=20 > Execution time of decode_write: 0.865000 milliseconds > encoded pkt -> decoded frame: 3 millisecs > sending an encoded pkt, tot_pkts sent: 19 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000019 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000019 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 15 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 14, ts=0.000015 count=20 > Execution time of decode_write: 6.163000 milliseconds > encoded pkt -> decoded frame: 8 millisecs > sending an encoded pkt, tot_pkts sent: 20 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000020 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000020 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 16 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 15, ts=0.000016 count=20 > Execution time of decode_write: 3.917000 milliseconds > encoded pkt -> decoded frame: 8 millisecs > sending an encoded pkt, tot_pkts sent: 21 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000021 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000021 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 17 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 16, ts=0.000017 count=20 > Execution time of decode_write: 1.443000 milliseconds > encoded pkt -> decoded frame: 4 millisecs > sending an encoded pkt, tot_pkts sent: 22 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000022 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000022 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 18 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 17, ts=0.000018 count=20 > Execution time of decode_write: 10.558000 milliseconds > encoded pkt -> decoded frame: 13 millisecs > sending an encoded pkt, tot_pkts sent: 23 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000023 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000023 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 19 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 18, ts=0.000019 count=20 > Execution time of decode_write: 1.837000 milliseconds > encoded pkt -> decoded frame: 5 millisecs > sending an encoded pkt, tot_pkts sent: 24 > [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000024 count=0 > [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000024 count=1 > frame received ! type: 0 ,keyframe: 0, tot frame decoded: 20 > [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue > [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 19, ts=0.000020 count=20 > Execution time of decode_write: 1.110000 milliseconds > encoded pkt -> decoded frame: 3 millisecs > … > > Thank you, > Alessandro Longobardi > >
