Hi, I am using the h264_v4l2m2m decoder with libavcodec on a Raspberry Pi Zero 2 running Bookworm OS. My codec has the AV_CODEC_CAP_DELAY capability set, which means "The decoder has a non-zero delay" (reference: https://ffmpeg.org/doxygen/6.0/group__lavc__core.html#ga3f55f5bcfbb12e06c7cb1195028855e6). I have observed that the decoder stores N encoded frames before outputting a decoded frame, and the value of N varies depending on the resolution, with higher resolution leading to a larger N. It means i have a delay in decoding about N*time_per_frame (e.g., at 25 fps, time_per_frame=40 milliseconds). Is there a way to reduce this N value? I am using an h624 1280x960 baseline streaming, I am observing that it stores internally 4 encoded frames before outputting a decoded one (check the log below): AV_CODEC_CAP_DELAY is set for the codec. Codec Format: h264_v4l2m2m [h264_mp4toannexb @ 0x68201020] The input looks like it is Annex B already [h264_v4l2m2m @ 0x68200490] probing device /dev/video1 [h264_v4l2m2m @ 0x68200490] driver 'unicam' on card 'unicam' in splane mode [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported [h264_v4l2m2m @ 0x68200490] probing device /dev/video0 [h264_v4l2m2m @ 0x68200490] driver 'unicam' on card 'unicam' in splane mode [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported [h264_v4l2m2m @ 0x68200490] probing device /dev/video31 [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-encode_image' in mplane mode [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported [h264_v4l2m2m @ 0x68200490] probing device /dev/video18 [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-image_fx' in mplane mode [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported [h264_v4l2m2m @ 0x68200490] probing device /dev/video12 [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported [h264_v4l2m2m @ 0x68200490] probing device /dev/video11 [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0x68200490] v4l2 output format not supported [h264_v4l2m2m @ 0x68200490] probing device /dev/video10 [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode [h264_v4l2m2m @ 0x68200490] Using device /dev/video10 [h264_v4l2m2m @ 0x68200490] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode [h264_v4l2m2m @ 0x68200490] requesting formats: output=H264 capture=YU12 [h264_v4l2m2m @ 0x68200490] output: H264 16 buffers initialized: 0000x0000, sizeimage 00524288, bytesperline 00000000 [h264_v4l2m2m @ 0x68200490] Driver 'bcm2835-codec': Quirks=0 [h264_v4l2m2m @ 0x68200490] Profile -99 <= 0 - check skipped [h264_v4l2m2m @ 0x68200490] Format drm_prime chosen by get_format(). [h264_v4l2m2m @ 0x68200490] avctx requested=-1 ((null)) 0x0; get_format requested=179 (drm_prime) plane_id: 61 sending an encoded pkt, tot_pkts sent: 1 [h264_v4l2m2m @ 0x68200490] output set status ON OK [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000001 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000001 count=1 Execution time of decode_write: 4.303000 milliseconds encoded pkt -> decoded frame: 7 millisecs sending an encoded pkt, tot_pkts sent: 2 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000002 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000002 count=1 Execution time of decode_write: 0.648000 milliseconds encoded pkt -> decoded frame: 4 millisecs sending an encoded pkt, tot_pkts sent: 3 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000003 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000003 count=1 [h264_v4l2m2m @ 0x68200490] Dq event 5 [h264_v4l2m2m @ 0x68200490] V4L2 capture changed: alloc=0 (32x32) -> (1280x960) [h264_v4l2m2m @ 0x68200490] Source change: Fmt: YU12, SAR: 0/0, wxh 1280x960 crop 1280x960 @ 0,0, reinit=1 [h264_v4l2m2m @ 0x68200490] capture: YU12 20 buffers initialized: 1280x0960, sizeimage 01843200, bytesperline 00001280 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 0, ts=0.000000 count=1 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 1, ts=0.000000 count=2 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 2, ts=0.000000 count=3 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 3, ts=0.000000 count=4 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 4, ts=0.000000 count=5 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 5, ts=0.000000 count=6 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 6, ts=0.000000 count=7 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 7, ts=0.000000 count=8 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 8, ts=0.000000 count=9 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 9, ts=0.000000 count=10 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 10, ts=0.000000 count=11 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 11, ts=0.000000 count=12 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 12, ts=0.000000 count=13 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 13, ts=0.000000 count=14 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 14, ts=0.000000 count=15 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 15, ts=0.000000 count=16 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 16, ts=0.000000 count=17 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 17, ts=0.000000 count=18 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 18, ts=0.000000 count=19 [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 19, ts=0.000000 count=20 [h264_v4l2m2m @ 0x68200490] capture set status ON OK Execution time of decode_write: 70.186000 milliseconds encoded pkt -> decoded frame: 72 millisecs sending an encoded pkt, tot_pkts sent: 4 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000004 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000004 count=1 Execution time of decode_write: 0.636000 milliseconds encoded pkt -> decoded frame: 30 millisecs sending an encoded pkt, tot_pkts sent: 5 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 1, ts=0.000005 count=1 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 1, ts=0.000005 count=2 [h264_v4l2m2m @ 0x68200490] ff_v4l2_context_frames_set: HWFramesContext set to yuv420p, 1280x960 [h264_v4l2m2m @ 0x68200490] Decode running ////////////////////////////////////////////////////////////////////////////////////////////////////////// ←- FROME HERE I STARTS RECEIVING DECODED FRAME, THEN MY STREAMING IS DELAYED OF 4 FRAMES. frame received ! type: 0 ,keyframe: 0, tot frame decoded: 1 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 0, ts=0.000001 count=20 Execution time of decode_write: 0.964000 milliseconds //FROM NOW I AM GETTING EXACTLY ONE DECODED FRAME FOR EACH ENCODED PACKET (AS EXPECTED) encoded pkt -> decoded frame: 2 millisecs sending an encoded pkt, tot_pkts sent: 6 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000006 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000006 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 2 h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 1, ts=0.000002 count=20 Execution time of decode_write: 1.181000 milliseconds encoded pkt -> decoded frame: 3 millisecs sending an encoded pkt, tot_pkts sent: 7 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000007 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000007 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 3 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 2, ts=0.000003 count=20 Execution time of decode_write: 0.883000 milliseconds encoded pkt -> decoded frame: 2 millisecs sending an encoded pkt, tot_pkts sent: 8 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000008 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000008 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 4 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 3, ts=0.000004 count=20 Execution time of decode_write: 42.602000 milliseconds encoded pkt -> decoded frame: 44 millisecs sending an encoded pkt, tot_pkts sent: 9 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000009 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000009 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 5 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 4, ts=0.000005 count=20 Execution time of decode_write: 0.948000 milliseconds encoded pkt -> decoded frame: 7 millisecs sending an encoded pkt, tot_pkts sent: 10 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000010 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000010 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 6 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 5, ts=0.000006 count=20 Execution time of decode_write: 0.993000 milliseconds encoded pkt -> decoded frame: 2 millisecs sending an encoded pkt, tot_pkts sent: 11 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000011 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000011 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 7 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 6, ts=0.000007 count=20 Execution time of decode_write: 1.356000 milliseconds encoded pkt -> decoded frame: 4 millisecs sending an encoded pkt, tot_pkts sent: 12 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000012 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000012 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 8 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 7, ts=0.000008 count=20 Execution time of decode_write: 15.023000 milliseconds encoded pkt -> decoded frame: 21 millisecs sending an encoded pkt, tot_pkts sent: 13 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000013 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000013 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 9 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 8, ts=0.000009 count=20 Execution time of decode_write: 18.632000 milliseconds encoded pkt -> decoded frame: 27 millisecs sending an encoded pkt, tot_pkts sent: 14 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000014 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000014 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 10 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 9, ts=0.000010 count=20 Execution time of decode_write: 8.249000 milliseconds encoded pkt -> decoded frame: 12 millisecs sending an encoded pkt, tot_pkts sent: 15 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000015 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000015 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 11 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 10, ts=0.000011 count=20 Execution time of decode_write: 9.017000 milliseconds encoded pkt -> decoded frame: 15 millisecs sending an encoded pkt, tot_pkts sent: 16 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000016 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000016 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 12 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 11, ts=0.000012 count=20 Execution time of decode_write: 0.882000 milliseconds encoded pkt -> decoded frame: 2 millisecs sending an encoded pkt, tot_pkts sent: 17 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000017 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000017 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 13 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 12, ts=0.000013 count=20 Execution time of decode_write: 1.041000 milliseconds encoded pkt -> decoded frame: 5 millisecs sending an encoded pkt, tot_pkts sent: 18 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000018 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000018 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 14 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 13, ts=0.000014 count=20 Execution time of decode_write: 0.865000 milliseconds encoded pkt -> decoded frame: 3 millisecs sending an encoded pkt, tot_pkts sent: 19 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000019 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000019 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 15 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 14, ts=0.000015 count=20 Execution time of decode_write: 6.163000 milliseconds encoded pkt -> decoded frame: 8 millisecs sending an encoded pkt, tot_pkts sent: 20 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000020 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000020 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 16 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 15, ts=0.000016 count=20 Execution time of decode_write: 3.917000 milliseconds encoded pkt -> decoded frame: 8 millisecs sending an encoded pkt, tot_pkts sent: 21 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000021 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000021 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 17 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 16, ts=0.000017 count=20 Execution time of decode_write: 1.443000 milliseconds encoded pkt -> decoded frame: 4 millisecs sending an encoded pkt, tot_pkts sent: 22 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000022 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000022 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 18 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 17, ts=0.000018 count=20 Execution time of decode_write: 10.558000 milliseconds encoded pkt -> decoded frame: 13 millisecs sending an encoded pkt, tot_pkts sent: 23 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000023 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000023 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 19 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 18, ts=0.000019 count=20 Execution time of decode_write: 1.837000 milliseconds encoded pkt -> decoded frame: 5 millisecs sending an encoded pkt, tot_pkts sent: 24 [h264_v4l2m2m @ 0x68200490] --- output pre VIDIOC_QBUF: index 0, ts=0.000024 count=0 [h264_v4l2m2m @ 0x68200490] --- output VIDIOC_QBUF: index 0, ts=0.000024 count=1 frame received ! type: 0 ,keyframe: 0, tot frame decoded: 20 [h264_v4l2m2m @ 0x68200490] capture: Buffer requeue [h264_v4l2m2m @ 0x68200490] --- capture VIDIOC_QBUF: index 19, ts=0.000020 count=20 Execution time of decode_write: 1.110000 milliseconds encoded pkt -> decoded frame: 3 millisecs … Thank you, Alessandro Longobardi
