Implement the HEVC codec variation for the RkVDEC driver. Currently only
the RK3399 is supported, but it is possible to enable the RK3288 as it
also supports this codec.
Changelog:
Changes since V1:
- Adjust NV15 documentation according to review requests by Laurent
- Add a new callback to check whether the supplied SPS (H264/HEVC) is
compatible with the provided pixel format on the OUTPUT queue. This is used
to ensure that streaming is only started with a compatible SPS and to unset
the SPS in case a new OUTPUT queue pixel format is set which is not
compatible with the latest SPS.
- Add a new callback to return a structure filled with major SPS attributes
shared by HEVC/H264 in a human readable format.
(e.g. subsampling=420 instead of chroma_format_idc=1, resolution in pixels
and bitdepth + 8)
The idea is to reduce code duplication when fetching those values and calculating
the actual width & height and to make the code more readable and self-documenting.
- Rework the SPS validation function from H264 to act as the SPS validation
callback, add helpful error messages to the checks
- Rework the way the SPS is stored in the driver and how the valid format is
determined. The general workflow is:
* Set the output format
* Try & Set the SPS control
* Driver figures out a matching CAPTURE queue format for the provided SPS and
stores the SPS in the internal context
* start streaming & prepare buffers & device_run ...
User-space is also allowed to change the SPS control at anytime, but it is
only accepted when no buffers are queued on the CAPTURE queue. Whenever the
driver needs to know the currently valid CAPTURE queue format, then it will
utilize the SPS in the context.
- Remove the new bit writer, as this tool should rather be implemented in a
general manner in a separate patch instead of as a part of this series.
- Ensure that streaming is only started once both queues are running
- Update copyright dates for my changes
The HEVC codec has been tested with the GStreamer V4L2 HEVC plugin.
As general changes were made to the RkVDEC driver I have tested all 3 supported
codecs with GStreamer.
Current Fluster score:
HEVC:
`Ran 134/147 tests successfully in 282.692 secs`
with
`python3 fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -ts JCT-VC-HEVC_V1 -j1`
H264:
`Ran 129/135 tests successfully in 78.950 secs`
with
`python3 fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -ts JVT-AVC_V1 -j1`
VP9:
`Ran 228/303 tests successfully in 327.417 secs`
with
`python3 fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -ts VP9-TEST-VECTORS -j1`
No regressions detected.
failed conformance tests:
- DBLK_D_VIXS_2 (Success on Hantro G2)
- DSLICE_A_HHI_5 (Success on Hantro G2)
- EXT_A_ericsson_4 (Success on Hantro G2)
- PICSIZE_A_Bossen_1 (Hardware limitation)
- PICSIZE_B_Bossen_1 (Hardware limitation)
- PICSIZE_C_Bossen_1 (Hardware limitation)
- PICSIZE_D_Bossen_1 (Hardware limitation)
- PPS_A_qualcomm_7 (Success on Hantro G2)
- SAODBLK_A_MainConcept_4 (Success on Hantro G2)
- SAODBLK_B_MainConcept_4 (Success on Hantro G2)
- TSUNEQBD_A_MAIN10_Technicolor_2 (Success on Hantro G2)
- VPSSPSPPS_A_MainConcept_1 (Success on Hantro G2)
- WPP_D_ericsson_MAIN10_2 (Fail on Hantro G2)
- WPP_D_ericsson_MAIN_2 (Fail on Hantro G2)
Not tested with FFMpeg so far.
Known issues:
- Unable to reliably decode multiple videos concurrently
- The SAODBLK_* tests timeout if the timeout time in fluster is lower than 120
- Currently the uv_virstride is calculated in a manner that is hardcoded
for the two available formats NV12 and NV15. (@config_registers)
Notable design decisions:
- The giant static array of cabac values is moved to a separate c file,
I did so because a separate .h file would be incorrect as it doesn't
expose anything of any value for any other file than the rkvdec-hevc.c
file. Other options were:
- Calculating the values instead of storing the results (No clear pattern
found for the calculation using the static array and the formulas from the
specification)
- Supply them via firmware (Adding firmware makes the whole software
way more complicated and the usage of the driver less obvious)
Ignored Checkpatch warnings:
```
WARNING: line length of 162 exceeds 100 columns
+ { .format = V4L2_PIX_FMT_NV15, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 1, .comp_planes = 2, .bpp = { 5, 5, 0, 0 }, .hdiv = 2, .vdiv = 2,
ERROR: trailing statements should be on next line
+ case V4L2_PIX_FMT_NV15: descr = "10-bit Y/CbCr 4:2:0 (Packed)"; break;
```
v4l2-compliance test:
```
v4l2-compliance 1.23.0-4989, 64 bits, 64-bit time_t
v4l2-compliance SHA: b3527e08fbbc 2023-01-10 11:22:13
Total for rkvdec device /dev/video1: 46, Succeeded: 46, Failed: 0, Warnings: 0
```
To: Mauro Carvalho Chehab <mchehab@xxxxxxxxxx>
To: Ezequiel Garcia <ezequiel@xxxxxxxxxxxxxxxxxxxx>
To: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
Cc: linux-media@xxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx
Cc: linux-rockchip@xxxxxxxxxxxxxxxxxxx
Cc: linux-staging@xxxxxxxxxxxxxxx
Cc: Jonas Karlman <jonas@xxxxxxxxx>
Cc: Alex Bee <knaerzche@xxxxxxxxx>
Cc: Nicolas Dufresne <nicolas.dufresne@xxxxxxxxxxxxx>
Cc: Collabora Kernel-domain <kernel@xxxxxxxxxxxxx>
Cc: Robert Beckett <bob.beckett@xxxxxxxxxxxxx>
Cc: Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx>
Cc: Andrzej Pietrasiewicz <andrzej.p@xxxxxxxxxxxxx>
Cc: Benjamin Gaignard <benjamin.gaignard@xxxxxxxxxxxxx>
---
Jonas Karlman (2):
media: v4l2: Add NV15 pixel format
media: v4l2-common: Add helpers to calculate bytesperline and sizeimage
Sebastian Fricke (10):
staging: media: rkvdec: Helper for buffer queue busy check
staging: media: rkvdec: Block start streaming until both queues run
staging: media: rkvdec: Add SPS structure to internal context
staging: media: rkvdec: Add a valid pixel format check as callback
staging: media: rkvdec: Add a routine to fetch SPS attributes as a callback
staging: media: rkvdec: Add a valid SPS check as a callback
staging: media: rkvdec: h264: Add callbacks for h264
staging: media: rkvdec: Wrapper for pixel format preparation
staging: media: rkvdec: Enable S_CTRL IOCTL
staging: media: rkvdec: Add HEVC backend
.../userspace-api/media/v4l/pixfmt-yuv-planar.rst | 75 +
drivers/media/v4l2-core/v4l2-common.c | 79 +-
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
drivers/staging/media/rkvdec/Makefile | 2 +-
drivers/staging/media/rkvdec/TODO | 11 +-
drivers/staging/media/rkvdec/rkvdec-h264.c | 105 +-
drivers/staging/media/rkvdec/rkvdec-hevc-data.c | 1845 ++++++++++++++++++++
drivers/staging/media/rkvdec/rkvdec-hevc.c | 910 ++++++++++
drivers/staging/media/rkvdec/rkvdec-regs.h | 1 +
drivers/staging/media/rkvdec/rkvdec.c | 300 +++-
drivers/staging/media/rkvdec/rkvdec.h | 14 +
include/uapi/linux/videodev2.h | 1 +
12 files changed, 3244 insertions(+), 100 deletions(-)
---
base-commit: 6599e683db1bf22fee74302c47e31b9a42a1c3d2
change-id: 20230101-patch-series-v2-6-2-rc1-be130fda994d
Best regards,
--
Sebastian Fricke <sebastian.fricke@xxxxxxxxxxxxx>
