Media Summit: V4L2 Future Work October 30, 2019 - Lyon, France Many thanks to the Linux Foundation for hosting this meeting. Much appreciated! Please reply to this report with any comments/corrections. Especially if I missed any action items or attendees! Original announcement: https://lore.kernel.org/linux-media/2ef17f32-5957-7b52-fea3-19913ec1c7b3@xxxxxxxxx/T/ Attendees: Michael Ashton <mpa@xxxxxx> (Facebook/Oculus) Boris Brezillon <boris.brezillon@xxxxxxxxxxxxx> Alexandre Courbot <acourbot@xxxxxxxxxxxx> (Google, Chrome OS) Nicolas Dufresne <nicolas@xxxxxxxxxxxxx> Tomasz Figa <tfiga@xxxxxxxxxxxx> (Google, Chrome OS) Ezequiel Garcia <ezequiel@xxxxxxxxxxxxx> Daniel Gomez <daniel@xxxxxxxx> Peter Griffin <peter.griffin@xxxxxxxxxx> Dafna Hirschfeld <dafna.hirschfeld@xxxxxxxxxxxxx> Eugen Hristev <eugen.hristev@xxxxxxxxxxxxx> Shuah Khan <skhan@xxxxxxxxxxxxxxxxxxx> Helen Koike <helen.koike@xxxxxxxxxxxxx> Jacopo Mondi <jacopo@xxxxxxxxxx> Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx> Niklas Söderlund <niklas.soderlund@xxxxxxxxxxxx> Stanimir Varbanov <stanimir.varbanov@xxxxxxxxxx> Hans Verkuil <hverkuil@xxxxxxxxx> (Cisco Systems Norway) V4L2 Future Work ---------------- This discussion was intentionally not very structured. The goal was to see what the main pain points are and how to fix those. Boris Brezillon made an RFC v3 with proposed new ioctls: https://patchwork.linuxtv.org/cover/59345/ The main blockers for drivers/userspace today with our current API are: - Missing format (DRM) modifier support - one dmabuf fd with per-plane offsets (mplane API today requires separate dmabuf fds for each plane) - userspace cannot specify offsets of planes Note that the DRM modifiers are per frame and not per plane. Typically the modifiers describe tiled and/or compressed formats. Compressed formats are typically lossless HW compression schemes (often proprietary) used to reduce memory bandwidth. Compressed formats are considered opaque formats by userspace. EGL has a dma-buf extension that supports modifiers (see EXT_image_dma_buf_import_modifiers). Also see Wayland protocol: https://gitlab.freedesktop.org/wayland/wayland-protocols/blob/master/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml While not strictly a blocker feature, it is desirable to get rid of the distinction between the single and multiplane APIs and always describe formats as multiplane. The advantage of the multiplane API is that bytesperline and sizeimage are described per plane, which is much easier to understand, and it does not require userspace to calculate bytesperline for the chroma planes. The RFCv3 replaces v4l2_format with a new struct, but the consensus was that it is much better to create a new ioctl just to replace the v4l2_pix_format struct and leave the other structs in the v4l2_format union alone. Two reasons: there is no need to change anything for the other VBI etc. buffer types, it's only v4l2_pix_format that no longer scales; and it is easy to extend a struct v4l2_pix_format_ext in the future. You can't really do that if it is in a union. If a new ENUM_FMT_EXT ioctl is created, then let it return all pixelformats + modifiers combinations rather than having to enumerate over all combinations. This reduces the number of ioctl calls that userspace has to do. To indicate if a format supports supports putting all planes in one buffer, or supports putting each plane in a separate buffer, new format flags are needed. Userspace needs to know this. The vb2 core should check if all planes belong to the same buffer or not, and validate it based on the driver requirements. I.e. in the case of dmabuf import it should check that all dmabuf FDs refer to the same memory and that the per-plane offsets are valid. The current RFCv3 drops the v4l2_timecode field. Nicolas mentioned that while this is not used by any mainline drivers (except virtual drivers) this is in fact used by out-of-tree drivers. While they don't use the same struct, they do return this information since it is used by broadcast systems. So this field needs to return. In the past there was a discussion of creating new fourcc codes to describe pixelformats for both DRM and V4L2. This didn't work out, but it was suggested that we just document the mapping between DRM and V4L2 pixelformats as a reference to end-users. We can maintain that as part of our V4L2 documentation. The suggestion was made to add a channel number to the new format struct and to the streaming I/O ioctls to indicate support for streaming multiple channels. This would allow the M2M framework to support multiple capture and output streams per instance, something that is not possible today. One use-case would be support for H.264 Scalable Video Coding. Another is support for compositors. This would need more research, but the new API should prepare for this. A known outstanding issue is syncing buffers between the CPU and the device. Drivers that want to update the buffer before returning it to userspace do not make the right dmabuf calls. A patch series fixing this was never finished. Google offered to pick this up and give it another try. The general consensus was that these proposed new ioctls only address part of the problems. The complexity of modern camera pipelines creates too many video and v4l-subdev devices, making it very hard for userspace to control all of this. Ideally it should be possible to control a camera ISP through a single /dev/mediaX only, rather than though a myriad of other devices. This is however a huge change, requiring a lot of work. Quite possible a completely new API would be required for this. There is real interest in solving this problem, which includes assigning resources to work on this. Google will collect some requirements and report on that later this year. Based on that we might decide on a three day brainstorm session in Brussels before or after FOSDEM (Feb 1-2, 2020). Action Items ------------ Nicolas Dufresne: - provide more info about timecode use in userspace for Hans Verkuil to verify if struct v4l2_timecode provides sufficient information. Tomasz Figa: - Check if we can drop support for using two memory banks for luma and chroma planes? It's only used by the s5p architecture, and dropping support for that would simplify some of the code. Note that s5p doesn't support dmabuf. - Continue work on my old patch series dealing with dma_buf_begin/end_cpu_access(): https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=vb2-cpu-access This series converts most drivers to use these dmabuf functions. See also this thread: https://patchwork.kernel.org/patch/7159871/ As mentioned in my last reply, the outstanding drivers that are not converted are netup_unidvb, coda, exynos4-is, rcar_jpu and au0828: as far as I remember I did not know how to convert coda and exynos4-is and never found the time to figure that out. The others are new and need to be looked at again. There are probably more drivers that need work since my patch series is from September 2015. - Collect API requirements for a camera ISP API and report on that later this year. Boris Brezillon: - Post an RFCv4 that takes more of these issues into accounts. Research multistream support a bit more to see what would be involved adding support for that.
