Hi Laurent On 19 May 2018 at 08:04, Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx> wrote: > Hi Dave, > > On Friday, 18 May 2018 18:37:01 EEST Dave Stevenson wrote: >> On 18 May 2018 at 16:05, Mauro Carvalho Chehab wrote: >> > Em Fri, 18 May 2018 15:27:24 +0300 >> >> <snip> >> >> >>> There, instead of an USB camera, the hardware is equipped with a >> >>> MC-based ISP, connected to its camera. Currently, despite having >> >>> a Kernel driver for it, the camera doesn't work with any >> >>> userspace application. >> >>> >> >>> I'm also aware of other projects that are considering the usage of >> >>> mc-based devices for non-dedicated hardware. >> >> >> >> What are those projects ? >> > >> > Well, cheap ARM-based hardware like RPi3 already has this issue: they >> > have an ISP (or some GPU firmware meant to emulate an ISP). While >> > those hardware could have multiple sensors, typically they have just >> > one. >> >> Slight hijack, but a closely linked issue for the Pi. >> The way I understand the issue of V4L2 / MC on Pi is a more >> fundamental mismatch in architecture. Please correct me if I'm wrong >> here. >> >> The Pi CSI2 receiver peripheral always writes the incoming data to >> SDRAM, and the ISP is then a memory to memory device. >> >> V4L2 subdevices are not dma controllers and therefore have no buffers >> allocated to them. So to support the full complexity of the pipeline >> in V4L2 requires that something somewhere would have to be dequeuing >> the buffers from the CSI receiver V4L2 device and queuing them to the >> input of a (theoretical) ISP M2M V4L2 device, and returning them once >> processed. The application only cares about the output of the ISP M2M >> device. > > Regardless of the software stack architecture, something running on the CPU > has to perform that job. We have decided that that "something" needs to run in > userspace, to avoid pushing use-case-dependent code to the kernel. > > Note that this isn't specific to the RPi. The OMAP3 ISP, while integrating the > CSI-2 receiver and being able to process data on the fly, can also write the > raw images to memory and then process them in memory-to-memory mode. This > feature is used mostly for still image capture to perform pre-processing with > the CPU (or possibly GPU) on the raw images before processing them in the ISP. > There's no way we could implement this fully in the kernel. Sure. I was mainly flagging that having to manage buffers also needs to be considered in order to make a usable system. Just configuring an MC pipeline won't solve all the issues. >> So I guess my question is whether there is a sane mechanism to remove >> that buffer allocation and handling from the app? Without it we are >> pretty much forced to hide bigger blobs of functionality to even >> vaguely fit in with V4L2. > > We need a way to remove that from the application, but it won't be pushed down > to the kernel. These tasks should be handled by a userspace framework, > transparently for the application. The purpose of this discussion is to decide > on the design of the framework. I'm in agreement there, but hadn't seen discussion on buffer management, only MC configuration. >> I'm at the point where it shouldn't be a huge amount of work to create >> at least a basic ISP V4L2 M2M device, but I'm not planning on doing it >> if it pushes the above buffer handling onto the app because it simply >> won't get used beyond demo apps. The likes of Cheese, Scratch, etc, >> just won't do it. >> >> >> To avoid ambiguity, the Pi has a hardware ISP block. There are other >> SoCs that use either GPU code or a DSP to implement their ISP. > > Is that ISP documented publicly ? Not publicly, and as it's Broadcom's IP we can't release it :-( What I have working is using the Broadcom MMAL API (very similar to OpenMAX IL) to wrap the ISP hardware block via the VideoCore firmware. Currently it has the major controls exposed (black level, digital gain, white balance, CCMs, lens shading tables) and I'll add additional controls as time permits or use cases require. Resizing and format conversion are done based on input and output formats. Defining stats regions and extracting the resulting stats is still to be done. Overall it keeps all the implementation details hidden so we don't break NDAs, but should allow efficient processing. It supports dma-bufs in and out, so no extra copies of the data should be required. I'm currently finishing off a V4L2 M2M wrapper around the MMAL video_encode and video_decode components, so modifying it to support the ISP component shouldn't be difficult if there is value in doing so. I know it's not the ideal, but our hands are tied. Dave
