Hi Karthik and Jacopo On Thu, 13 Oct 2022 at 08:18, Jacopo Mondi <jacopo@xxxxxxxxxx> wrote: > > Hello Karthik > > On Wed, Oct 12, 2022 at 10:59:50PM -0700, Karthik Poduval wrote: > > Hi All, > > > > I have hardware that does some sort of image manipulation. The > > hardware takes 2 inputs. > > - image buffer > > - config param buffer > > and generates one output which is also an image buffer. > > The input and output images formats fall under standard image > > definitions of V4L2 like various YUV/RGB formats (interleaved or > > multiplanar). > > > > The config param buffer is kind of like a set of instructions for the > > hardware that needs to be passed with every input and output image > > which tells the hardware how to process the image. > > The hardware will be given different input images and output images > > every time and possibly different config param buffers too (in some > > cases). The config param buffers may have variable sizes too based on > > the nature of processing for that frame, but input and output images > > are fixed in size for a given context. I should also mention that the > > config param buffers are a few KBs in size so zero copy is a > > requirement. The config params buffers are written by userspace > > (possibly also driver in kernel space) and read by hardware. > > > > This sounds very much how a regular M2M ISP driver works. I can't tell > about codecs as I'm no expert there, but I expect them to be similar, > so your use case is covered by existing drivers. > > > Here were two mechanisms I had in mind while trying to design a V4L2 > > M2M driver for this hardware. > > - Use a custom multiplanar input format where one plane is a config > > param buffer with remaining planes for input images (in case the input > > image is also multiplanar). > > If you're wondering how to pass parameters to the HW I suggest to > consider registering an output video device node, where you simply > queue buffers with your parameters to. > > Your HW could be modeled as a single subdevice with 3 video device > nodes, one output device for input images, one output device for > parameters, and one capture device for output images. > > +-----------+ > +----+ | HW subdav | +------+ > | In | ---> 0 0 --> | out | > +----+ | | +------+ > +-----0-----+ > ^ > | > +--------+ > | params | > +--------+ The main drawback of this over the codec model of a single video device with both an _OUTPUT and _CAPTURE queue is that you can not run multiple instances simultaneously - there is no way to tie the relevant clients together. I don't know whether supporting simultaneous multiple clients is a requirement in this case, but that may be a key decision in choosing how to represent the device. Dave > The parameters buffer can be of modeled using the v4l2_meta_format[1] > interface. The data format of the buffer could be defined as a custom > metadata format, you can see examples here [2] > > [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-meta.html#c.v4l2_meta_format > [2] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/meta-formats.html#meta-formats > > I suggest to look at the IPU3 and RkISP1 drivers for reference. > > > - Use dmabuf heaps to allocate config param buffer. Tie this config > > param buffer fd to an input buffer (using request API). Driver would > > have to attach the config param buffer dmabuf fd, use it and detach. > > > > You should be able to easily allocate buffers in the video device as > you would easily do and export them as dmabuf fds by using > VIDIOC_EXPBUF [3]. > > Once you have them you can map them in your application code and > write their content. > > [3] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/vidioc-expbuf.html > > > Any comments/concerns about the above two mechanisms ? > > Any other better ideas ? > > Are there any existing V4L2 M2M mechanisms present to deal with per > > frame param buffers that are also zero copy ? > > Is the media request API able to do zero copy for setting compound > > controls for large (several KBs) compound controls ? (making the above > > dmabuf heap approach unnecessary) > > Now, all the above assumes your parameters buffer is modeled as a > structure of parameters (and possibly data tables). If you are instead > looking at something that can be modeled through controls you might > have better guidance by looking at how codecs work, but there I can't > help much ;) > > Hope it helps > j > > > > -- > > Regards, > > Karthik Poduval
