On 10/09/2022 18:14, Laurent Pinchart wrote: > On Sat, Sep 10, 2022 at 02:21:10PM +0200, Hans Verkuil wrote: >> On 29/08/2022 09:15, Jacopo Mondi wrote: >>> Hello >>> >>> I have prepared a slide deck to brief on the discussion for >>> the topic specified in the subject. Slides are available at >>> https://nc.nibble.pw/s/oib8jzNjjtgB9c6 as an attachment of ~5MB would >>> probably be caught by the mailing list filter >>> >>> I'll here copy a brief summary as presented in Hans' agenda: >>> >>> ------------------------------------------------------------------------------- >>> 11:45 Jacopo: Representing addition sensor processing stages. >>> >>> How to represent additional processing stages that happens >>> on the sensor side, mostly additional subsampling/cropping that happen >>> between the analogue cropping on the full pixel array and the final >>> image sent on the wire. >>> >>> https://lore.kernel.org/linux-media/CAPY8ntA06L1Xsph79sv9t7MiDSNeSO2vADevuXZdXQdhWpSmow@xxxxxxxxxxxxxx/ >>> >>> Dave made a good introduction of the issue his email which got >>> largely unanswered. >>> >>> The issue is particularly relevant for RAW sensors, where applying >>> subsampling has an impact on the sensor's sensitivity and requires >>> to adjust the gains and exposure accordingly. >>> ------------------------------------------------------------------------------- >>> >>> The slide deck: >>> >>> - Introduces the topic in slides [1-10] >>> >>> - Presents Dave's proposed use case and how it is currently handled in >>> mainline drivers in slides [11-31] using as example the imx274, >>> mt9p031 and CCS drivers. >>> >>> - Proposes a best-effort solution for imx219 in slides [32-37] >>> >>> - The last two slides presents two discussion topics which I will copy >>> here >>> >>> ------------------------------------------------------------------------ >>> Do we need new abstractions ? >>> >>> - An image sensor abstraction built on top of multiple subdevices >>> - Would it really make userspace any simpler ? >>> >>> - Image sensor specific SELECTION_TARGETS >>> - Conflicts with format selection are bound to happen with a single >>> source pad >>> - Can a “special” sink pad that represents the raw pixel array help ? >>> - Does not scale to represent complex sensors with multiple scalers >>> ------------------------------------------------------------------------ >>> >>> The imx219 proposal can be discussed by email or in person, while the >>> questions in the last two slides might be worth being discussed during >>> the meeting. >> >> I've always wondered why binning and skipping wasn't implemented as controls. >> You set them in userspace based on your needs, and they change the formats >> and selection rectangles accordingly. >> >> Trying to model it with selection rectangles is basically impossible. > > I think it was done this way because binning and skipping are a form of > scaling, and scaling is modelled through selection rectangles. However, > scaling is supposed to be controlled by the crop and compose rectangles > on the sink pad, and most camera sensor drivers expose a single subdev, > with a source pad only. We have sensor drivers in mainline that control > binning/skipping using the source pad crop rectangle and the source pad > format, which isn't compliant with the specification. > >> For the discussion it would be good to know what kind of binning and skipping >> schemes there are. Would it be fairly straightforward to translate it to a >> control? Or are there many variations between vendors? > > Controls may be an option, but I don't think they would solve the full > problem. A raw camera sensor typically has > > - A pixel array (obviously) with a (usually) rectangular shape, made of > "active" pixels (known to be good), "margin" pixels (too close to the > edges to be guaranteed to offer good quality) and "optical black" > pixels (under an opaque layer, used to measure the black level). > Depending on the sensor, the "margin" pixels may or may not be > readable. > > - An analog crop rectangle that restricts the pixels being read out, and > this affects timings. Reducing analog crop increases frame rate (but > reduces the field of view). Many sensors allow full control of the > analog crop rectangle, some restrict it to a set of sizes horizontally > (and possibly vertically). This is a hardware restriction, not a > driver restriction. > > - Binning and/or skipping, which allow subsampling by different factors > (commonly 1/2 and 1/4, possibly other power of two, or maybe other > factors). Skipping will just subsample, binning will average. The > exact way this is implemented depends on the colour filter array > pattern (Bayer, quad Bayer, ...). > > - A digital crop rectangle. It is usually freely configuration, and it > doesn't affect timings (conceptually it just turns valid pixels into > blanking pixels by chaning the length of the horizontal and vertical > blanking). > > - A scaler that can scale the image down, more advanced than just > subsampling. This is less common than the above features. If a scaler > is present, there can also be another crop rectangle at the output of > the scaler. > > - The may also be a FIFO in the middle of the pipeline that can re-time > signals, to have an output pixel clock with a different frequency than > the pixel array read clock. > > I'm probably missing something, Sakari may be able to fill the gaps. > > The way we model all this in the V4L2 subdev API is complex, both for > camera sensor drivers and for userspace. I'd like a different model > inside the kernel, I think we need a v4l2_raw_sensor (name to be > bike-shedded) object that driver will implement, with an in-kernel > interface towards driver that will match the above model much more > closely. This would then be translated to subdev ops for userspace in a > shared layer to avoid the boiler plate code and complexity in all > drivers. Whether or not we want a different interface towards userspace > is also a question I'd like to discuss, but I don't want to couple it > with the in-kernel interface discussion. > The various analog/digital crop operations and scaling are all good fits for a rectangle-based API. But not binning/skipping IMHO: those will change the size by specific factors (dividing by some integer value). It's not a good fit for a rectangle-based API IMHO. Controls, however, will fit nicely. Regards, Hans
