Re: [Media Summit] Using the selection API to control image sensor subsampling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Hans, Laurent,

On Sun, Sep 11, 2022 at 09:16:29AM +0200, Hans Verkuil wrote:
> On 10/09/2022 18:14, Laurent Pinchart wrote:
> > On Sat, Sep 10, 2022 at 02:21:10PM +0200, Hans Verkuil wrote:
> >> On 29/08/2022 09:15, Jacopo Mondi wrote:
> >>> Hello
> >>>
> >>>    I have prepared a slide deck to brief on the discussion for
> >>> the topic specified in the subject. Slides are available at
> >>> https://nc.nibble.pw/s/oib8jzNjjtgB9c6 as an attachment of ~5MB would
> >>> probably be caught by the mailing list filter
> >>>
> >>> I'll here copy a brief summary as presented in Hans' agenda:
> >>>
> >>> -------------------------------------------------------------------------------
> >>> 11:45 Jacopo: Representing addition sensor processing stages.
> >>>
> >>>   How to represent additional processing stages that happens
> >>>   on the sensor side, mostly additional subsampling/cropping that happen
> >>>   between the analogue cropping on the full pixel array and the final
> >>>   image sent on the wire.
> >>>
> >>>   https://lore.kernel.org/linux-media/CAPY8ntA06L1Xsph79sv9t7MiDSNeSO2vADevuXZdXQdhWpSmow@xxxxxxxxxxxxxx/
> >>>
> >>>   Dave made a good introduction of the issue his email which got
> >>>   largely unanswered.
> >>>
> >>>   The issue is particularly relevant for RAW sensors, where applying
> >>>   subsampling has an impact on the sensor's sensitivity and requires
> >>>   to adjust the gains and exposure accordingly.
> >>> -------------------------------------------------------------------------------
> >>>
> >>> The slide deck:
> >>>
> >>> - Introduces the topic in slides [1-10]
> >>>
> >>> - Presents Dave's proposed use case and how it is currently handled in
> >>>   mainline drivers in slides [11-31] using as example the imx274,
> >>>   mt9p031 and CCS drivers.
> >>>
> >>> - Proposes a best-effort solution for imx219 in slides [32-37]
> >>>
> >>> - The last two slides presents two discussion topics which I will copy
> >>>   here
> >>>
> >>>   ------------------------------------------------------------------------
> >>>   Do we need new abstractions ?
> >>>
> >>>   - An image sensor abstraction built on top of multiple subdevices
> >>>     - Would it really make userspace any simpler ?
> >>>
> >>>   - Image sensor specific SELECTION_TARGETS
> >>>     - Conflicts with format selection are bound to happen with a single
> >>>       source pad
> >>>       - Can a “special” sink pad that represents the raw pixel array help ?
> >>>     - Does not scale to represent complex sensors with multiple scalers
> >>>   ------------------------------------------------------------------------
> >>>
> >>> The imx219 proposal can be discussed by email or in person, while the
> >>> questions in the last two slides might be worth being discussed during
> >>> the meeting.
> >>
> >> I've always wondered why binning and skipping wasn't implemented as controls.
> >> You set them in userspace based on your needs, and they change the formats
> >> and selection rectangles accordingly.
> >>
> >> Trying to model it with selection rectangles is basically impossible.
> >
> > I think it was done this way because binning and skipping are a form of
> > scaling, and scaling is modelled through selection rectangles. However,
> > scaling is supposed to be controlled by the crop and compose rectangles
> > on the sink pad, and most camera sensor drivers expose a single subdev,
> > with a source pad only. We have sensor drivers in mainline that control
> > binning/skipping using the source pad crop rectangle and the source pad
> > format, which isn't compliant with the specification.
> >
> >> For the discussion it would be good to know what kind of binning and skipping
> >> schemes there are. Would it be fairly straightforward to translate it to a
> >> control? Or are there many variations between vendors?

In my limited knowledge there might of course be various ways in how
binning/subsampling is performed, but I don't expect hundreds of
different schemes, and not that many differences between vendors ?

I know in example of a sensor that can perform binning with 4
different schemes, all of them selectable by the user. Regardless of
the fancy names the vendor gave them, the underlying schemes
themselves were pretty standard stuff.

Also, this shouldn't be a huge problem. We just add a new control option
as soon as a new scheme is used by a driver (I'm more concerned about
documenting them than implementing them to be honest).

> >
> > Controls may be an option, but I don't think they would solve the full
> > problem. A raw camera sensor typically has
> >
> > - A pixel array (obviously) with a (usually) rectangular shape, made of
> >   "active" pixels (known to be good), "margin" pixels (too close to the
> >   edges to be guaranteed to offer good quality) and "optical black"
> >   pixels (under an opaque layer, used to measure the black level).
> >   Depending on the sensor, the "margin" pixels may or may not be
> >   readable.
> >
> > - An analog crop rectangle that restricts the pixels being read out, and
> >   this affects timings. Reducing analog crop increases frame rate (but
> >   reduces the field of view). Many sensors allow full control of the
> >   analog crop rectangle, some restrict it to a set of sizes horizontally
> >   (and possibly vertically). This is a hardware restriction, not a
> >   driver restriction.
> >
> > - Binning and/or skipping, which allow subsampling by different factors
> >   (commonly 1/2 and 1/4, possibly other power of two, or maybe other
> >   factors). Skipping will just subsample, binning will average. The
> >   exact way this is implemented depends on the colour filter array
> >   pattern (Bayer, quad Bayer, ...).
> >
> > - A digital crop rectangle. It is usually freely configuration, and it
> >   doesn't affect timings (conceptually it just turns valid pixels into
> >   blanking pixels by chaning the length of the horizontal and vertical
> >   blanking).
> >
> > - A scaler that can scale the image down, more advanced than just
> >   subsampling. This is less common than the above features. If a scaler
> >   is present, there can also be another crop rectangle at the output of
> >   the scaler.
> >
> > - The may also be a FIFO in the middle of the pipeline that can re-time
> >   signals, to have an output pixel clock with a different frequency than
> >   the pixel array read clock.
> >
> > I'm probably missing something, Sakari may be able to fill the gaps.
> >
> > The way we model all this in the V4L2 subdev API is complex, both for
> > camera sensor drivers and for userspace. I'd like a different model
> > inside the kernel, I think we need a v4l2_raw_sensor (name to be
> > bike-shedded) object that driver will implement, with an in-kernel
> > interface towards driver that will match the above model much more
> > closely. This would then be translated to subdev ops for userspace in a
> > shared layer to avoid the boiler plate code and complexity in all
> > drivers. Whether or not we want a different interface towards userspace
> > is also a question I'd like to discuss, but I don't want to couple it
> > with the in-kernel interface discussion.
> >
>
> The various analog/digital crop operations and scaling are all good fits
> for a rectangle-based API. But not binning/skipping IMHO: those will
> change the size by specific factors (dividing by some integer value).

I personally don't see that much difference in using a control or a
rectangle to model any additional scaling steps, as both would
anyway require to model the sensor with multiple subdevs as soon as we
get past the single "analog crop + subsampling" case.

A control for binning/skipping can be a neat trick that allows to
represent the additional scaling step (binning/skipping) on subdevs
with a single source pad, something we all agree today is not possible
and wrongly implemented in mainline drivers, but as soon as we have a
digital scaler after the binning stage, or any other point in the
processing pipeline that changes the image resolution, the solution
won't scale.

To be clear: I think having multiple subdevs is the best
option when it comes to drivers, and the worse when it comes to
userspace.

If I put my userspace developer hat on, what I would like
would be to have a discoverable list of (possibly ordered) targets
to apply sizes on through a parameters list.

        {
                .analog_crop = Wa x Ya
                .digital_scaler = Wa / 2 x Ya / 2
                .digital_crop = ...

         }

* We can of course discuss the naming scheme of the targets, how sub-sampling
  has to be specified etc

An helper layer might translate the parameters list to the correct
"subdev[pad]/target" registered by the sensor driver.

But yes, a parameters list, or any other API we might propose, have to
assume a rather precise knowledge of how the sensor has to be
configured to reside in userspace. Unfortunately I don't see any way
around that if we aim to extend the set of configurable features
available to applications and I would like to get consensus on this as
well, as it was listed in one of Dave's pain points.

Thanks

> It's not a good fit for a rectangle-based API IMHO.
>
> Controls, however, will fit nicely.
>
> Regards,
>
> 	Hans



[Index of Archives]     [Linux Input]     [Video for Linux]     [Gstreamer Embedded]     [Mplayer Users]     [Linux USB Devel]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [Yosemite Backpacking]

  Powered by Linux