> -----Original Message-----
> From: Harry Wentland <harry.wentland@xxxxxxx>
> Sent: Wednesday, August 30, 2023 1:10 AM
> To: Shankar, Uma <uma.shankar@xxxxxxxxx>; intel-gfx@xxxxxxxxxxxxxxxxxxxxx; dri-
> devel@xxxxxxxxxxxxxxxxxxxxx
> Cc: Borah, Chaitanya Kumar <chaitanya.kumar.borah@xxxxxxxxx>; wayland-
> devel@xxxxxxxxxxxxxxxxxxxxx
> Subject: Re: [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane Color
> Pipeline
>
>
>
> On 2023-08-29 12:03, Uma Shankar wrote:
> > Add the documentation for the new proposed Plane Color Pipeline.
> >
> > Co-developed-by: Chaitanya Kumar Borah
> > <chaitanya.kumar.borah@xxxxxxxxx>
> > Signed-off-by: Chaitanya Kumar Borah <chaitanya.kumar.borah@xxxxxxxxx>
> > Signed-off-by: Uma Shankar <uma.shankar@xxxxxxxxx>
> > ---
> > .../gpu/rfc/plane_color_pipeline.rst | 394 ++++++++++++++++++
> > 1 file changed, 394 insertions(+)
> > create mode 100644 Documentation/gpu/rfc/plane_color_pipeline.rst
> >
> > diff --git a/Documentation/gpu/rfc/plane_color_pipeline.rst
> > b/Documentation/gpu/rfc/plane_color_pipeline.rst
> > new file mode 100644
> > index 000000000000..60ce515b6ea7
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/plane_color_pipeline.rst
> > @@ -0,0 +1,394 @@
> > +=======================================
> > + Plane Color Pipeline: A UAPI proposal
> > +=======================================
> > +
> > +To build the proposal on, lets take the premise of a color pipeline
> > +as shown below.
> > +
> > + +-------------------------------------------+
> > + | RAM |
> > + | +------+ +---------+ +---------+ |
> > + | | FB 1 | | FB 2 | | FB N | |
> > + | +------+ +---------+ +---------+ |
> > + +-------------------------------------------+
> > + | Plane Color Hardware Block |
> > + +--------------------------------------------+
> > + | +---v-----+ +---v-------+ +---v------+ |
> > + | | Plane A | | Plane B | | Plane N | |
> > + | | Pre-CSC | | Pre-CSC | | Pre-CSC | |
> > + | +---+-----+ +---+-------+ +---+------+ |
> > + | | | | |
> > + | +---v-----+ +---v-------+ +---v------+ |
> > + | |Plane A | | Plane B | | Plane N | |
> > + | |CSC/CTM | | CSC/CTM | | CSC/CTM | |
> > + | +---+-----+ +----+------+ +----+-----+ |
> > + | | | | |
> > + | +---v-----+ +----v------+ +----v-----+ |
> > + | | Plane A | | Plane B | | Plane N | |
> > + | |Post-CSC | | Post-CSC | | Post-CSC | |
> > + | +---+-----+ +----+------+ +----+-----+ |
> > + | | | | |
> > + +--------------------------------------------+
> > ++------v--------------v---------------v-------|
> > +|| ||
> > +|| Pipe Blender ||
> > ++--------------------+------------------------+
> > +| | |
> > +| +-----------v----------+ |
> > +| | Pipe Pre-CSC | |
> > +| | | |
> > +| +-----------+----------+ |
> > +| | Pipe Color |
> > +| +-----------v----------+ Hardware |
> > +| | Pipe CSC/CTM | |
> > +| | | |
> > +| +-----------+----------+ |
> > +| | |
> > +| +-----------v----------+ |
> > +| | Pipe Post-CSC | |
> > +| | | |
> > +| +-----------+----------+ |
> > +| | |
> > ++---------------------------------------------+
> > + |
> > + v
> > + Pipe Output
> > +
> > +Each plane consists of the following color blocks
> > + * Pre-CSC : This block can used to linearize the input frame buffer data.
> > + The linear data then can be further acted on by the following
> > + color hardware blocks in the display hardware pipeline
> > +
> > + * CSC/CTM: Used to program color transformation matrix, this block is used
> > + to perform color space conversions like BT2020 to BT709 or BT601
> > + etc. This block acts on the linearized data coming from the
> > + Pre-CSC HW block.
> > +
> > + * Post-CSC: This HW block can be used to non-linearize frame buffer data to
> > + match the sink. Another use case of it could be to perform Tone
> > + mapping for HDR use-cases.
> > +
> > +Data from multiple planes will then be fed to pipe/crtc where it will get blended.
> > +There is a similar set of HW blocks available at pipe/crtc level
> > +which acts on this blended data.
> > +
> > +Below is a sample usecase fo video playback with sub-titles and
> > +playback controls
> > +
> > +┌────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
> > +│FB1 │ │PRE-CSC │ │ CTM Matrix │ │ POST-CSC │
> > +│ ├───►│Linearize ├────►│ BT709 to ├───►│ SDR to HDR │
> > +│BT709 SDR │ │ │ │ BT2020 │ │ Tone Mapping├─────┐
> > +└────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
> │
> > +(subtitles) │
> > + │
> > +┌────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
> │
> > +│FB2 │ │PRE-CSC │ │ CTM Matrix │ │ POST-CSC │ │
> > +│ ├───►│Linearize ├────►│ BT601 to ├───►│ SDR to HDR ├───┐ │
> > +│BT601 SDR │ │ │ │ BT2020 │ │ Tone Mapping│ │ │
> > +└────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
> │ │
> > +(Playback controls UI) │ │
> > + │ │
> > +┌────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
> │ │
> > +│FB2 │ │PRE-CSC │ │ CTM Matrix │ │ POST-CSC │ │ │
> > +│ ├───►│Linearize ├────►│ BT601 to ├───►│ SDR to HDR ├─┐ │ │
> > +│BT2020 HDR │ │ │ │ pass-through│ │ pass-through│ │ │ │
> > +└────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
> │ │
> > +(video frame) │ │ │
> > +
> > +│ │ │
> >
> +┌──────────────────────────────────────────────────────────────────
> ──
> > +────┴─┴─┘
> > +│
> > +│ ┌─────────────┐ ┌─────────────┐ ┌───────────────┐
> > +│ │ CRTC │ │ CRTC │ │ CRTC │
> > +└─┤ PRE-CSC ├─────►│ CSC/CTM ├─────►│ POST-CSC ├─────► TO
> Port
> > + │ │ │ │ │ │
> > + └─────────────┘ └─────────────┘ └───────────────┘
> > +
> > +This RFC is intended to propose an uAPI for the pre-blending color
> > +pipeline (however, can be also extended to post blending pipeline).
> > +
> > +Below are the design considerations while designing the uAPI.
> > +
> > +1. Individual HW blocks can be muxed. (e.g. out of two HW blocks only
> > +one can be used) 2. Position of the HW block can be programmable 3.
> > +LUTs can be one dimentional or three dimentional 4. Number of LUT
> > +entries can vary across platforms 5. Precision of LUT entries can
> > +vary across platforms 6. Distribution of LUT entries may vary. e.g
> > +Mutli-segmented, Logarithmic,
> > + Piece-Wise Linear(PWL) etc
> > +7. There can be parameterized/non-parameterized fixed function HW blocks.
> > + e.g. Just a hardware bit, to convert from one color space to another.
> > +
> > +Plane Color Pipeline: Design Proposal
> > +=====================================
> > +Each Color Hardware block will be represented by the structure drm_color_op.
> > +
> > +struct drm_color_op {
> > + enum color_op_block name;
> > + enum color_op_type type;
> > + u32 blob_id;
> > + u32 private_flags;
> > +};
> > +
> > +The members of which will constitute:
> > +1. name: A standardised enum for the color hardware block 2. type:
> > +The type of mathematical operation done by the hardware block.
> > + e.g. 1D Curve, 3D Curve, Matrix etc.
> > +3. blob id: Id pointing to a blob containing information about the hardware
> > + block advertising the respective capabilities to the userspace.
> > + It can be an optional field depending on the members "name" and "type".
> > +4. private_flags: This can be used to provide vendor specific hints
> > + to user space
> > +
> > +
> > + For example to represent LUTs, we introduce the drm_color_lut_range
> > + structure. It can represent LUTs with varied number of entries and
> > + distributions (Multi segmented, Logarithmic etc).
> > +
> > + struct drm_color_lut_range {
> > + /* DRM_MODE_LUT_* */
> > + __u32 flags;
> > + /* number of points on the curve */
> > + __u16 count;
> > + /* input/output bits per component */
> > + __u8 input_bpc, output_bpc;
> > + /* input start/end values */
> > + __s32 start, end;
> > + /* output min/max values */
> > + __s32 min, max;
> > + };
> > +
> > +Note: More details on exact usage and implementation of this structure can be
> > + found in the RFC. This structure is taken as is from the series.
> > + https://patchwork.freedesktop.org/series/90825/
> > + However, we can add more members to it to encompass all use-cases.
> > + For example. we can add a precision field to advertise the
> > + bitdepth of the LUTs. Similarly, we can reserve some bits in the flag
> > + field for vendor specific use cases.
> > +
> > + At the same time, we don't need to pass any additional information for the
> > + CSC block as userspace and driver already agrees struct drm_color_ctm as
> > + a uAPI to pass on data.
> > +
> > +These color operations will form the building blocks of a color
> > +pipeline which best represents the underlying Hardware. Color
> > +operations can be re-arranged, substracted or added to create
> > +distinct color pipelines to accurately describe the Hardware blocks present in the
> display engine.
> > +
>
> What do you mean when you're talking of re-arranging, or subtracting color
> operations?
I will re-phrase this to avoid confusion. The pipeline can be defined by driver by
Putting together the hardware blocks in some order supported by respective hardware.
If any new generation changes the behaviour, a new pipeline can be defined and exposed
for that particular platform. So pipeline definition is fully flexible but done by driver.
Userspace will just be able to use that and cannot alter the same once exposed by driver.
> > +In this proposal, a color pipeline is represented as an array of
> > +struct drm_color_op.
> > +
> > +struct drm_color_op color_pipeline_1[]
> > +
> > +For example to represent the pre-blending color pipeline as described
> > +above
> > +
> > +We can create a color pipeline like this.
> > +
> > +struct drm_color_op color_pipeline_1[] = {
> > + {
> > + .name = DRM_CB_PRE_CSC,
> > + .type = CURVE_1D,
> > + .blob_id = 0; /* actual value to be populated during plane
> > + initialization*/
> > + },
> > + {
> > + .name = DRM_CB_CSC,
> > + .type = MATRIX,
> > + .blob_id = 0;
> > + },
> > + {
> > + .name = DRM_CB_POST_CSC,
> > + .type = CURVE_1D,
> > + .blob_id = 0;
> > + },
> > +};
> > +
> > +Then, for individual color operation, we add blobs to advertise the
> > +capability of the particular Hardware block. In case of the example
> > +pipeline, we add blobs of type drm_color_lut_range for the "pre-csc" and "post-
> csc".
> > +For the "csc" block we pass no blob id to user space as it is known
> > +to both user space and driver that drm_color_ctm structure is to be
> > +used for such operation.
> > +
> > +To represent, this in a diagram.
> > +
> > + struct drm_color_op color_pipeline_1[]
> > + +---------------------------+
> > + | | drm_color_op
> > + | +---------------------+--+-----------+---------------------+
> > + | | | | | |
> > + | | | | | +-----------------+ |
> > + | | | | | | name | |
> > + | | | | | +-----------------+ |
> > + | | | | | | type | |
> > + | | color_op_1 | | | +-----------------+ |
> > + | | | | | | blob id | +--------+
> > + | | | | | +-----------------+ | |
> > + | | | | | | private | | |
> > + | | | | | | flags | | |
> > + | | | | | +-----------------+ | |
> > + | | | | | | |
> > + | +---------------------+--+-----------+---------------------+ |
> > + | | |
> > + | | |
> > + | +---------------------+ | |
> > + | | | | drm_color_lut_range |
> > + | | color_op_2 | | +-------------------------+ |
> > + | | | | | | |
> > + | | | | | +---------------------+ | |
> > + | +---------------------+ | | | segment 1 { | |<---+
> > + | | | | ... | |
> > + | +---------------------+ | | | .input_bpc = 16, | |
> > + | | | | | | .output_bpc = 16, | |
> > + | | color_op_3 | | | | ... | |
> > + | | | | | | } | |
> > + | | | | | +---------------------+ |
> > + | +---------------------+ | | |
> > + | . | | +---------------------+ |
> > + | . | | | segment 2 { | |
> > + | . | | | ... | |
> > + +---------------------------+ | | } | |
> > + | | | |
> > + | | | |
> > + | | | |
> > + | +---------------------+ |
> > + | . |
> > + | . |
> > + | . |
> > +
> > + +-------------------------+
> > +
> > +
> > +
> > +This color pipeline is then packaged within a blob for the user space
> > +to retrieve it. Details can be found in the next section
> > +
>
> Not sure I like blobs that contain other blob ids.
It provides flexibility and helps with just one interface to userspace. Its easy to handle and
manage once we get the hang of it 😊.
We can clearly define the steps of parsing and data structures to be used while interpreting
and parsing the blobs.
> > +Exposing a color pipeline to user space
> > +=======================================
> > +
> > +To advertise the available color pipelines, an immutable ENUM
> > +property "GET_COLOR_PIPELINE" is introduced.
> > +This enum property has blob id's as values. With each blob id
> > +representing a distinct color pipeline based on underlying HW
> > +capabilities and their respective combinations.
> > +
> > +The following output of drm_info [1], shows how color pipelines are
> > +visible to userspace.
> > +
> > +├───Plane 0
> > + │ ├───Object ID: 31
> > + │ ├───CRTCs: {0}
> > + │ ├───Legacy info
> > + ...
> > + │ ├───"GET_COLOR_PIPELINE" (immutable): enum {no color pipeline,
> > + color pipeline 1, color pipeline 2}=
> no color pipeline
> > +
> > +To understand the capabilities of individual pipelines, first the
> > +userspace need to retrieve the pipeline blob with the blob ids
> > +retrieved by reading the enum property.
> > +
> > +Once the color pipeline is retrieved, user can then parse through
> > +individual drm_color_op blocks to understand the capabilities of each
> > +hardware block.
> > +
> > +Check IGT series to see how user space can parse through color pipelines.
> > +Refer the IGT series here:
> > +https://patchwork.freedesktop.org/series/123018/
> > +
> > +Setting up a color pipeline
> > +===========================
> > +
> > +Once the user space decides on a color pipeline, it can set the
> > +pipeline and the corresponding data for the hardware blocks within
> > +the pipeline with the BLOB property "SET_COLOR_PIPELINE".
> > +
> > +To achieve this two structures are introduced
> > +
> > +1. struct drm_color_op_data: It represents data to be passed onto individual
> > + color hardware blocks. It
> contains three members
> > + a) name: to identify the color operation block
> > + b) blob_id: pointing to the blob with data for the
> > + corresponding hardware block
> > +
> > + struct drm_color_op_data {
> > + enum color_op_block name;
> > + u32 blob_id;
> > + };
> > +
> > +2. struct drm_color_pipeline: This structure represents the aggregate
> > + pipeline to be set. it contains the following members
> > + a) num: pipeline number to be selected
> > + b) size: size of the data to be passed onto
> the driver
> > + c) data: array of struct
> drm_color_op_data with data for
> > + the hardware block/s that userspace wants to
> > + set values for.
> > +
> > + struct drm_color_pipeline {
> > + int num;
> > + int size;
> > + struct drm_color_op_data *data;
> > + };
> > +
> > + User can either:
> > + 1. send data for all the color operations in a pipeline as shown in [2].
> > + The color operation data need not be in order that the pipeline advertises
> > + however, it should not contain data for any
> > + color operation that is not present in the pipeline.
> > +
> > + Note: This check for pipeline is not yet implemented but if the
> > + wider proposal is accepted we have few solutions in mind.
> > +
> > + 2. send data for a subset of color operations as shown in [3].For the
> > + color operation that userspace does not send data will retain their
> > + older state.
> > +
> > + 3. reset/disable the pipeline by setting the "SET_COLOR_PIPELINE" blob
> > + property to NULL as shown in both [2] and [3]
> > +
> > + 4. change the color pipeline as demonstrated in [3].
> > + On the new pipeline, the user is expected to setup all the color hardware
> block
> > + Once the user requests a pipeline change, the driver will provide it a clean
> slate
> > + which means that all the data previously set by the user will be discarded
> even if
> > + there are common hardware blocks between the two(previous and current)
> pipelines.
> > +
> > +IGT implementation can be found here [4]
> > +
> > +Representing Fixed function hardware
> > +====================================
> > +
> > +To provide support for fixed function hardware, the driver could
> > +expose vendor specific struct drm_color_op with parameters that both
> > +the userspace and driver agrees on. To demonstrate, let's consider a
> > +hyphothetical fixed function hardware block that converts BT601 to BT2020.
> > +The driver can choose to advertise the block as such.
> > +
> > +struct drm_color_op color_pipeline_X[] = {
> > + {
> > + .name = DRM_CB_PRIVATE,
> > + .type = FIXED_FUNCTION,
> > + .blob_id = 45;
> > + },
> > +}
> > +
> > +Where the blob represents some vendor specific enums, strings or any
> > +other appropriate data types which both the user-space and drivers are aligned
> on.
> > +
> > +blob:45 {
> > + VENDORXXX_BT602_TO_BT2020,
> > +};
> > +
> > +For enabling or passing parameters to such blocks, the user can send
> > +data to the driver wrapped within a blob like any other color operation block.
> > +
> > + struct drm_color_op_data {
> > + .name = DRM_CB_PRIVATE;
> > + .blob_id = 46;
> > + } ;
> > +
> > +where blob with id 46 contains data to enable the fixed function hardware(s).
> > +
>
> One major thing missing from the RFC is an explanation on why we want to go with a
> prescriptive model, rather than a descriptive model. This question came up in Dave's
> response to Simon's RFC:
> https://lore.kernel.org/dri-
> devel/CAPM=9tz54Jc1HSjdh5A7iG4X8Gvgg46qu7Ezvgnmj4N6gbY+Kw@xxxxxxxxxxxxx
> m/
>
> This is a rough attempt at such an explanation:
> https://gitlab.freedesktop.org/hwentland/linux/-
> /merge_requests/5/diffs?commit_id=bc99737623796b39925767d6f8cbc097ad0b4d
> 8d
Sure Harry, totally agree to this and will include in documentation to highlight the rationale
of going with prescriptive model.
Regards,
Uma Shankar
> Harry
>
> > +References
> > +==========
> > +
> > +[1] https://gitlab.freedesktop.org/emersion/drm_info
> > +[2]
> > +https://patchwork.freedesktop.org/patch/554827/?series=123018&rev=1
> > +[3]
> > +https://patchwork.freedesktop.org/patch/554826/?series=123018&rev=1
> > +[4] https://patchwork.freedesktop.org/series/123018/
