On Wed, Dec 05, 2018 at 06:08:38PM -0800, Randy Dunlap wrote:
> On 12/5/18 2:25 AM, james qian wang (Arm Technology China) wrote:
> > Signed-off-by: James (Qian) Wang <james.qian.wang@xxxxxxx>
> > ---
> > Documentation/gpu/drivers.rst | 1 +
> > Documentation/gpu/komeda-kms.rst | 483 +++++++++++++++++++++++++++++++
> > 2 files changed, 484 insertions(+)
> > create mode 100644 Documentation/gpu/komeda-kms.rst
>
> Hi,
>
> I have some editing changes for you to consider, although I did have a few
> problems with reading the text.
>
Thank you Randy, I'll correct the doc according to your comments in the next version.
>
> > diff --git a/Documentation/gpu/komeda-kms.rst b/Documentation/gpu/komeda-kms.rst
> > new file mode 100644
> > index 000000000000..8af925ca0869
> > --- /dev/null
> > +++ b/Documentation/gpu/komeda-kms.rst
> > @@ -0,0 +1,483 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +==============================
> > + drm/komeda ARM display driver
> > +==============================
> > +
> > +The drm/komeda driver supports for the ARM display processor D71 and later
>
> driver supports the ARM
>
> > +IPs, this document is for giving a brief overview of driver design: how it
>
> IPs. This document gives a brief overview of the driver design:
>
> (although I hate using "IPs" like that.)
How about "Product" or "Chipset"
>
> > +works and why design it like that.
> > +
> > +Overview of D71 like display IPs
> > +================================
> > +
> > +From D71, Arm display IP begins to adopt a flexible and modularized
>
> ARM
Sorry my fault, I used "ARM" in the initial lines which misleads you, but after
Arm reband last year, the "Arm" is the right version.
And I'll unify all the usage to "Arm" in the next version.
>
> > +architecture. A display pipeline is made up of multiple individual and
> > +functional pipeline stages called components, and every component has some
> > +specific capabilities that can give the flowed pipeline pixel data a
> > +particular processing.
> > +
> > +Typical D71 components:
> > +
> > +Layer
> > +-----
> > +Layer is the first pipeline stage, which is for preparing the pixel data
> > +for the next stage. It fetches the pixel from memory, decodes it if it's
> > +AFBC, rotates the source image, unpacks or converts YUV pixels to the device
> > +internal RGB pixels, then adjust the color_space of pixels if need.
>
> adjusts if needed.
>
>
> > +
> > +Scaler
> > +------
> > +As its name, scaler is taking responsability for scaling, and D71 also
>
> As its name suggests, scaler takes (or has) responsibility for
>
> > +supports image enhancements by scaler.
> > +The usage of scaler is very flexible and can be connected to layer output
> > +for layer scaling, or connected to compositor and scale the whole display
> > +frame and then feed the output data into wb_layer which will then write it
> > +into memory.
> > +
> > +Compositor (compiz)
> > +-------------------
> > +Compositor is for blending multiple layers or pixel data flows into one
> > +single display frame, and its output frame can be fed into post image
> > +processor and then on the monitor or fed into wb_layer and written to memory
> > +at the same time. And user also can insert a scaler between compositor and
> > +wb_layer to down scale the display frame first and then writing to memory.
>
> and then write to memory.
>
> > +
> > +Writeback Layer (wb_layer)
> > +--------------------------
> > +Writeback layer do the opposite things of Layer, Which connect to compiz and try
>
> does which
>
> > +to write the composition result to memory.
> > +
> > +Post image processor (improc)
> > +-----------------------------
> > +Post image processor is for adjusting frame data like gamma and color space
> > +to fit the requirements of the monitor.
> > +
> > +Timing controller (timing_ctrlr)
> > +--------------------------------
> > +Final stage of display pipeline, Timing controller is not for the pixel
> > +handling, but only for controlling the display timing.
> > +
> > +Merger
> > +------
> > +D71 scaler mostly only has half the horizontal input/output capabilities compare
>
> compared
>
> > +with Layer, Like if Layer supports 4K input size, the scaler only can supports
>
> like support
>
> > +2K input/output in some time. To achieve the fully frame scaling, D71 introduce
>
> full introduces
>
> > +Layer Split, which split the whole image to two half part and feed them to two
>
> splits parts and feeds
>
> > +Layers A and B, and do the scaling independently, after scaling the result need
>
> does independently. After scaling the result needs
>
> > +to be feed to merger to merge two part image, and then output to compiz.
>
> fed
>
> > +
> > +Splitter
> > +--------
> > +Similar to Layer Split, but Splitter is used for writeback, which split the
>
> splits
>
> > +compiz result to two part and then feed them to two scaler.
>
> parts and then feeds them to two scalers.
>
> > +
> > +Possible D71 Pipeline usage
> > +===========================
> > +
> > +Benefit from the modularized architecture, D71 pipelines can be easily adjusted
>
> Benefitting from
>
> > +to fit different usages, following are some typical pipeline data flow
>
> usages. Following
>
> > +configurations:
> > +
> > +Single pipeline data flow
> > +-------------------------
> > +
> > +.. kernel-render:: DOT
> > + :alt: Single pipeline digraph
> > + :caption: Single pipeline data flow
> > +
> > + digraph single_ppl {
> > + rankdir=LR;
> > +
> > + subgraph {
> > + "Memory";
> > + "Monitor";
> > + }
> > +
> > + subgraph cluster_pipeline {
> > + style=dashed
> > + node [shape=box]
> > + {
> > + node [bgcolor=grey style=dashed]
> > + "Scaler-0";
> > + "Scaler-1";
> > + "Scaler-0/1"
> > + }
> > +
> > + node [bgcolor=grey style=filled]
> > + "Layer-0" -> "Scaler-0"
> > + "Layer-1" -> "Scaler-0"
> > + "Layer-2" -> "Scaler-1"
> > + "Layer-3" -> "Scaler-1"
> > +
> > + "Layer-0" -> "Compiz"
> > + "Layer-1" -> "Compiz"
> > + "Layer-2" -> "Compiz"
> > + "Layer-3" -> "Compiz"
> > + "Scaler-0" -> "Compiz"
> > + "Scaler-1" -> "Compiz"
> > +
> > + "Compiz" -> "Scaler-0/1" -> "Wb_layer"
> > + "Compiz" -> "Improc" -> "Timing Controller"
> > + }
> > +
> > + "Wb_layer" -> "Memory"
> > + "Timing Controller" -> "Monitor"
> > + }
> > +
> > +Dual pipeline with Slave enabled
> > +--------------------------------
> > +
> > +If pipeline_B is free, D71 supports redirect its compositor output to
>
> D71 supports redirecting
>
> > +pipeline_A as an input of compositor of pipeline_A, then pipeline_B doesn't
> > +have its output and work as a slave of pipeline_A.
>
> confusing!
Sorry.
This is for describing the two working mode of D71.
Since D71 has two pipelines, which can work independently and separately.
- Single pipeline mode
Layer_0 -> (scaler) ->\
Layer_1 -> (scaler) ->\ /-> (scaler) -> wb_layer -> memory
compiz ->
Layer_2 -> (scaler) ->/ \-> improc ->timing_ctrlr ->monitor
Layer_3 -> (scaler) ->/
Or work together to drive only one display output.
On this mode, pipeline_B doesn't work indenpendently, but outputs its
composition result into pipeline_A, and its pixel timing also derived from
Pipeline_A.timeing_ctrlr, The pipeline_B works just like a "slave" of
pipeline_A(master)
- Slave enabled data flow
Pipe_B.Layer_0 -> (Pipe_B.scaler) ->\ ---------- Pipeline_A (Slave)
Pipe_B.Layer_1 -> (Pipe_B.scaler) ->\
Pipe_B.compiz -> Pipe_A.compiz
Pipe_B.Layer_2 -> (Pipe_B.scaler) ->/
Pipe_B.Layer_3 -> (Pipe_B.scaler) ->/
Pipe_B.compiz ->\ ------ Pipeline_A (Master)
Layer_4 -> (scaler) ->\
Layer_5 -> (scaler) ->\ /-> (scaler) -> wb_layer -> memory
compiz_A ->
Layer_6 -> (scaler) ->/ \-> improc ->timing_ctrlr ->monitor
Layer_7 -> (scaler) ->/
>
> > +NOTE: Since the compiz component doesn't output alpha value, the slave pipeline
> > +only can be used for bottom layers composition.
> > +
> > +.. kernel-render:: DOT
> > + :alt: Slave pipeline digraph
> > + :caption: Slave pipeline enabled data flow
> > +
> > + digraph slave_ppl {
> > + rankdir=LR;
> > +
> > + subgraph {
> > + "Memory";
> > + "Monitor";
> > + }
> > + node [shape=box]
> > + subgraph cluster_pipeline_slave {
> > + style=dashed
> > + label="Slave Pipeline_B"
> > + node [shape=box]
> > + {
> > + node [bgcolor=grey style=dashed]
> > + "Slave.Scaler-0";
> > + "Slave.Scaler-1";
> > + }
> > +
> > + node [bgcolor=grey style=filled]
> > + "Slave.Layer-0" -> "Slave.Scaler-0"
> > + "Slave.Layer-1" -> "Slave.Scaler-0"
> > + "Slave.Layer-2" -> "Slave.Scaler-1"
> > + "Slave.Layer-3" -> "Slave.Scaler-1"
> > +
> > + "Slave.Layer-0" -> "Slave.Compiz"
> > + "Slave.Layer-1" -> "Slave.Compiz"
> > + "Slave.Layer-2" -> "Slave.Compiz"
> > + "Slave.Layer-3" -> "Slave.Compiz"
> > + "Slave.Scaler-0" -> "Slave.Compiz"
> > + "Slave.Scaler-1" -> "Slave.Compiz"
> > + }
> > +
> > + subgraph cluster_pipeline_master {
> > + style=dashed
> > + label="Master Pipeline_A"
> > + node [shape=box]
> > + {
> > + node [bgcolor=grey style=dashed]
> > + "Scaler-0";
> > + "Scaler-1";
> > + "Scaler-0/1"
> > + }
> > +
> > + node [bgcolor=grey style=filled]
> > + "Layer-0" -> "Scaler-0"
> > + "Layer-1" -> "Scaler-0"
> > + "Layer-2" -> "Scaler-1"
> > + "Layer-3" -> "Scaler-1"
> > +
> > + "Slave.Compiz" -> "Compiz"
> > + "Layer-0" -> "Compiz"
> > + "Layer-1" -> "Compiz"
> > + "Layer-2" -> "Compiz"
> > + "Layer-3" -> "Compiz"
> > + "Scaler-0" -> "Compiz"
> > + "Scaler-1" -> "Compiz"
> > +
> > + "Compiz" -> "Scaler-0/1" -> "Wb_layer"
> > + "Compiz" -> "Improc" -> "Timing Controller"
> > + }
> > +
> > + "Wb_layer" -> "Memory"
> > + "Timing Controller" -> "Monitor"
> > + }
> > +
> > +Sub-pipelines for input and output
> > +----------------------------------
> > +
> > +A complete display pipeline can be easily dividied into three sub-pipelines
>
> divided
>
> > +according to the in/out usage.
> > +
> > +Layer(input) pipeline
> > +~~~~~~~~~~~~~~~~~~~~~
> > +
> > +.. kernel-render:: DOT
> > + :alt: Layer data digraph
> > + :caption: Layer (input) data flow
> > +
> > + digraph layer_data_flow {
> > + rankdir=LR;
> > + node [shape=box]
> > +
> > + {
> > + node [bgcolor=grey style=dashed]
> > + "Scaler-n";
> > + }
> > +
> > + "Layer-n" -> "Scaler-n" -> "Compiz"
> > + }
> > +
> > +.. kernel-render:: DOT
> > + :alt: Layer Split digraph
> > + :caption: Layer Split pipeline
> > +
> > + digraph layer_data_flow {
> > + rankdir=LR;
> > + node [shape=box]
> > +
> > + "Layer-0/1" -> "Scaler-0" -> "Merger"
> > + "Layer-2/3" -> "Scaler-1" -> "Merger"
> > + "Merger" -> "Compiz"
> > + }
> > +
> > +Writeback(output) pipeline
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +.. kernel-render:: DOT
> > + :alt: writeback digraph
> > + :caption: Writeback(output) data flow
> > +
> > + digraph writeback_data_flow {
> > + rankdir=LR;
> > + node [shape=box]
> > +
> > + {
> > + node [bgcolor=grey style=dashed]
> > + "Scaler-n";
> > + }
> > +
> > + "Compiz" -> "Scaler-n" -> "Wb_layer"
> > + }
> > +
> > +.. kernel-render:: DOT
> > + :alt: split writeback digraph
> > + :caption: Writeback(output) Split data flow
> > +
> > + digraph writeback_data_flow {
> > + rankdir=LR;
> > + node [shape=box]
> > +
> > + "Compiz" -> "Splitter"
> > + "Splitter" -> "Scaler-0" -> "Merger"
> > + "Splitter" -> "Scaler-1" -> "Merger"
> > + "Merger" -> "Wb_layer"
> > + }
> > +
> > +Display output pipeline
> > +~~~~~~~~~~~~~~~~~~~~~~~
> > +.. kernel-render:: DOT
> > + :alt: display digraph
> > + :caption: display output data flow
> > +
> > + digraph single_ppl {
> > + rankdir=LR;
> > + node [shape=box]
> > +
> > + "Compiz" -> "Improc" -> "Timing Controller"
> > + }
> > +
> > +In the following section we'll see these three sub-pipelines will be handled
> > +by KMS-plane/wb_conn/crtc respectively.
> > +
> > +Komeda Resource abstraction
> > +===========================
> > +
> > +struct komeda_pipeline/component
> > +--------------------------------
> > +
> > +To fully utilize and easily access/configure the HW, driver side also use a
>
> the driver side also uses a
>
> > +similar architecture: Pipeline/Component to describe the HW features and
> > +capabilities. and a specific component includes two part:
>
> capabilities, parts:
>
> > +
> > +- Data flow controlling.
> > +- Specific component capabilities and features.
> > +
> > +So driver defines common header struct komeda_component to describe the data
>
> So the driver defines a common header
>
> > +flow controlling. and all specific component are subclass of this base structure.
>
> flow control and all specific components are a subclass of
>
> > +
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_pipeline.h
> > + :internal:
> > +
> > +Resource discovery and initialization
> > +=====================================
> > +
> > +Pipeline and component are used to describe how to handle the pixel data.
> > +We still need a @struct komeda_dev to describe the whole view of the device, and
> > +the control-abilites of device.
> > +
> > +Now we have &komeda_dev, &komeda_pipeline, &komeda_component, but how to fill dev
>
> Maybe:
> We have &komeda_dev, &komeda_pipeline, and &komeda_component. Now to fill the device
>
> > +with pipelines. Since komeda is not only for D71 but also intended for later IPs,
> > +of course we’d better share as much as possible between different IPs, to
>
> different IPs. To
>
> > +achieve it split komeda device into two layers: CORE and CHIP.
>
> achieve this, split the
>
> > +
> > +- CORE: for common features and capabilities handling.
> > +- CHIP: for register program and HW specific feature (limitition) handling.
>
> (limitation)
>
> > +
> > +CORE access to CHIP by three chip func:
>
> CORE can access CHIP by three chip function structures:
>
> > +
> > +- struct komeda_dev_funcs
> > +- struct komeda_pipeline_funcs
> > +- struct komeda_component_funcs
> > +
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_dev.h
> > + :internal:
> > +
> > +Format handling
> > +===============
> > +
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_format_caps.h
> > + :internal:
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_framebuffer.h
> > + :internal:
> > +
> > +Attach komeda_dev to DRM-KMS
> > +============================
> > +
> > +Komeda abstracts resources by pipeline/component, but DRM-KMS uses
> > +crtc/plane/connector. one KMS-obj can not represent only one single component,
>
> One cannot [preferably]
>
> > +since the requirements of a single KMS object cannot simply be achieved by a
> > +single component, usually that needs multiple components to fit the requirement.
> > +Like set mode, gamma, ctm for KMS all target on CRTC-obj, but komeda needs
> > +compiz, improc and timing_ctrlr to work together to fit these requirements.
> > +And a KMS-Plane may requires multiple komeda resources: layer/scaler/compiz.
>
> require
>
> > +
> > +So, One KMS-Obj represents a sub-pipeline of komeda resources.
>
> one
>
> > +
> > +- Plane: `Layer(input) pipeline`_
> > +- Wb_connector: `Writeback(output) pipeline`_
> > +- Crtc: `Display output pipeline`_
> > +
> > +So, for komeda, we treat KMS crtc/plane/connector as users of pipeline and
> > +component, and one time a pipeline/component only can be used by one user.
>
> and at any one time
>
> > +And pipeline/component will be treated as private object of DRM-KMS, the state
>
> of DRM-KMS; the state
>
> > +will be managed drm_atomic_state as well.
>
> will be managed by drm_atomic_state as well.
>
> > +
> > +How to map plane to Layer(input) pipeline
> > +-----------------------------------------
> > +
> > +Komeda has mutiple Layer input pipelines, see:
> > +- `Single pipeline data flow`_
> > +- `Dual pipeline with Slave enabled`_
> > +
> > +the easist way is binding a plane to a fixed Layer pipeline. but consider the
>
> The easiest pipeline,
>
>
> > +komeda capabilities:
> > +
> > +- Layer Split, See `Layer(input) pipeline`_
> > +
> > + Which actually handles a image by two Layer pipelines. if one plane only
>
> If
>
> > + represents a fixed Layer, then the Layer_Split need to be handled in user
>
> needs
>
> > + mode, for configure the pipeline as Layer_Split, user need to so much
>
> mode: to configure the pipeline as Layer_Split, the user needs much
>
> > + details of HW, and pass down lots of private properties.
>
> detail of the HW, and must pass down lots of private properties.
>
> > + But if handle it in kernel the logic will be smooth and simple.
>
> But if handled in the kernel, the logic will be smooth and simple.
>
> So above, I tried to write what I thought you meant, but saying that Layer_Split
> needs to be handled in user mode ... and then saying that if it is handled in the
> kernel, it would be smooth and simple -- that seems to be contradictory to me,
> so I guess I didn't understand it very well.
>
Sorry, I need to reorganize my sentence. maybe like below:
Layer_Split is quite complicated feature, which splits a big image into
two parts and handle it by two layers and scalers individually.
But it imports a edge problem or effect in the middle of image after the split.
To avoid such problem, need a complicated Split calculation and some special
configurationes to the layer and scaler.
We'd better hide such HW related complexity to user mode.
> > +
> > +- Slave pipeline, See `Dual pipeline with Slave enabled`_
> > +
> > + Since the compiz component doesn't output alpha value, the slave pipeline
> > + only can be used for bottom layers composition, komeda driver wants to hide
>
> composition. The komeda driver wants to hide
>
> > + this limitation to user, the way is pickup suitable Layer according to
>
> to the user. The way to do this is to pick a suitable Layer
> according to Plane->zpos.
>
> > + Plane->zpos.
> > +
> > + Slave pipeline capabilities of komeda also means the plane in komeda is CRTC
> > + shareable. assign real Layer to plane in kernal according to the real HW
>
> Assign kernel
>
> > + usage and state can also easy the user driver for the sharable plane.
>
> ??confused!! shareable
>
> > +
> > +So for komeda, the KMS-plane doesn't represent a fixed komeda layer pipeline,
> > +but mutiple Layers with same capabilities, komeda will select one or more Layers
>
> multiple capabilities.
> > +to fit the requirement of one KMS-plane.
> > +
> > +Make component/pipeline to be drm_private_obj
> > +---------------------------------------------
> > +
> > +Add :c:type:`drm_private_obj` to :c:type:`komeda_component`, :c:type:`komeda_pipeline`
> > +
> > +.. code-block:: c
> > +
> > + struct komeda_component {
> > + struct drm_private_obj obj;
> > + …
>
> Prefer ASCII "...".
>
> > + }
> > +
> > + struct komeda_pipeline {
> > + struct drm_private_obj obj;
> > + …
>
> same.
>
> > + }
> > +
> > +Tracking component_state/pipeline_state by drm_atomic_state
> > +-----------------------------------------------------------
> > +
> > +Add :c:type:`drm_private_state` and user to :c:type:`komeda_component_state`,
> > +:c:type:`komeda_pipeline_state`
> > +
> > +.. code-block:: c
> > +
> > + struct komeda_component_state {
> > + struct drm_private_state obj;
> > + void *binding_user;
> > + …
>
> same.
>
> > + }
> > +
> > + struct komeda_pipeline_state {
> > + struct drm_private_state obj;
> > + struct drm_crtc *crtc;
> > + ...
>
> like that :)
>
> > + }
> > +
> > +komeda component validation
> > +---------------------------
> > +
> > +Komeda has multiple types of components, but the process of validation are
>
> is
>
> > +similar, usually include following steps:
>
> including the following steps:
>
> > +
> > +.. code-block:: c
> > +
> > + int komeda_xxxx_validate(struct komeda_component_xxx xxx_comp,
> > + struct komeda_component_output *input_dflow,
> > + struct drm_plane/crtc/connector *user,
> > + struct drm_plane/crtc/connector_state, *user_state)
> > + {
> > + setup 1: check if component is needed, like the scaler is optional depend
>
> depending
>
> > + on the user_state, if unneeded, just return, the caller will put
>
> user_state; if unneeded, just return, and the caller will put
>
> > + the data flow into next stage.
> > + Setup 2: check user_state with component features and capabilities to see
> > + if can meet, if not return fail.
>
> if requirements can be met; if not, return fail.
>
> > + Setup 3: get component_state from drm_atomic_state, and try set user to
>
> try to set
>
> > + component, fail if component has been assigned to another user already.
>
> component;
>
> > + Setup 3: configure the component_state, like set its input component,
> > + convert user_state to component specific state.
> > + Setup 4: adjust the input_dflow and prepare it for the next stage.
> > + }
> > +
> > +komeda_kms Abstraction
> > +----------------------
> > +
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_kms.h
> > + :internal:
> > +
> > +komde_kms Functions
> > +-------------------
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_crtc.c
> > + :internal:
> > +.. kernel-doc:: drivers/gpu/drm/arm/display/komeda/komeda_plane.c
> > + :internal:
> > +
> > +Build komeda to be a linux module driver
>
> Linux
>
> > +========================================
> > +
> > +Now we have two devices:
> > +
> > +- komeda_dev is for describing the real display hardware.
> > +- komeda_kms_dev is for attaching or connecting komeda_dev to DRM-KMS.
> > +
> > +all komeda operations are supplied or operated by komeda_dev or komeda_kms_dev,
>
> All
>
> > +the module driver is only a simple wrapper to pass the linux command
>
> Linux
>
> > +(probe/remove/pm) into komeda_dev or komeda_kms_dev.
> >
>
> HTH.
> --
> ~Randy
