On Tue, Oct 22, 2024 at 10:33:53PM -0700, anish kumar wrote:
> Added following changes:
> 1. Components provided by remoteproc framework.
> 2. Remoteproc driver responsibilities.
> 3. Remoteproc framework responsibilities.
> 4. Better explanation of how to ask for resources
> from the framework by the remote processor.
>
> Signed-off-by: anish kumar <yesanishhere@xxxxxxxxx>
> ---
> .../driver-api/remoteproc/remoteproc.rst | 596 +++++++-----------
> 1 file changed, 243 insertions(+), 353 deletions(-)
>
> diff --git a/Documentation/driver-api/remoteproc/remoteproc.rst b/Documentation/driver-api/remoteproc/remoteproc.rst
> index 9cccd3dd6a4b..7ca545eea153 100644
> --- a/Documentation/driver-api/remoteproc/remoteproc.rst
> +++ b/Documentation/driver-api/remoteproc/remoteproc.rst
> @@ -1,359 +1,249 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> ==========================
> Remote Processor Framework
> ==========================
>
> -Introduction
> -============
> -
> -Modern SoCs typically have heterogeneous remote processor devices in asymmetric
> -multiprocessing (AMP) configurations, which may be running different instances
> -of operating system, whether it's Linux or any other flavor of real-time OS.
> -
> -OMAP4, for example, has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP.
> -In a typical configuration, the dual cortex-A9 is running Linux in a SMP
> -configuration, and each of the other three cores (two M3 cores and a DSP)
> -is running its own instance of RTOS in an AMP configuration.
> -
> -The remoteproc framework allows different platforms/architectures to
> -control (power on, load firmware, power off) those remote processors while
> -abstracting the hardware differences, so the entire driver doesn't need to be
> -duplicated. In addition, this framework also adds rpmsg virtio devices
> -for remote processors that supports this kind of communication. This way,
> -platform-specific remoteproc drivers only need to provide a few low-level
> -handlers, and then all rpmsg drivers will then just work
> -(for more information about the virtio-based rpmsg bus and its drivers,
> -please read Documentation/staging/rpmsg.rst).
> -Registration of other types of virtio devices is now also possible. Firmwares
> -just need to publish what kind of virtio devices do they support, and then
> -remoteproc will add those devices. This makes it possible to reuse the
> -existing virtio drivers with remote processor backends at a minimal development
> -cost.
> -
> -User API
> -========
> -
> -::
> -
> - int rproc_boot(struct rproc *rproc)
> -
> -Boot a remote processor (i.e. load its firmware, power it on, ...).
> -
> -If the remote processor is already powered on, this function immediately
> -returns (successfully).
> -
> -Returns 0 on success, and an appropriate error value otherwise.
> -Note: to use this function you should already have a valid rproc
> -handle. There are several ways to achieve that cleanly (devres, pdata,
> -the way remoteproc_rpmsg.c does this, or, if this becomes prevalent, we
> -might also consider using dev_archdata for this).
> -
> -::
> -
> - void rproc_shutdown(struct rproc *rproc)
> -
> -Power off a remote processor (previously booted with rproc_boot()).
> -In case @rproc is still being used by an additional user(s), then
> -this function will just decrement the power refcount and exit,
> -without really powering off the device.
> -
> -Every call to rproc_boot() must (eventually) be accompanied by a call
> -to rproc_shutdown(). Calling rproc_shutdown() redundantly is a bug.
> -
> -.. note::
> -
> - we're not decrementing the rproc's refcount, only the power refcount.
> - which means that the @rproc handle stays valid even after
> - rproc_shutdown() returns, and users can still use it with a subsequent
> - rproc_boot(), if needed.
> -
> -::
> -
> - struct rproc *rproc_get_by_phandle(phandle phandle)
> -
> -Find an rproc handle using a device tree phandle. Returns the rproc
> -handle on success, and NULL on failure. This function increments
> -the remote processor's refcount, so always use rproc_put() to
> -decrement it back once rproc isn't needed anymore.
> -
> -Typical usage
> -=============
> -
> -::
> -
> - #include <linux/remoteproc.h>
> -
> - /* in case we were given a valid 'rproc' handle */
> - int dummy_rproc_example(struct rproc *my_rproc)
> - {
> - int ret;
> -
> - /* let's power on and boot our remote processor */
> - ret = rproc_boot(my_rproc);
> - if (ret) {
> - /*
> - * something went wrong. handle it and leave.
> - */
> - }
> -
> - /*
> - * our remote processor is now powered on... give it some work
> - */
> -
> - /* let's shut it down now */
> - rproc_shutdown(my_rproc);
> - }
> -
> -API for implementors
> -====================
> -
> -::
> -
> - struct rproc *rproc_alloc(struct device *dev, const char *name,
> - const struct rproc_ops *ops,
> - const char *firmware, int len)
> -
> -Allocate a new remote processor handle, but don't register
> -it yet. Required parameters are the underlying device, the
> -name of this remote processor, platform-specific ops handlers,
> -the name of the firmware to boot this rproc with, and the
> -length of private data needed by the allocating rproc driver (in bytes).
> -
> -This function should be used by rproc implementations during
> -initialization of the remote processor.
> -
> -After creating an rproc handle using this function, and when ready,
> -implementations should then call rproc_add() to complete
> -the registration of the remote processor.
> -
> -On success, the new rproc is returned, and on failure, NULL.
> -
> -.. note::
> -
> - **never** directly deallocate @rproc, even if it was not registered
> - yet. Instead, when you need to unroll rproc_alloc(), use rproc_free().
> -
> -::
> -
> - void rproc_free(struct rproc *rproc)
> -
> -Free an rproc handle that was allocated by rproc_alloc.
> -
> -This function essentially unrolls rproc_alloc(), by decrementing the
> -rproc's refcount. It doesn't directly free rproc; that would happen
> -only if there are no other references to rproc and its refcount now
> -dropped to zero.
> -
> -::
> -
> - int rproc_add(struct rproc *rproc)
> -
> -Register @rproc with the remoteproc framework, after it has been
> -allocated with rproc_alloc().
> -
> -This is called by the platform-specific rproc implementation, whenever
> -a new remote processor device is probed.
> -
> -Returns 0 on success and an appropriate error code otherwise.
> -Note: this function initiates an asynchronous firmware loading
> -context, which will look for virtio devices supported by the rproc's
> -firmware.
> -
> -If found, those virtio devices will be created and added, so as a result
> -of registering this remote processor, additional virtio drivers might get
> -probed.
> -
> -::
> -
> - int rproc_del(struct rproc *rproc)
> -
> -Unroll rproc_add().
> -
> -This function should be called when the platform specific rproc
> -implementation decides to remove the rproc device. it should
> -_only_ be called if a previous invocation of rproc_add()
> -has completed successfully.
> -
> -After rproc_del() returns, @rproc is still valid, and its
> -last refcount should be decremented by calling rproc_free().
> -
> -Returns 0 on success and -EINVAL if @rproc isn't valid.
> -
> -::
> -
> - void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type)
> -
> -Report a crash in a remoteproc
> -
> -This function must be called every time a crash is detected by the
> -platform specific rproc implementation. This should not be called from a
> -non-remoteproc driver. This function can be called from atomic/interrupt
> -context.
> -
> -Implementation callbacks
> +.. Contents:
> +
> + 1. Introduction
> + 2. Remoteproc framework responsibilities
> + 3. Remoteproc driver responsibilities
> + 4. Virtio and rpmsg
> +
> +1. Introduction
> +===============
> +
> +Modern System on Chips (SoCs) typically integrate heterogeneous remote
> +processor devices in asymmetric multiprocessing (AMP) configurations.
> +These processors may run different operating systems, such as Linux and
> +various real-time operating systems (RTOS).
You are moving things around _and_ making modifications to the text in the same
patch, something I specifically asked not to do. Moreover, the above conveys
exactly the same information as found in [1] but using different words. I
am in favour of enhancing documentation but not creating unneeded churn.
I found several instances of the same rewording pattern in the sections below.
As such I will not look at the other patches nor move forward with this set.
Thanks,
Mathieu
[1]. https://elixir.bootlin.com/linux/v6.12-rc4/source/Documentation/staging/remoteproc.rst
> +
> +For example, the OMAP4 platform features dual Cortex-A9 cores, dual
> +Cortex-M3 cores, and a C64x+ DSP. In a standard setup, the Cortex-A9
> +cores execute Linux in a symmetric multiprocessing (SMP) configuration,
> +while the M3 cores and DSP run independent instances of an RTOS.
> +
> +The remoteproc framework allows various platforms and architectures to
> +manage remote processors, including operations such as powering on,
> +loading firmware, and powering off. This framework abstracts hardware
> +differences, promoting code reuse and minimizing duplication. It also
> +supports rpmsg virtio devices for remote processors that utilize this
> +communication method. Consequently, platform-specific remoteproc drivers
> +need only implement a few low-level handlers, enabling seamless operation
> +of all rpmsg drivers. (For more details about the virtio-based rpmsg
> +bus and its drivers, refer to rpmsg documentation.)
> +
> +Additionally, the framework allows for the registration of various
> +virtio devices. Firmware can publish the types of virtio devices it
> +supports, facilitating their addition to the remoteproc framework. This
> +flexibility enables the reuse of existing virtio drivers with remote
> +processor backends at minimal development cost.
> +
> +The primary purpose of the remoteproc framework is to download firmware
> +for remote processors and manage their lifecycle. The framework consists
> +of several key components:
> +
> +- **Character Driver**: Provides userspace access to control the remote
> + processor.
> +- **ELF Utility**: Offers functions for handling ELF files and managing
> + resources requested by the remote processor.
> +- **Remoteproc Core**: Manages firmware downloads and recovery actions
> + in case of a remote processor crash.
> +- **Coredump**: Provides facilities for coredumping and tracing from
> + the remote processor in the event of a crash.
> +- **Userspace Interaction**: Uses sysfs and debugfs to manage the
> + lifecycle and status of the remote processor.
> +- **Virtio Support**: Facilitates interaction with the virtio and
> + rpmsg bus.
> +
> +From here on, references to "framework" denote the remoteproc
> +framework, and "driver" refers to the remoteproc driver that utilizes
> +the framework for managing remote processors.
> +
> +2. Remoteproc framework Responsibilities
> +========================================
> +
> +The framework begins by gathering information about the firmware file
> +to be downloaded through the request_firmware function. It supports
> +the ELF format and parses the firmware image to identify the physical
> +addresses that need to be populated from the corresponding ELF sections.
> +The framework also requires knowledge of the logical or I/O-mapped
> +addresses in the application processor. Once this information is
> +obtained from the driver, the framework transfers the data to the
> +specified addresses and starts the remote, along with
> +any devices physically or logically connected to it.
> +
> +Dependent devices, referred to as `subdevices` within the framework,
> +are also managed post-registration by their respective drivers.
> +Subdevices can register themselves using `rproc_(add/remove)_subdev`.
> +Non-remoteproc drivers can use subdevices as a way to logically connect
> +to remote and get lifecycle notifications of the remote.
> +
> +The framework oversees the lifecycle of the remote and
> +provides the `rproc_report_crash` function, which the driver invokes
> +upon receiving a crash notification from the remote. The
> +notification method can differ based on the design of the remote
> +processor and its communication with the application processor. For
> +instance, if the remote is a DSP equipped with a watchdog,
> +unresponsive behavior triggers the watchdog, generating an interrupt
> +that routes to the application processor, allowing it to call
> +`rproc_report_crash` in the driver's interrupt context.
> +
> +During crash handling, the framework performs the following actions:
> +
> +a. Sends a request to stop the remote and any connected or
> + dependent subdevices.
> +b. Generates a coredump, dumping all `resources` requested by the
> + remote alongside relevant debugging information. Resources are
> + explained below.
> +c. Reloads the firmware and restarts the remote.
> +
> +If the `RPROC_FEAT_ATTACH_ON_RECOVERY` flag is set, the detach and
> +attach callbacks of the driver are invoked without reloading the
> +firmware. This is useful when the remote requires no
> +assistance for recovery, or when the application processor can restart
> +independently. After recovery, the application processor can reattach
> +to the remote.
> +
> +The remote can request resources from the framework, which
> +allocates a ".resource_table" section. During the ELF parsing phase,
> +the framework identifies this section and calls the appropriate
> +handler to allocate the requested resources.
> +
> +Resource management within the framework can accommodate any type of
> +`fw_resource_type`.
> +
> +.. code-block:: c
> +
> + enum fw_resource_type {
> + RSC_CARVEOUT = 0,
> + RSC_DEVMEM = 1,
> + RSC_TRACE = 2,
> + RSC_VDEV = 3,
> + RSC_LAST = 4,
> + RSC_VENDOR_START = 128,
> + RSC_VENDOR_END = 512,
> + };
> +
> + struct resource_table {
> + u32 ver;
> + u32 num;
> + u32 reserved[2];
> + u32 offset[];
> + } __packed;
> +
> + struct fw_rsc_hdr {
> + u32 type;
> + u8 data[];
> + } __packed;
> +
> +For example, if the remote requests both `RSC_TRACE` and
> +`RSC_CARVEOUT` for memory allocation, the ELF firmware can be structured
> +as follows:
> +
> +.. code-block:: c
> +
> + #define MAX_SHARED_RESOURCE 2
> + #define LOG_BUF_SIZE 1000
> + #define CARVEOUT_DUMP_PA 0x12345678
> + #define CARVEOUT_DUMP_SIZE 2000
> +
> + struct shared_resource_table {
> + u32 ver;
> + u32 num;
> + u32 reserved[2];
> + u32 offset[MAX_SHARED_RESOURCE];
> + struct fw_rsc_trace log_trace;
> + struct fw_rsc_carveout dump_carveout;
> + };
> +
> + volatile struct shared_resource_table table = {
> + .ver = 1,
> + .num = 2,
> + .reserved = {0, 0},
> + .offset = {
> + offsetof(struct resource_table, log_trace),
> + offsetof(struct resource_table, dump_carveout),
> + },
> + .log_trace = {
> + RSC_TRACE,
> + (u32)log_buf, LOG_BUF_SIZE, 0, "log_trace",
> + },
> + .dump_carveout = {
> + RSC_CARVEOUT,
> + (u32)FW_RSC_ADDR_ANY, CARVEOUT_PA, 0, "carveout_dump",
> + },
> + };
> +
> +The framework creates a sysfs file when it encounters the `RSC_TRACE`
> +type to expose log information to userspace. Other resource types are
> +handled accordingly. In the example above, `CARVEOUT_DUMP_SIZE` bytes
> +of DMA memory will be allocated starting from `CARVEOUT_DUMP_PA`.
> +
> +
> +3. Remoteproc driver responsibilities
> +=====================================
> +
> +The driver must provide the following information to the core:
> +
> +a. Translate device addresses (physical addresses) found in the ELF
> + firmware to virtual addresses in Linux using the `da_to_va`
> + callback. This allows the framework to copy ELF firmware from the
> + filesystem to the addresses expected by the remote since
> + the framework cannot directly access those physical addresses.
> +b. Prepare/unprepare the remote prior to firmware loading,
> + which may involve allocating carveout and reserved memory regions.
> +c. Implement methods for starting and stopping the remote,
> + whether by setting registers or sending explicit interrupts,
> + depending on the hardware design.
> +d. Provide attach and detach callbacks to start the remote
> + without loading the firmware. This is beneficial when the remote
> + processor is already loaded and running.
> +e. Implement a load callback for firmware loading, typically using
> + the ELF loader provided by the framework; currently, only ELF
> + format is supported.
> +f. Invoke the framework's crash handler API upon detecting a remote
> + crash.
> +
> +Drivers must fill the `rproc_ops` structure and call `rproc_alloc`
> +to register themselves with the framework.
> +
> +.. code-block:: c
> +
> + struct rproc_ops {
> + int (*prepare)(struct rproc *rproc);
> + int (*unprepare)(struct rproc *rproc);
> + int (*start)(struct rproc *rproc);
> + int (*stop)(struct rproc *rproc);
> + int (*attach)(struct rproc *rproc);
> + int (*detach)(struct rproc *rproc);
> + void * (*da_to_va)(struct rproc *rproc, u64 da, size_t len,
> + bool *is_iomem);
> + int (*parse_fw)(struct rproc *rproc, const struct firmware *fw);
> + int (*handle_rsc)(struct rproc *rproc, u32 rsc_type,
> + void *rsc, int offset, int avail);
> + int (*load)(struct rproc *rproc, const struct firmware *fw);
> + //snip
> + };
> +
> +
> +4. Virtio and Remoteproc
> ========================
>
> -These callbacks should be provided by platform-specific remoteproc
> -drivers::
> -
> - /**
> - * struct rproc_ops - platform-specific device handlers
> - * @start: power on the device and boot it
> - * @stop: power off the device
> - * @kick: kick a virtqueue (virtqueue id given as a parameter)
> - */
> - struct rproc_ops {
> - int (*start)(struct rproc *rproc);
> - int (*stop)(struct rproc *rproc);
> - void (*kick)(struct rproc *rproc, int vqid);
> - };
> -
> -Every remoteproc implementation should at least provide the ->start and ->stop
> -handlers. If rpmsg/virtio functionality is also desired, then the ->kick handler
> -should be provided as well.
> -
> -The ->start() handler takes an rproc handle and should then power on the
> -device and boot it (use rproc->priv to access platform-specific private data).
> -The boot address, in case needed, can be found in rproc->bootaddr (remoteproc
> -core puts there the ELF entry point).
> -On success, 0 should be returned, and on failure, an appropriate error code.
> -
> -The ->stop() handler takes an rproc handle and powers the device down.
> -On success, 0 is returned, and on failure, an appropriate error code.
> -
> -The ->kick() handler takes an rproc handle, and an index of a virtqueue
> -where new message was placed in. Implementations should interrupt the remote
> -processor and let it know it has pending messages. Notifying remote processors
> -the exact virtqueue index to look in is optional: it is easy (and not
> -too expensive) to go through the existing virtqueues and look for new buffers
> -in the used rings.
> -
> -Binary Firmware Structure
> -=========================
> -
> -At this point remoteproc supports ELF32 and ELF64 firmware binaries. However,
> -it is quite expected that other platforms/devices which we'd want to
> -support with this framework will be based on different binary formats.
> -
> -When those use cases show up, we will have to decouple the binary format
> -from the framework core, so we can support several binary formats without
> -duplicating common code.
> -
> -When the firmware is parsed, its various segments are loaded to memory
> -according to the specified device address (might be a physical address
> -if the remote processor is accessing memory directly).
> -
> -In addition to the standard ELF segments, most remote processors would
> -also include a special section which we call "the resource table".
> -
> -The resource table contains system resources that the remote processor
> -requires before it should be powered on, such as allocation of physically
> -contiguous memory, or iommu mapping of certain on-chip peripherals.
> -Remotecore will only power up the device after all the resource table's
> -requirement are met.
> -
> -In addition to system resources, the resource table may also contain
> -resource entries that publish the existence of supported features
> -or configurations by the remote processor, such as trace buffers and
> -supported virtio devices (and their configurations).
> -
> -The resource table begins with this header::
> -
> - /**
> - * struct resource_table - firmware resource table header
> - * @ver: version number
> - * @num: number of resource entries
> - * @reserved: reserved (must be zero)
> - * @offset: array of offsets pointing at the various resource entries
> - *
> - * The header of the resource table, as expressed by this structure,
> - * contains a version number (should we need to change this format in the
> - * future), the number of available resource entries, and their offsets
> - * in the table.
> - */
> - struct resource_table {
> - u32 ver;
> - u32 num;
> - u32 reserved[2];
> - u32 offset[0];
> - } __packed;
> -
> -Immediately following this header are the resource entries themselves,
> -each of which begins with the following resource entry header::
> -
> - /**
> - * struct fw_rsc_hdr - firmware resource entry header
> - * @type: resource type
> - * @data: resource data
> - *
> - * Every resource entry begins with a 'struct fw_rsc_hdr' header providing
> - * its @type. The content of the entry itself will immediately follow
> - * this header, and it should be parsed according to the resource type.
> - */
> - struct fw_rsc_hdr {
> - u32 type;
> - u8 data[0];
> - } __packed;
> -
> -Some resources entries are mere announcements, where the host is informed
> -of specific remoteproc configuration. Other entries require the host to
> -do something (e.g. allocate a system resource). Sometimes a negotiation
> -is expected, where the firmware requests a resource, and once allocated,
> -the host should provide back its details (e.g. address of an allocated
> -memory region).
> -
> -Here are the various resource types that are currently supported::
> -
> - /**
> - * enum fw_resource_type - types of resource entries
> - *
> - * @RSC_CARVEOUT: request for allocation of a physically contiguous
> - * memory region.
> - * @RSC_DEVMEM: request to iommu_map a memory-based peripheral.
> - * @RSC_TRACE: announces the availability of a trace buffer into which
> - * the remote processor will be writing logs.
> - * @RSC_VDEV: declare support for a virtio device, and serve as its
> - * virtio header.
> - * @RSC_LAST: just keep this one at the end
> - * @RSC_VENDOR_START: start of the vendor specific resource types range
> - * @RSC_VENDOR_END: end of the vendor specific resource types range
> - *
> - * Please note that these values are used as indices to the rproc_handle_rsc
> - * lookup table, so please keep them sane. Moreover, @RSC_LAST is used to
> - * check the validity of an index before the lookup table is accessed, so
> - * please update it as needed.
> - */
> - enum fw_resource_type {
> - RSC_CARVEOUT = 0,
> - RSC_DEVMEM = 1,
> - RSC_TRACE = 2,
> - RSC_VDEV = 3,
> - RSC_LAST = 4,
> - RSC_VENDOR_START = 128,
> - RSC_VENDOR_END = 512,
> - };
> -
> -For more details regarding a specific resource type, please see its
> -dedicated structure in include/linux/remoteproc.h.
> -
> -We also expect that platform-specific resource entries will show up
> -at some point. When that happens, we could easily add a new RSC_PLATFORM
> -type, and hand those resources to the platform-specific rproc driver to handle.
> -
> -Virtio and remoteproc
> -=====================
> -
> -The firmware should provide remoteproc information about virtio devices
> -that it supports, and their configurations: a RSC_VDEV resource entry
> -should specify the virtio device id (as in virtio_ids.h), virtio features,
> -virtio config space, vrings information, etc.
> -
> -When a new remote processor is registered, the remoteproc framework
> -will look for its resource table and will register the virtio devices
> -it supports. A firmware may support any number of virtio devices, and
> -of any type (a single remote processor can also easily support several
> -rpmsg virtio devices this way, if desired).
> -
> -Of course, RSC_VDEV resource entries are only good enough for static
> -allocation of virtio devices. Dynamic allocations will also be made possible
> -using the rpmsg bus (similar to how we already do dynamic allocations of
> -rpmsg channels; read more about it in rpmsg.txt).
> +The firmware must provide remoteproc with information regarding the
> +virtio devices it supports and their configurations: an `RSC_VDEV`
> +resource entry should detail the virtio device ID (as defined in
> +`virtio_ids.h`), virtio features, virtio config space, vrings
> +information, etc.
> +
> +Upon registration of a new remote, the remoteproc framework
> +searches for its resource table and registers the supported virtio
> +devices. A firmware may support multiple virtio devices, of various
> +types (a single remote can support multiple rpmsg virtio
> +devices if required).
> +
> +Moreover, `RSC_VDEV` resource entries suffice for static allocation
> +of virtio devices. Dynamic allocations will also be supported using
> +the rpmsg bus, akin to the handling of dynamic allocations for rpmsg
> +channels. For more information, refer to rpmsg documentation.
> --
> 2.39.3 (Apple Git-146)
>
