Hi, Wentong, On 2/15/23 5:43 PM, Laurent Pinchart wrote: > Hello Wentong, > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote: >> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a >> companion chip designed to provide secure and low power vision capability >> to IA platforms. IVSC is available in existing commercial platforms from >> multiple OEMs. >> >> The primary use case of IVSC is to bring in context awareness. IVSC >> interfaces directly with the platform main camera sensor via a CSI-2 link >> and processes the image data with the embedded AI engine. The detected >> events are sent over I2C to ISH (Intel Sensor Hub) for additional data >> fusion from multiple sensors. The fusion results are used to implement >> advanced use cases like: >> - Face detection to unlock screen >> - Detect user presence to manage backlight setting or waking up system > > Do you have plan to support these features in the ivsc driver in the > future ? > >> Since the Image Processing Unit(IPU) used on the host processor needs to >> configure the CSI-2 link in normal camera usages, the CSI-2 link and >> camera sensor can only be used in mutually-exclusive ways by host IPU and >> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU >> driver can take ownership of the CSI-2 link and camera sensor using >> interfaces provided by this IVSC driver. >> >> Switching ownership requires an interface with two different hardware >> modules inside IVSC. The software interface to these modules is via Intel >> MEI (The Intel Management Engine) commands. These two hardware modules >> have two different MEI UUIDs to enumerate. These hardware modules are: >> - ACE (Algorithm Context Engine): This module is for algorithm computing >> when IVSC owns camera sensor. Also ACE module controls camera sensor's >> ownership. This hardware module is used to set ownership of camera sensor. >> - CSI (Camera Serial Interface): This module is used to route camera >> sensor data either to IVSC or to host for IPU driver and application. >> >> IVSC also provides a privacy mode. When privacy mode is turned on, >> camera sensor can't be used. This means that both ACE and host IPU can't >> get image data. And when this mode is turned on, host IPU driver is >> informed via a registered callback, so that user can be notified. > > How does the privacy mode work, and how can the user trust that the > closed-source IVSC and IME firmwares will honour the privacy settings ? > Continue with question from Laurent, How IVSC handle the privacy request from user? Is there some notification mechanism to user-space? I'd have concern if IVSC driver need private callback to request back-end(e.g. ISP driver) to handle stream cutting. >> In summary, to acquire ownership of camera by IPU driver, first ACE >> module needs to be informed of ownership and then to setup MIPI CSI-2 >> link for the camera sensor and IPU. >> >> Implementation: >> There are two different drivers to handle ACE and CSI hardware modules >> inside IVSC. >> - mei_csi: MEI client driver to send commands and receive notifications >> from CSI module. >> - mei_ace: MEI client driver to send commands and get status from ACE >> module. >> Interface is exposed via ivsc.h to acquire and release camera sensor and >> CSI-2 link. > > Do I understand correctly, from your diagram below, that the > communication between the IME and IVSC goes through SPI ? > >> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor. >> ----------------------------------------------------------------------------- >> | Host Processor | >> | | >> | ----------------- ----------------- --------------- | >> | | | | | | | I2C | >> | | IPU | | ISH | |camera driver|--| | >> | | | | | | | | | >> | ----------------- ----------------- --------------- | | >> | | | | | | >> | | | --------------- | | >> | | | | | | | >> | | | | IVSC driver | | | >> | | | | | | | >> | | | --------------- | | >> | | | | | | >> ----------------|-----------------------|----------------------|---------|--- >> | CSI | I2C |SPI | >> | | | | >> ----------------|-----------------------|---------------- | | >> | IVSC | | | | >> | | | | | >> | ----------------- ----------------- | | | >> | | | | | | | | >> | | CSI | | ACE | |------| | >> | | | | | | | >> | ----------------- ----------------- | | >> | | | I2C | | >> ----------------|-----------------------|---------------- | >> | CSI | | >> | | | >> -------------------------------- | >> | | I2C | >> | camera sensor |-----------------------------| >> | | >> -------------------------------- >> >> Wentong Wu (3): >> media: pci: intel: ivsc: Add CSI submodule >> media: pci: intel: ivsc: Add ACE submodule >> media: pci: intel: ivsc: Add acquire/release API for ivsc >> >> drivers/media/pci/Kconfig | 1 + >> drivers/media/pci/intel/Makefile | 2 + >> drivers/media/pci/intel/ivsc/Kconfig | 12 + >> drivers/media/pci/intel/ivsc/Makefile | 7 + >> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++ >> drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++ >> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++ >> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++ >> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++ >> include/linux/ivsc.h | 74 ++++ >> 10 files changed, 1090 insertions(+) >> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig >> create mode 100644 drivers/media/pci/intel/ivsc/Makefile >> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c >> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c >> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h >> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c >> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h >> create mode 100644 include/linux/ivsc.h > -- Best regards, Bingbu Cao
