On Tue, Mar 28, 2023 at 07:32:27AM +0000, Wu, Wentong wrote: > > > > -----Original Message----- > > From: Sakari Ailus <sakari.ailus@xxxxxxxxxxxxxxx> > > Sent: Monday, March 27, 2023 5:37 PM > > > > Hi Wentong, > > > > On Mon, Mar 27, 2023 at 08:13:29AM +0000, Wu, Wentong wrote: > > > > > > > > > > -----Original Message----- > > > > From: Sakari Ailus <sakari.ailus@xxxxxxxxxxxxxxx> > > > > Sent: Monday, March 27, 2023 3:50 PM > > > > > > > > Hi Wentong, > > > > > > > > On Mon, Mar 27, 2023 at 07:33:48AM +0000, Wu, Wentong wrote: > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Sakari Ailus <sakari.ailus@xxxxxxxxxxxxxxx> > > > > > > Sent: Monday, March 27, 2023 3:21 PM > > > > > > > > > > > > Hi Wentong, > > > > > > > > > > > > On Mon, Mar 27, 2023 at 02:23:05PM +0800, Wentong Wu wrote: > > > > > > > Intel Visual Sensing Controller (IVSC), codenamed "Clover > > > > > > > Falls", is a companion chip designed to provide secure and low > > > > > > > power vision capability to IA platforms. IVSC is available in > > > > > > > existing commercial platforms from multiple OEMs. > > > > > > > > > > > > > > The primary use case of IVSC is to bring in context awareness. > > > > > > > IVSC interfaces directly with the platform main camera sensor > > > > > > > via a CSI-2 link and processes the image data with the > > > > > > > embedded AI engine. The detected events are sent over I2C to > > > > > > > ISH (Intel Sensor > > > > > > > Hub) for additional data fusion from multiple sensors. The > > > > > > > fusion results are used to implement advanced use cases like: > > > > > > > - Face detection to unlock screen > > > > > > > - Detect user presence to manage backlight setting or waking > > > > > > > up system > > > > > > > > > > > > > > Since the Image Processing Unit(IPU) used on the host > > > > > > > processor needs to configure the CSI-2 link in normal camera > > > > > > > usages, the > > > > > > > CSI-2 link and camera sensor can only be used in > > > > > > > mutually-exclusive ways by host IPU and IVSC. By default the > > > > > > > IVSC owns the CSI-2 link and camera sensor. The IPU driver can > > > > > > > take ownership of the CSI-2 link and camera sensor using > > > > > > > interfaces exported > > > > via v4l2 sub-device. > > > > > > > > > > > > > > Switching ownership requires an interface with two different > > > > > > > hardware modules inside IVSC. The software interface to these > > > > > > > modules is via Intel MEI (The Intel Management Engine) commands. > > > > > > > These two hardware modules have two different MEI UUIDs to > > > > > > > enumerate. These hardware > > > > > > modules are: > > > > > > > - ACE (Algorithm Context Engine): This module is for > > > > > > > algorithm computing when IVSC owns camera sensor. Also ACE > > > > > > > module controls camera sensor's ownership. This hardware > > > > > > > module is used to set ownership of > > > > > > camera sensor. > > > > > > > - CSI (Camera Serial Interface): This module is used to route > > > > > > > camera sensor data either to IVSC or to host for IPU driver and > > application. > > > > > > > > > > > > > > IVSC also provides a privacy mode. When privacy mode is turned > > > > > > > on, camera sensor can't be used. This means that both ACE and > > > > > > > host IPU can't get image data. And when this mode is turned > > > > > > > on, users are informed via > > > > > > > v4l2 control API. > > > > > > > > > > > > > > In summary, to acquire ownership of camera by IPU driver, > > > > > > > first ACE module needs to be informed of ownership and then to > > > > > > > setup MIPI CSI-2 link for the camera sensor and IPU. > > > > > > > > > > > > > > Implementation: > > > > > > > There are two different drivers to handle ACE and CSI hardware > > > > > > > modules inside IVSC. > > > > > > > - ivsc_csi: MEI client driver to send commands and receive > > > > > > > notifications from CSI module. > > > > > > > - ivsc_ace: MEI client driver to send commands and get status > > > > > > > from ACE module. > > > > > > > Interface is exposed via v4l2 sub-devcie APIs to acquire and > > > > > > > release camera sensor and CSI-2 link. > > > > > > > > > > > > Thanks for the update. > > > > > > > > > > > > Could you elaborate the decision of keeping the csi_bridge > > > > > > entirely separate from the cio2_bridge (to be turned to ipu_bridge first)? > > > > > > Both are doing essentially the same and using the same data > > > > > > structures, > > > > aren't they? > > > > > > > > > > yes, they're doing same thing to bridge the software nodes needed > > > > > by v4l2, but they have different type devices(pci and mei_client > > > > > device) and dependency. And they have same SSDB definition in > > > > > DSDT, so the structures are almost same. > > > > > > > > If there are differences, what are they? > > > > > > > The properties of swnode are same, because we need same info to > > > configure CSI2. But csi bridge can get sensor connected to IVSC by > > > dependency info instead of just polling the only supported sensors for > > > ipu. Also IVSC > > > > It's totally fine the method for finding the sensor is different, this should be an > > extra argument for the bridge init function and a small number of lines of > > additional code. > > > > > doesn't need vcm. And the topology of dependency is different. > > > > The VCM is optional for current cio2_bridge, too. > > > > It also seems that this currently creates swnodes on mei csi side only for the > > connection towards the sensor, not the IPU. > > IPU could find the MEI CSI software node by software_node_find_by_name, > and then setup the bridge between MEI CSI and IPU. > > > At the moment you can't reliably > > add software nodes to a device that has may be already probing so both of > > these should be created at the same time. > > If you're worried about sequence, IPU driver could make software node and > register it, MEI CSI bridge could find IPU software node by software_node_find_by_name, > if the return value is NULL, driver will defer the probe. The IPU driver would also need to figure out whether a given sensor should be behind the IVSC. I imagine it'll be hard to achieve this reasonably neatly if you split the implementation into different parts --- both of which need the same information from the same place (and in neither case it's under the same device than to which the driver is bound). > > > > > > > > > > What comes to cio2_bridge, the fact that it's related to a PCI > > > > device doesn't seem to matter after initialisation so it could as well work > > with struct device. > > > > > > > > > > > > > > I have no idea what the ipu bridge would be like, but IVSC csi > > > > > bridge can be configured via kconfig to enable/disable. > > > > > > > > Please work out the details with Bingbu. > > > > > > @bingbu.cao@xxxxxxxxxxxxxxx @Sakari Ailus Please share your design idea > > here. > > > What the ipu bridge would be like? What's the responsibility of ipu bridge? > > > > I'd expect Bingbu to have patches to turn the current cio2_bridge to an > > ipu_bridge at some point. These should come on top of those patches. > > When will this be ready if you already make the plan? But could you please help > review my other patches except bridge? Please configure your e-mail client to wrap your lines at most at 74 characters or so. I think it's binding this all together that is the concern, the individual drivers much less so. -- Regards, Sakari Ailus
