On 11/18/2014 02:15 PM, Sakari Ailus wrote: > Hi Shuah, > > Many thanks for the RFC, for and being so patient with my terrible review > schedule. My comments below. > > On Tue, Nov 04, 2014 at 04:08:50PM -0700, Shuah Khan wrote: >> Hi Mauro, >> >> Here is the RFC as promised. I also included the Media controller >> as a an alternative and captured the discussion in the thread on >> that topic. Please review. >> >> -- Shuah >> >> ----------------------------------------------------------------- >> RFC Media Token API Specification >> >> Let's start with a diagram of a media device (without IR, eeprom >> and others): >> >> http://linuxtv.org/downloads/presentations/typical_hybrid_hardware.png >> >> The dot lines represent the parts of the graph that are switched by >> the tuner, DMA or input select. >> >> Please notice that the DMA engines, together with the stuff needed to >> control A/V switches is at one single chip. Changing the registers there >> can affect the other streams, specially on most sophisticated devices >> like cx231xx, where it even has a power management IP block that >> validates if a device to be turned on/off won't exceed the maximum >> drain current of 500mA. That's basically why we need to do a temporary >> lock alsa, dvb, v4l and IR drivers when doing certain changes. >> >> Also, please notice that I2C buses that can be as slow as 10kbps >> are used to control for several devices, like: >> - the tuner >> - the Digital TV (DTV) demod >> - Analog and/or Video demod (sometimes embedded at the main >> chip) >> - DTV demux (sometimes embedded at the main chip) >> - The remote controller (sometimes embedded at the main chip) >> >> For some devices, after powered on, or when certain parameters change, a >> new firmware (and sometimes a hardware reset) is required. The firmware >> size can be about 64KB or even bigger. >> >> Also, the A/V switch it is actually two independent switches (or one >> switch for video and one audio mux for audio) that needs to be changed >> together when the source changes. >> >> There are two components that are shared there between analog and >> digital: the tuner (where the signal is captured) and the DMA engine >> used to stream analog and Digital TV (dvb). >> >> PS.: the diagram is over-simplified, as the tuner is just one of the >> possible inputs for the analog part of the device. Other possible >> inputs are S-Video, composite, HDMI, etc. >> >> Sometimes, the audio DMA is also shared, e. g. just one stream comes >> from the hardware. It is up to the driver to split audio and video and >> send them to the V4L2 and ALSA APIs. This is the case of tm6000 driver. >> >> Those shared components can be used either at analog or digital mode, >> but not at the same time. >> >> Also, programming the V4L2 analog and audio DMA and demods should be >> done via V4L2 API, as this API allows the selection of the proper >> audio/video input (almost all devices have multiple analog inputs). >> >> Please notice that, if the tuner is on digital mode, the entire analog >> path is disabled, including ALSA output. >> >> If the tuner is on analog mode, both ALSA and V4L2 can work at the >> same time. However, during the period where the tuner firmware is >> loaded, and during the DMA configuration and input selection time, >> neither ALSA or V4L2 can stream. Such configuration/firmware load >> is commanded via V4L2 API, as ALSA knows nothing about tuner or >> input selection. >> >> At a higher level the problem description is: >> >> There are 3 different device files that get created to control >> tuner and audio functions on a media device. 4 drivers (dvb, >> v4l2, alsa, and the main usb driver for the usb device), and >> 3 core APIs (dvb-core, v4l-core, audio) that control the tuner >> and audio hardware and provide user API to these 3 device files. >> >> The above driver model is simplified, there's 4th component for >> some drivers: the mceusb driver, that handles remote controllers. >> The mceusb handles the Microsoft Media Center Remote Control >> protocol. It supports stand alone remote controller devices, but >> it also supports a few USB devices that use a separate interface >> for IR. >> >> There are currently some issues on cx231xx and mceusb, as both drivers >> can be used at the same time, but, when cx231xx sends certain commands, >> the mceusb IR polls fail. This is out of the scope of the audio lock, >> but it also needs to be addressed some day. >> >> Most media user applications, drivers and the core have no knowledge >> of each other. The only thing that is common across all these drivers >> is the parent device for the main usb device which is controlled by >> the usb driver. >> >> Some media user applications like MythTV can handle all 3 APIs, >> however, MythTV doesn't know how to associate ALSA, V4L2 and DVB >> devnodes that belong to the same device. If MythTV finds, 3 V4L2 >> nodes, 3 ALSA nodes, and 1 DVB node, it doesn't know which device >> is associated with the DVB node. >> >> Almost all applications that are aware of V4L2 API are also aware of >> ALSA API and may associate audio and video, as there is a way to >> associate it using sysfs. However, several applications don't use it. >> >> The premise for the main design idea in this series is creating >> a common construct at the parent device structure that is visible >> to all drivers to act as a master access control (lock). Let's call >> this media token object with two sub-tokens one for tuner and another >> for audio. >> >> Each of the APIS evolved separately, hence have their own backwards >> compatibility to maintain. Starting with v4l2: >> >> V4L2 case: >> Multiple v4l2 applications are allowed to open /dev/video0 in >> read/write mode with no restrictions as long as the tuner is in >> analog mode. V4L2-core handles conflicting requests between v4l2 >> applications. V4L2-core doesn't have the knowledge that the tuner >> is in use by a dvb and/or audio is in use. Individual drivers >> may have this knowledge as, except for one case (bttv driver), >> they share some data. >> >> As soon as a V4L2 application starts, digital stream glitches and >> audio glitches. >> >> DVB case: >> Multiple dvb applications can open the dvb device in read only mode. >> As soon an application open the device read/write mode a separate >> kthread is kicked off to handle the request. Only one application >> can open the device in read/write mode. There's no issue with ALSA >> in R/O mode, as the application is not allowed to modify anything >> with the stream. This is used only to monitor an already opened >> device in R/W mode. >> >> Similar to V4L2-core case, dvb-core doesn't have any knowledge that >> the tuner is in use by v4l2 and/or audio is in use. As soon as a >> dvb application starts v4l2 video glitches and audio glitches. >> >> Audio case: >> Same scenario is applicable to audio application. When a v4l2 or dvb >> application starts, audio application gets impacted. >> >> Problems to address: >> >> Dvb owns tuner and audio: another dvb, v4l2 application and >> ALSA application should detect >> tuner/audio busy right away and exit. >> Dvb applications don't use audio node, >> however, devices can't use audio hardware >> while in DVB mode. >> >> V4l2 owns tuner and audio: dvb should detect tuner/audio busy >> right away and exit. >> The V4L2-core should only hold the token for >> the required time to initialize the device >> and/or load the firmware. >> ALSA applications should wait for v4L2-core >> to finish programming at audio, and should >> keep working after that. >> >> Audio owns audio: dvb applications should detect audio busy and >> exit. V4L2 applications should work. However, when certain V4L2 >> ioctls are issued, the audio device driver should not send any >> command to the hardware. After such commands, the audio mixers >> may change. This is why two tokens are necessary, one for tuner >> and another for audio. >> >> Because of the above mentioned difference in behavior between dvb >> and v4l applications when audio is busy, two tokens (one for tuner >> and another for audio) are necessary and audio token lock should not >> be held at ALSA open/close. >> >> Special cases: >> >> Dvb applications access tuner in exclusive mode. i.e only one dvb >> application at a time is allowed to open the device read/write mode. >> Dvb applications don't use audio node, however, devices can't use >> audio hardware while in DVB mode. Dvb applications receive data as >> MPEG-TS, using a separate device node. The same DMA engine that >> provides video (and, sometimes audio) is used by the DVB device >> node, making it inaccessible to audio applications while tuner >> is in DVB mode. Hence, the need to prevent audio applications from >> accessing audio node when tuner is in DVB mode. As a result, dvb-core >> will have to hold tuner and audio tokens so v4l2-core and ALSA know >> that audio is not available. Dvb disables audio hardware so it could >> be powered-off in some cases. >> >> Audio applications access audio in exclusive mode. i.e only one audio >> application at a time is allowed to open the device in read/write mode. >> Audio applications create threads and thread closes and re-opens the >> audio device. Threads can do this and hence something that higher level >> construct has to allow. Audio application has to hold audio token so >> dvb and v4l2 know that it is in use. >> >> V4l2 applications access tuner and audio in shared v4l2 mode. >> i.e several v4l2 processes and threads could use tuner and audio >> at the same time. V4L2 core handles concurrency. There's just >> one file handler with full control to start/stop stream at V4L2 >> side. The higher level construct should not break the ability of >> multiple v4l2 applications to access tuner and audio in shared >> mode, and disallow dvb and audio applications access when they >> are in use by the V4L2-core. >> >> Dvb-core when it gets the tuner, it should also obtain audio right >> away. v4l2-core when it gets the tuner, it should get the audio at >> the same time. When dvb-core has the tuner, v4l2 shouldn't get it >> and vice versa. >> >> When dvb-core has the audio locked, audio application should detect >> condition and stop streaming, as part of the hardware can be powered >> off. It can only return opening the device after DVB releases audio >> token. >> >> When v4l2-core has audio locked, audio application should detect the >> condition and stop sending commands to audio hardware. It can only >> resume audio access after V4L2 releases audio token. >> >> Open issues: >> During testing, snd_pcm_lib_ioctls are coming from dvb application. >> It is likely that these are related to the audio output and not audio >> capture or the application in question is an hybrid one. This issue >> needs further investigation. >> >> Alternatives: (proposed by Sakari Ailus) >> Can Media controller be used to solve the problem? >> >> The usage of the media controller for this specific usage is that >> we should not force userspace applications to be aware of the >> media controller just because of hardware locking. >> >> Currently, media entities may only be entities bound to a given >> subsystem, but likely need to change media controller for complex >> embedded DVB device support ... >> >> In case of the Media controller, mutual exclusion of different users >> is currently performed by adding the entities to a pipeline and >> incrementing the streaming count once streaming is enabled --- on >> different interfaces streaming may mean a different thing. >> >> However, we'll still need to find a way for ALSA to prevent it to use >> the audio demod and DMA engine that will be powered off when DVB is >> streaming. >> >> The Media controller interface does not handle serializing potential >> users that may wish to configure the device. Handling serializing is >> necessary if Media controller is extended instead of pursuing Media >> Token API to solve the problem. >> >> Reconfiguring the DMA engine and some other registers via V4L2 API >> should be blocked. The same applies to firmware load, if the device >> is using tuner input for analog TV. >> >> If we use the media controller, we'll need to add a state to it, >> to indicate that a block at the pipeline is being reconfigured. >> >> It is dependent on Media Controller adoption on ALSA as well. > > Thank you for the detailed description of the problem domain. > > Using Media controller for this at this point isn't straightforward, and > especially for existing drivers for which the current APIs serve the > existing devices well enough, perhaps not the best solution even when the > missing pieces were implemented. > > More complex devices, though, still need MC in order to control them in a > meaningful way. If a tuner is connected into one, the media token framework > wouldn't help there. In other words, we'll need something for such devices > as well. That'd be proper MC support for DVB and ALSA, but it'll be a > separate discussion. > > One big upside in this patchset is that it does not change the user space > interface. Right that is the goal. > > One concern I have is how generic this framework really is. Do you see > potential use cases outside the current one? > > I'd move this under drivers/media, and possibly think of the naming of the > framework a little bit. Others gave the same feedback on both of your points on generic and location. I moved it under drivers/media for the patch v3 pre-work I did before we started the RFC discussion. So we are good there. > > I'll review the rest of the v2. > Great. Please do and give me feedback. I will wait for that before I get started with patch v3 work. thanks, -- Shuah -- Shuah Khan Sr. Linux Kernel Developer Samsung Research America (Silicon Valley) shuahkh@xxxxxxxxxxxxxxx | (970) 217-8978 -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
