Hi, With the work done on the media request API and the cedrus driver for Allwinner ARM SoCs, we now have a kernel interface for exposing fixed- hardware video decoding pipelines (currently MPEG-2 and H.264, with H.265 on the way). Some work remains on the per-format interface and we are looking to improve latency-related aspects, but we are all set to have a nice interface here, that plays well with e.g. ffmpeg. A specific situation came to my interest, which is apparently quite common: some platforms have general-purpose microcontrollers embedded, which can help with video decoding. They are however rarely to never used to do the decoding itself (since they are general-purpose, not DSPs) and just coordinate the decoding with the fixed-pipeline decoding hardware block. The advantage is that the interface is just a simple mailbox and the raw video bitstream from the file can be passed directly without the need for userspace to do any parsing that the codec requires. One side-effect from this setup is that the actual hardware register layout of the decoder is hidden away in a non-free piece of microcontroller firmware, that's usually loaded at run-time. With the recent developments on the media interface, we could interface with these hardware decoders directly, which offers various advantages: - we no longer need a 3rd party external non-free firmware, which just makes distribution easier for everyone and allows support in fully- free setups; - all the usual advantages of having free code that can be fixed and updated instead of an obscure binary that many not always be doing the right thing; - parsing of the slices is probably best done in userspace, and I heard that ffmpeg does this threaded, so there could be a latency advantage there as well, not to mention that it avoids the drag of a mailbox interface altogether; - the general-purpose micro-controller can then be reused for something useful where it could actually make a performance difference. As far as I understand, it seems that the video decoder for MT8173 fails in that category, where a MD32 general-purpose micro-controller is used to only do the parsing. We even have device-tree nodes about the decoder and encoder, but no register layout. So I was wondering if the linux-media community should set some boundaries here and push towards native implementations instead of firmware-based ones. My opininon is that it definitely should. It seems that other platforms (e.g. Tegra K1 and onwards) are in the same situation, and I think the ChromiumOS downstream kernel uses an obscure firmware on a general-purpose auxiliary ARM core (that's also used at boot time IIRC). What do you think? Cheers, Paul -- Paul Kocialkowski, Bootlin Embedded Linux and kernel engineering https://bootlin.com
