On 04/08/2023 14:19, Shengjiu Wang wrote: > On Wed, Aug 2, 2023 at 8:28 PM Hans Verkuil <hverkuil@xxxxxxxxx> wrote: >> >> On 02/08/2023 14:02, Shengjiu Wang wrote: >>> On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@xxxxxxx> wrote: >>>> >>>> On Wed, 02 Aug 2023 09:32:37 +0200, >>>> Hans Verkuil wrote: >>>>> >>>>> Hi all, >>>>> >>>>> On 25/07/2023 08:12, Shengjiu Wang wrote: >>>>>> Audio signal processing has the requirement for memory to >>>>>> memory similar as Video. >>>>>> >>>>>> This patch is to add this support in v4l2 framework, defined >>>>>> new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and >>>>>> V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format >>>>>> for audio case usage. >>>>>> >>>>>> The created audio device is named "/dev/audioX". >>>>>> >>>>>> And add memory to memory support for two kinds of i.MX ASRC >>>>>> module >>>>> >>>>> Before I spend time on this: are the audio maintainers OK with doing >>>>> this in V4L2? >>>>> >>>>> I do want to have a clear statement on this as it is not something I >>>>> can decide. >>>> >>>> Well, I personally don't mind to have some audio capability in v4l2 >>>> layer. But, the only uncertain thing for now is whether this is a >>>> must-have or not. >>>> >>> >>> Thanks, I am also not sure about this. I am also confused that why >>> there is no m2m implementation for audio in the kernel. Audio also >>> has similar decoder encoder post-processing as video. >>> >>>> >>>> IIRC, the implementation in the sound driver side was never done just >>>> because there was no similar implementation? If so, and if the >>>> extension to the v4l2 core layer is needed, shouldn't it be more >>>> considered for the possible other route? >>>> >>> >>> Actually I'd like someone could point me to the other route. I'd like to >>> try. >>> >>> The reason why I select to extend v4l2 for such audio usage is that v4l2 >>> looks best for this audio m2m implementation. v4l2 is designed for m2m >>> usage. if we need implement another 'route', I don't think it can do better >>> that v4l2. >>> >>> I appreciate that someone can share his ideas or doable solutions. >>> And please don't ignore my request, ignore my patch. >> >> To give a bit more background: if it is decided to use the v4l API for this >> (and I have no objection to this from my side since API/framework-wise it is a >> good fit for this), then there are a number of things that need to be done to >> get this into the media subsystem: >> >> - documentation for the new uAPI >> - add support for this to v4l2-ctl >> - add v4l2-compliance tests for the new device >> - highly desirable: have a virtual driver (similar to vim2m) that supports this: >> it could be as simple as just copy input to output. This helps regression >> testing. >> - it might need media controller support as well. TBD. >> >> None of this is particularly complex, but taken all together it is a fair >> amount of work that also needs a lot of review time from our side. >> >> I want to add one more option to the mix: drivers/media/core/v4l2-mem2mem.c is >> the main m2m framework, but it relies heavily on the videobuf2 framework for >> the capture and output queues. >> >> The core vb2 implementation in drivers/media/common/videobuf2/videobuf2-core.c >> is independent of V4L2 and can be used by other subsystems (in our case, it is >> also used by the DVB API). It is a possibility to create an alsa version of >> v4l2-mem2mem.c that uses the core vb2 code with an ALSA uAPI on top. >> >> So in drivers/media/common/videobuf2/ you would have a videobuf2-alsa.c besides >> the already existing videobuf2-v4l2.c and -dvb.c. >> >> Perhaps parts of v4l2-mem2mem.c can be reused as well in that case, but I am >> not sure if it is worth the effort. I suspect copying it to an alsa-mem2mem.c >> and adapting it for alsa is easiest if you want to go that way. >> > > Thanks. > > Does this means that videobuf2-v4l2.c and v4l2-mem2mem.c are dedicate > for video device? if audio want to use v4l2 framework, need to create > videobuf2-alsa.c and alsa-mem2mem.c, but it may cause a lot of function > duplicate. The videobuf2-v4l2.c sits on top of videobuf2-core.c and provides the V4L2 uAPI for the streaming functionality. If you don't want to use the V4L2 uAPI for this, then you would need a videobuf2-alsa.c that provides a (possibly new) ALSA uAPI. Whether that makes sense is something I cannot decide. v4l2-mem2mem.c uses videobuf2-v4l2.c, so if you need a ALSA version, then you probably need to create an alsa-mem2mem.c (possibly some functionality can be shared). It's just a third option, and it can be useful if there is a strong desire to keep the uAPI for this functionality entirely within the ALSA subsystem, but you want to reuse the streaming I/O functionality that the videobuf2 core provides. If the decision is that it is fine to use the V4L2 uAPI for this type of audio functionality through a /dev/v4l-audioX device, then just ignore this option and use V4L2. Regards, Hans
