On Tue, Nov 21, 2023 at 5:22 PM Hans Verkuil <hverkuil@xxxxxxxxx> wrote:
>
> On 21/11/2023 08:37, Shengjiu Wang wrote:
> > Audio signal processing also has the requirement for memory to
> > memory similar as Video.
> >
> > This asrc memory to memory (memory ->asrc->memory) case is a non
> > real time use case.
> >
> > User fills the input buffer to the asrc module, after conversion, then asrc
> > sends back the output buffer to user. So it is not a traditional ALSA playback
> > and capture case.
> >
> > It is a specific use case, there is no reference in current kernel.
> > v4l2 memory to memory is the closed implementation, v4l2 current
> > support video, image, radio, tuner, touch devices, so it is not
> > complicated to add support for this specific audio case.
> >
> > Because we had implemented the "memory -> asrc ->i2s device-> codec"
> > use case in ALSA. Now the "memory->asrc->memory" needs
> > to reuse the code in asrc driver, so the first 3 patches is for refining
> > the code to make it can be shared by the "memory->asrc->memory"
> > driver.
> >
> > The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
> > user applications only use the ioctl of v4l2 framework.
> >
> > Other change is to add memory to memory support for two kinds of i.MX ASRC
> > module.
> >
> > changes in v10
> > - remove FIXED_POINT type
> > - change code base on media: v4l2-ctrls: add support for fraction_bits
> > - fix issue reported by kernel test robot
> > - remove module_alias
>
> Note that I still need a patch for vivid adding a fixed point test control.
>
> I think I want two controls: one INTEGER Q16 and one INTEGER64 Q63 (a nice
> corner case).
>
Not sure if we can do like this:
diff --git a/drivers/media/test-drivers/vivid/vivid-core.h
b/drivers/media/test-drivers/vivid/vivid-core.h
index cfb8e66083f6..def8cf6c30c1 100644
--- a/drivers/media/test-drivers/vivid/vivid-core.h
+++ b/drivers/media/test-drivers/vivid/vivid-core.h
@@ -222,6 +222,8 @@ struct vivid_dev {
struct v4l2_ctrl *boolean;
struct v4l2_ctrl *int32;
struct v4l2_ctrl *int64;
+ struct v4l2_ctrl *int32Q16;
+ struct v4l2_ctrl *int64Q63;
struct v4l2_ctrl *menu;
struct v4l2_ctrl *string;
struct v4l2_ctrl *bitmask;
diff --git a/drivers/media/test-drivers/vivid/vivid-ctrls.c
b/drivers/media/test-drivers/vivid/vivid-ctrls.c
index f2b20e25a7a4..c912b6776775 100644
--- a/drivers/media/test-drivers/vivid/vivid-ctrls.c
+++ b/drivers/media/test-drivers/vivid/vivid-ctrls.c
@@ -182,6 +182,28 @@ static const struct v4l2_ctrl_config vivid_ctrl_int64 = {
.step = 1,
};
+static const struct v4l2_ctrl_config vivid_ctrl_int32Q16 = {
+ .ops = &vivid_user_gen_ctrl_ops,
+ .id = VIVID_CID_INTEGER,
+ .name = "Integer 32 Bits Q16",
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .min = 0xffffffff80000000ULL,
+ .max = 0x7fffffff,
+ .step = 1,
+ .fraction_bits = 16,
+};
+
+static const struct v4l2_ctrl_config vivid_ctrl_int64Q63 = {
+ .ops = &vivid_user_gen_ctrl_ops,
+ .id = VIVID_CID_INTEGER64,
+ .name = "Integer 64 Bits Q63",
+ .type = V4L2_CTRL_TYPE_INTEGER64,
+ .min = 0x8000000000000000ULL,
+ .max = 0x7fffffffffffffffLL,
+ .step = 1,
+ .fraction_bits = 63,
+};
+
static const struct v4l2_ctrl_config vivid_ctrl_u32_array = {
.ops = &vivid_user_gen_ctrl_ops,
.id = VIVID_CID_U32_ARRAY,
@@ -1670,6 +1692,8 @@ int vivid_create_controls(struct vivid_dev *dev,
bool show_ccs_cap,
dev->button = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_button, NULL);
dev->int32 = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_int32, NULL);
dev->int64 = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_int64, NULL);
+ dev->int32Q16 = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_int32Q16, NULL);
+ dev->int64Q63 = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_int64Q63, NULL);
dev->boolean = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_boolean, NULL);
dev->menu = v4l2_ctrl_new_custom(hdl_user_gen, &vivid_ctrl_menu, NULL);
dev->string = v4l2_ctrl_new_custom(hdl_user_gen,
&vivid_ctrl_string, NULL);
(END)
> Regards,
>
> Hans
>
> >
> > changes in v9:
> > - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER.
> > - add MEDIA_INTF_T_V4L_AUDIO
> > - add media controller support
> > - refine the vim2m-audio to support 8k<->16k conversion.
> >
> > changes in v8:
> > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008
> > - update doc for FIXED_POINT
> > - address comments for imx-asrc
> >
> > changes in v7:
> > - add acked-by from Mark
> > - separate commit for fixed point, m2m audio class, audio rate controls
> > - use INTEGER_MENU for rate, FIXED_POINT for rate offset
> > - remove used fmts
> > - address other comments for Hans
> >
> > changes in v6:
> > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace
> > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two.
> > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT
> > - fix warning by kernel test rebot
> > - remove some unused format V4L2_AUDIO_FMT_XX
> > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver.
> > - rename audm2m to viaudm2m.
> >
> > changes in v5:
> > - remove V4L2_AUDIO_FMT_LPCM
> > - define audio pixel format like V4L2_AUDIO_FMT_S8...
> > - remove rate and format in struct v4l2_audio_format.
> > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
> > - updata document accordingly.
> >
> > changes in v4:
> > - update document style
> > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
> >
> > changes in v3:
> > - Modify documents for adding audio m2m support
> > - Add audio virtual m2m driver
> > - Defined V4L2_AUDIO_FMT_LPCM format type for audio.
> > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
> > - with modification in v4l-utils, pass v4l2-compliance test.
> >
> > changes in v2:
> > - decouple the implementation in v4l2 and ALSA
> > - implement the memory to memory driver as a platfrom driver
> > and move it to driver/media
> > - move fsl_asrc_common.h to include/sound folder
> >
> > Shengjiu Wang (14):
> > ASoC: fsl_asrc: define functions for memory to memory usage
> > ASoC: fsl_easrc: define functions for memory to memory usage
> > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
> > ASoC: fsl_asrc: register m2m platform device
> > ASoC: fsl_easrc: register m2m platform device
> > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
> > media: v4l2: Add audio capture and output support
> > media: uapi: Define audio sample format fourcc type
> > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO
> > media: uapi: Add audio rate controls support
> > media: uapi: Declare interface types for Audio
> > media: uapi: Add an entity type for audio resampler
> > media: imx-asrc: Add memory to memory driver
> > media: vim2m-audio: add virtual driver for audio memory to memory
> >
> > .../media/mediactl/media-types.rst | 11 +
> > .../userspace-api/media/v4l/buffer.rst | 6 +
> > .../userspace-api/media/v4l/common.rst | 1 +
> > .../media/v4l/dev-audio-mem2mem.rst | 71 +
> > .../userspace-api/media/v4l/devices.rst | 1 +
> > .../media/v4l/ext-ctrls-audio-m2m.rst | 41 +
> > .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++
> > .../userspace-api/media/v4l/pixfmt.rst | 1 +
> > .../media/v4l/vidioc-enum-fmt.rst | 2 +
> > .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +
> > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 +
> > .../media/v4l/vidioc-querycap.rst | 3 +
> > .../media/videodev2.h.rst.exceptions | 3 +
> > .../media/common/videobuf2/videobuf2-v4l2.c | 4 +
> > drivers/media/platform/nxp/Kconfig | 13 +
> > drivers/media/platform/nxp/Makefile | 1 +
> > drivers/media/platform/nxp/imx-asrc.c | 1264 +++++++++++++++++
> > drivers/media/test-drivers/Kconfig | 11 +
> > drivers/media/test-drivers/Makefile | 1 +
> > drivers/media/test-drivers/vim2m-audio.c | 799 +++++++++++
> > drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +
> > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 +
> > drivers/media/v4l2-core/v4l2-dev.c | 21 +
> > drivers/media/v4l2-core/v4l2-ioctl.c | 66 +
> > drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +-
> > include/media/v4l2-dev.h | 2 +
> > include/media/v4l2-ioctl.h | 34 +
> > .../fsl => include/sound}/fsl_asrc_common.h | 60 +
> > include/uapi/linux/media.h | 2 +
> > include/uapi/linux/v4l2-controls.h | 9 +
> > include/uapi/linux/videodev2.h | 41 +
> > sound/soc/fsl/fsl_asrc.c | 144 ++
> > sound/soc/fsl/fsl_asrc.h | 4 +-
> > sound/soc/fsl/fsl_asrc_dma.c | 2 +-
> > sound/soc/fsl/fsl_easrc.c | 233 +++
> > sound/soc/fsl/fsl_easrc.h | 6 +-
> > 36 files changed, 2977 insertions(+), 7 deletions(-)
> > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
> > create mode 100644 drivers/media/platform/nxp/imx-asrc.c
> > create mode 100644 drivers/media/test-drivers/vim2m-audio.c
> > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
> >
>
