Hi Amirreza,
Thanks for proposing this.
On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
<quic_azarrabi@xxxxxxxxxxx> wrote:
>
>
> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>
> Hi Jens,
>
> > Hi Amirreza,
> >
> > On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > <quic_azarrabi@xxxxxxxxxxx> wrote:
> >>
> >> The default context has a lifespan similar to the tee_device.
Since it's associated with tee_device context, let's call it obvious
via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
patch).
> >> It is used as a context for shared memory if the context to which the
> >> shared memory belongs is released, making the tee_shm an orphan.
> >> This allows the driver implementing shm_unregister to safely make
> >> subsequent calls, such as to a supplicant if needed.
> >>
> >> It also enables users to free the shared memory while the driver is
> >> blocked on unregister_tee_device safely.
> >>
> >> Preferably, this should be used for all driver internal uses, using
> >> teedev_get_def_context rather than calling teedev_open.
Makes sense to me.
> >>
> >> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@xxxxxxxxxxx>
> >> ---
> >> drivers/tee/optee/core.c | 2 +-
> >> drivers/tee/optee/ffa_abi.c | 2 +-
> >> drivers/tee/optee/smc_abi.c | 2 +-
> >> drivers/tee/tee_core.c | 83 +++++++++++++++++++++++++++++----------------
> >> drivers/tee/tee_private.h | 3 --
> >> drivers/tee/tee_shm.c | 18 ++--------
> >> include/linux/tee_core.h | 15 ++++++++
> >> include/linux/tee_drv.h | 7 ----
> >> 8 files changed, 73 insertions(+), 59 deletions(-)
> >>
> >> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> >> index c75fddc83576..78d43d0c8014 100644
> >> --- a/drivers/tee/optee/core.c
> >> +++ b/drivers/tee/optee/core.c
> >> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> >>
> >> optee_notif_uninit(optee);
> >> optee_shm_arg_cache_uninit(optee);
> >> - teedev_close_context(optee->ctx);
> >> +
> >> /*
> >> * The two devices have to be unregistered before we can free the
> >> * other resources.
> >> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> >> index f3af5666bb11..6ad94f0788ad 100644
> >> --- a/drivers/tee/optee/ffa_abi.c
> >> +++ b/drivers/tee/optee/ffa_abi.c
> >> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> >> optee_shm_arg_cache_init(optee, arg_cache_flags);
> >> mutex_init(&optee->rpmb_dev_mutex);
> >> ffa_dev_set_drvdata(ffa_dev, optee);
> >> - ctx = teedev_open(optee->teedev);
> >> + ctx = teedev_get_def_context(optee->teedev);
> >> if (IS_ERR(ctx)) {
> >> rc = PTR_ERR(ctx);
> >> goto err_rhashtable_free;
> >> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> >> index e9456e3e74cc..c77a3e631d04 100644
> >> --- a/drivers/tee/optee/smc_abi.c
> >> +++ b/drivers/tee/optee/smc_abi.c
> >> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> >> mutex_init(&optee->rpmb_dev_mutex);
> >>
> >> platform_set_drvdata(pdev, optee);
> >> - ctx = teedev_open(optee->teedev);
> >> + ctx = teedev_get_def_context(optee->teedev);
> >> if (IS_ERR(ctx)) {
> >> rc = PTR_ERR(ctx);
> >> goto err_supp_uninit;
> >> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> >> index 93f3b330aec8..805e1336089d 100644
> >> --- a/drivers/tee/tee_core.c
> >> +++ b/drivers/tee/tee_core.c
> >> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >> goto err;
> >> }
> >>
> >> - kref_init(&ctx->refcount);
> >> ctx->teedev = teedev;
> >> INIT_LIST_HEAD(&ctx->list_shm);
> >> rc = teedev->desc->ops->open(ctx);
> >> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >> }
> >> EXPORT_SYMBOL_GPL(teedev_open);
> >>
> >> -void teedev_ctx_get(struct tee_context *ctx)
> >> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> >> {
> >> - if (ctx->releasing)
> >> - return;
> >> + int rc;
> >> + struct tee_context *ctx = &teedev->def_ctx;
> >>
> >> - kref_get(&ctx->refcount);
> >> -}
> >> + ctx->teedev = teedev;
> >> + INIT_LIST_HEAD(&ctx->list_shm);
> >> + rc = teedev->desc->ops->open(ctx);
> >> + if (rc)
> >> + return ERR_PTR(rc);
> >
> > I think ctx->teedev and ctx->list_shm must always be initialized or
> > &teedev->def_ctx can't be used in teedev_close_context().
>
> True, but &teedev->def_ctx is never used in teedev_close_context().
> The closing of the &teedev->def_ctx simply ignored. So once opened,
> &teedev->def_ctx will always remain open until the tee_device is alive.
>
> > We could initialize teedev->def_ctx on the first call to teedev_open()
> > on that tee_device. We need a way to tell the
> > teedev->desc->ops->open() to the backed driver that it's initializing
> > the default context though, or optee_open() can't handle the
> > tee-supplicant case properly.
> >
>
> That's a good point. This way, it is guaranteed that there is one def_ctx
> per teedev. There should be a way to tell the open() callback that it is
> a def_ctx, so it is not registered as a supplicant context.
>
>
> > Should we allow this function to be called more than once for each teedev?
>
> Yes, moving to teedev_open() will fix the issue.
>
> > Do we need serialization in this function if it's called after the
> > driver is probed?
> >
>
> True. I'll make sure there is no race.
>
> >>
> >> -static void teedev_ctx_release(struct kref *ref)
> >> -{
> >> - struct tee_context *ctx = container_of(ref, struct tee_context,
> >> - refcount);
> >> - ctx->releasing = true;
> >> - ctx->teedev->desc->ops->release(ctx);
> >> - kfree(ctx);
> >> + return ctx;
> >> }
> >> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> >>
> >> -void teedev_ctx_put(struct tee_context *ctx)
> >> +void teedev_close_context(struct tee_context *ctx)
> >> {
> >> - if (ctx->releasing)
> >> + struct tee_device *teedev = ctx->teedev;
> >> + struct tee_shm *shm;
> >> +
> >> + if (ctx == &teedev->def_ctx)
> >> return;
> >>
> >> - kref_put(&ctx->refcount, teedev_ctx_release);
> >> -}
> >> + teedev->desc->ops->release(ctx);
> >>
> >> -void teedev_close_context(struct tee_context *ctx)
> >> -{
> >> - struct tee_device *teedev = ctx->teedev;
> >> + mutex_lock(&teedev->mutex);
> >> + list_for_each_entry(shm, &ctx->list_shm, link) {
> >> + /* Context released. However, shm still holding a teedev reference.
> >> + * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> >> + * fails (i.e. it is not accessible from userspace) but shm still
> >> + * holds a valid context for further clean up, e.g. shm_unregister().
> >> + */
> >
> > /*
> > * Please format
> > * multiline comments
> > * like this. Please
> > * keep the lines at
> > * max 80 columns
> > * here and at other
> > * places in the patch-
> > * set.
> > */
> >
>
> Ack.
>
> >> + shm->ctx = &teedev->def_ctx;
> >
> > shm->ctx will always point to a valid context, even if it is the
> > default context. It seems that we can always get hold of the correct
> > teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > redundant teedev in struct tee_shm"?
> >
>
> It was there in case we wanted to use NULL, but with def_ctx, it is not
> necessary. I am withdrawing that commit. :).
>
> > Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > teedev->def_ctx.list_shm?
+1
> >
>
> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> we are closing the def_ctx, the list is guaranteed to be empty.
>
> However, I understand it is cleaner and more consistent to do that rather
> than making changes to tee_shm_put().
>
> I'll do it.
>
> >> + }
> >> + mutex_unlock(&teedev->mutex);
> >>
> >> - teedev_ctx_put(ctx);
> >> + kfree(ctx);
> >> tee_device_put(teedev);
> >> }
> >> EXPORT_SYMBOL_GPL(teedev_close_context);
> >> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> >>
> >> teedev->desc = teedesc;
> >> teedev->pool = pool;
> >> + /* Only open default context when teedev_get_def_context() called. */
> >> + teedev->def_ctx.teedev = NULL;
Why don't you open the device context here only? This will associate
it automatically with teedev lifespan and then
teedev_get_def_context() will just return a reference to that.
-Sumit
> >>
> >> return teedev;
> >> err_devt:
> >> @@ -1027,16 +1035,31 @@ EXPORT_SYMBOL_GPL(tee_device_register);
> >>
> >> void tee_device_put(struct tee_device *teedev)
> >> {
> >> - mutex_lock(&teedev->mutex);
> >> - /* Shouldn't put in this state */
> >> - if (!WARN_ON(!teedev->desc)) {
> >> - teedev->num_users--;
> >> - if (!teedev->num_users) {
> >> - teedev->desc = NULL;
> >> - complete(&teedev->c_no_users);
> >> - }
> >> + const struct tee_desc *desc;
> >> +
> >> + scoped_guard(mutex, &teedev->mutex) {
> >> + desc = teedev->desc;
> >> +
> >> + /* Shouldn't put in this state */
> >> + if (WARN_ON(!desc))
> >> + return;
> >> +
> >> + /* If there is still users for teedev */
> >> + if (--teedev->num_users)
> >
> > Please do teedev->num_users-- first and then check. It makes the code
> > easier to read.
>
> Ack.
>
> >
> >> + return;
> >> +
> >> + /* tee_device_unregister() has been called and there is no
> >> + * user in userspace or kernel, including orphan shm for teedev.
> >> + * Set teedev->desc to NULL, so that teedev can not be reused.
> >> + */
> >> + teedev->desc = NULL;
> >> }
> >> - mutex_unlock(&teedev->mutex);
> >> +
> >> + /* Release the default context */
> >> + desc->ops->release(&teedev->def_ctx);
> >
> > This should only be done if teedev->def_ctx has been initialized.
> >
>
> Ack.
>
> > Cheers,
> > Jens
>
> Thank you very much for your comments :).
> If you're okay with introducing def_ctx, I'll prepare a complete patchset
> with all the details.
>
> Best Regards,
> Amir
>
>
> >
> >> + teedev->def_ctx.teedev = NULL;
> >> +
> >> + complete(&teedev->c_no_users);
> >> }
> >>
> >> bool tee_device_get(struct tee_device *teedev)
> >> diff --git a/drivers/tee/tee_private.h b/drivers/tee/tee_private.h
> >> index 9bc50605227c..6c7bcc308958 100644
> >> --- a/drivers/tee/tee_private.h
> >> +++ b/drivers/tee/tee_private.h
> >> @@ -17,9 +17,6 @@ int tee_shm_get_fd(struct tee_shm *shm);
> >> bool tee_device_get(struct tee_device *teedev);
> >> void tee_device_put(struct tee_device *teedev);
> >>
> >> -void teedev_ctx_get(struct tee_context *ctx);
> >> -void teedev_ctx_put(struct tee_context *ctx);
> >> -
> >> struct tee_shm *tee_shm_alloc_user_buf(struct tee_context *ctx, size_t size);
> >> struct tee_shm *tee_shm_register_user_buf(struct tee_context *ctx,
> >> unsigned long addr, size_t length);
> >> diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
> >> index c0164c0f4a01..f07274291edf 100644
> >> --- a/drivers/tee/tee_shm.c
> >> +++ b/drivers/tee/tee_shm.c
> >> @@ -59,8 +59,6 @@ static void tee_shm_release(struct tee_shm *shm)
> >> release_registered_pages(shm);
> >> }
> >>
> >> - teedev_ctx_put(shm->ctx);
> >> -
> >> kfree(shm);
> >>
> >> tee_device_put(teedev);
> >> @@ -93,13 +91,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
> >> shm->flags = flags;
> >> shm->teedev = teedev;
> >> shm->id = id;
> >> -
> >> - /*
> >> - * We're assigning this as it is needed if the shm is to be
> >> - * registered. If this function returns OK then the caller expected
> >> - * to call teedev_ctx_get() or clear shm->ctx in case it's not
> >> - * needed any longer.
> >> - */
> >> shm->ctx = ctx;
> >>
> >> rc = teedev->pool->ops->alloc(teedev->pool, shm, size, align);
> >> @@ -112,7 +103,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
> >> list_add_tail(&shm->link, &ctx->list_shm);
> >> mutex_unlock(&teedev->mutex);
> >>
> >> - teedev_ctx_get(ctx);
> >> return shm;
> >> err_kfree:
> >> kfree(shm);
> >> @@ -295,12 +285,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
> >> goto err_dev_put;
> >> }
> >>
> >> - teedev_ctx_get(ctx);
> >> -
> >> shm = kzalloc(sizeof(*shm), GFP_KERNEL);
> >> if (!shm) {
> >> ret = ERR_PTR(-ENOMEM);
> >> - goto err_ctx_put;
> >> + goto err_dev_put;
> >> }
> >>
> >> refcount_set(&shm->refcount, 1);
> >> @@ -313,7 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
> >> num_pages = iov_iter_npages(iter, INT_MAX);
> >> if (!num_pages) {
> >> ret = ERR_PTR(-ENOMEM);
> >> - goto err_ctx_put;
> >> + goto err_dev_put;
> >> }
> >>
> >> shm->pages = kcalloc(num_pages, sizeof(*shm->pages), GFP_KERNEL);
> >> @@ -361,8 +349,6 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
> >> kfree(shm->pages);
> >> err_free_shm:
> >> kfree(shm);
> >> -err_ctx_put:
> >> - teedev_ctx_put(ctx);
> >> err_dev_put:
> >> tee_device_put(teedev);
> >> return ret;
> >> diff --git a/include/linux/tee_core.h b/include/linux/tee_core.h
> >> index a38494d6b5f4..13393ddac530 100644
> >> --- a/include/linux/tee_core.h
> >> +++ b/include/linux/tee_core.h
> >> @@ -44,6 +44,7 @@
> >> * @idr: register of user space shared memory objects allocated or
> >> * registered on this device
> >> * @pool: shared memory pool
> >> + * @def_ctx: default context used if there is no context available, e.g. internal driver calls.
> >> */
> >> struct tee_device {
> >> char name[TEE_MAX_DEV_NAME_LEN];
> >> @@ -60,6 +61,7 @@ struct tee_device {
> >>
> >> struct idr idr;
> >> struct tee_shm_pool *pool;
> >> + struct tee_context def_ctx;
> >> };
> >>
> >> /**
> >> @@ -309,6 +311,19 @@ static inline bool tee_param_is_memref(struct tee_param *param)
> >> */
> >> struct tee_context *teedev_open(struct tee_device *teedev);
> >>
> >> +/**
> >> + * teedev_get_def_context() - Get default context for a struct tee_device
> >> + * @teedev: Device to open
> >> + *
> >> + * Unlike a context that returned from teedev_open(), the default context is static
> >> + * and available as long as @teedev has a user ''other then this context''. This context
> >> + * can be used for driver internal operation and clean up where a context should be
> >> + * available, while tee_device_unregister() is waiting for other users to go away.
> >> + *
> >> + * @return a pointer to struct tee_context on success or an ERR_PTR on failure.
> >> + */
> >> +struct tee_context *teedev_get_def_context(struct tee_device *teedev);
> >> +
> >> /**
> >> * teedev_close_context() - closes a struct tee_context
> >> * @ctx: The struct tee_context to close
> >> diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
> >> index 1b57cddfecc8..9633e14ba484 100644
> >> --- a/include/linux/tee_drv.h
> >> +++ b/include/linux/tee_drv.h
> >> @@ -7,7 +7,6 @@
> >> #define __TEE_DRV_H
> >>
> >> #include <linux/device.h>
> >> -#include <linux/kref.h>
> >> #include <linux/list.h>
> >> #include <linux/mod_devicetable.h>
> >> #include <linux/tee.h>
> >> @@ -25,10 +24,6 @@ struct tee_device;
> >> * @teedev: pointer to this drivers struct tee_device
> >> * @list_shm: List of shared memory object owned by this context
> >> * @data: driver specific context data, managed by the driver
> >> - * @refcount: reference counter for this structure
> >> - * @releasing: flag that indicates if context is being released right now.
> >> - * It is needed to break circular dependency on context during
> >> - * shared memory release.
> >> * @supp_nowait: flag that indicates that requests in this context should not
> >> * wait for tee-supplicant daemon to be started if not present
> >> * and just return with an error code. It is needed for requests
> >> @@ -41,8 +36,6 @@ struct tee_context {
> >> struct tee_device *teedev;
> >> struct list_head list_shm;
> >> void *data;
> >> - struct kref refcount;
> >> - bool releasing;
> >> bool supp_nowait;
> >> bool cap_memref_null;
> >> };
> >>
> >> --
> >> 2.34.1
> >>
