On Thu, Jan 02, 2025 at 11:30:11AM +0100, Danilo Krummrich wrote:
> On Tue, Dec 24, 2024 at 09:53:23PM +0000, Gary Guo wrote:
> > On Thu, 19 Dec 2024 18:04:10 +0100
> > Danilo Krummrich <dakr@xxxxxxxxxx> wrote:
> >
> > > Add a Rust abstraction for the kernel's devres (device resource
> > > management) implementation.
> > >
> > > The Devres type acts as a container to manage the lifetime and
> > > accessibility of device bound resources. Therefore it registers a
> > > devres callback and revokes access to the resource on invocation.
> > >
> > > Users of the Devres abstraction can simply free the corresponding
> > > resources in their Drop implementation, which is invoked when either the
> > > Devres instance goes out of scope or the devres callback leads to the
> > > resource being revoked, which implies a call to drop_in_place().
> > >
> > > Signed-off-by: Danilo Krummrich <dakr@xxxxxxxxxx>
> > > ---
> > > MAINTAINERS | 1 +
> > > rust/helpers/device.c | 10 +++
> > > rust/helpers/helpers.c | 1 +
> > > rust/kernel/devres.rs | 178 +++++++++++++++++++++++++++++++++++++++++
> > > rust/kernel/lib.rs | 1 +
> > > 5 files changed, 191 insertions(+)
> > > create mode 100644 rust/helpers/device.c
> > > create mode 100644 rust/kernel/devres.rs
> > >
> > > <snip>
> > >
> > > +pub struct Devres<T>(Arc<DevresInner<T>>);
> > > +
> > > +impl<T> DevresInner<T> {
> > > + fn new(dev: &Device, data: T, flags: Flags) -> Result<Arc<DevresInner<T>>> {
> > > + let inner = Arc::pin_init(
> > > + pin_init!( DevresInner {
> > > + data <- Revocable::new(data),
> > > + }),
> > > + flags,
> > > + )?;
> > > +
> > > + // Convert `Arc<DevresInner>` into a raw pointer and make devres own this reference until
> > > + // `Self::devres_callback` is called.
> > > + let data = inner.clone().into_raw();
> > > +
> > > + // SAFETY: `devm_add_action` guarantees to call `Self::devres_callback` once `dev` is
> > > + // detached.
> > > + let ret = unsafe {
> > > + bindings::devm_add_action(dev.as_raw(), Some(Self::devres_callback), data as _)
> > > + };
> > > +
> > > + if ret != 0 {
> > > + // SAFETY: We just created another reference to `inner` in order to pass it to
> > > + // `bindings::devm_add_action`. If `bindings::devm_add_action` fails, we have to drop
> > > + // this reference accordingly.
> > > + let _ = unsafe { Arc::from_raw(data) };
> > > + return Err(Error::from_errno(ret));
> > > + }
> > > +
> > > + Ok(inner)
> > > + }
> > > +
> > > + #[allow(clippy::missing_safety_doc)]
> > > + unsafe extern "C" fn devres_callback(ptr: *mut kernel::ffi::c_void) {
> > > + let ptr = ptr as *mut DevresInner<T>;
> > > + // Devres owned this memory; now that we received the callback, drop the `Arc` and hence the
> > > + // reference.
> > > + // SAFETY: Safe, since we leaked an `Arc` reference to devm_add_action() in
> > > + // `DevresInner::new`.
> > > + let inner = unsafe { Arc::from_raw(ptr) };
> > > +
> > > + inner.data.revoke();
> > > + }
> > > +}
> > > +
> > > +impl<T> Devres<T> {
> > > + /// Creates a new [`Devres`] instance of the given `data`. The `data` encapsulated within the
> > > + /// returned `Devres` instance' `data` will be revoked once the device is detached.
> > > + pub fn new(dev: &Device, data: T, flags: Flags) -> Result<Self> {
> > > + let inner = DevresInner::new(dev, data, flags)?;
> > > +
> > > + Ok(Devres(inner))
> > > + }
> > > +
> > > + /// Same as [`Devres::new`], but does not return a `Devres` instance. Instead the given `data`
> > > + /// is owned by devres and will be revoked / dropped, once the device is detached.
> > > + pub fn new_foreign_owned(dev: &Device, data: T, flags: Flags) -> Result {
> > > + let _ = DevresInner::new(dev, data, flags)?;
> > > +
> > > + Ok(())
> > > + }
> > > +}
> > > +
> > > +impl<T> Deref for Devres<T> {
> > > + type Target = Revocable<T>;
> > > +
> > > + fn deref(&self) -> &Self::Target {
> > > + &self.0.data
> > > + }
> > > +}
> > > +
> > > +impl<T> Drop for Devres<T> {
> > > + fn drop(&mut self) {
> > > + // Revoke the data, such that it gets dropped already and the actual resource is freed.
> > > + //
> > > + // `DevresInner` has to stay alive until the devres callback has been called. This is
> > > + // necessary since we don't know when `Devres` is dropped and calling
> > > + // `devm_remove_action()` instead could race with `devres_release_all()`.
> >
> > IIUC, the outcome of that race is the `WARN` if
> > devres_release_all takes the spinlock first and has already remvoed the
> > action?
>
> Yes, this was one issue. But I think there was another when you have a class
> `Registration` that is owned by a `Devres`, which holds private data that is
> encapsulated in a `Devres` too.
>
> I have this case in Nova where the DRM device' private data holds the PCI bar,
> and the DRM device registration has a reference of the corresponding DRM device.
>
> But maybe this also was something else. I will double check and if I can confirm
> that the WARN_ON() in devm_remove_action() is the only issue, we can certainly
> change this.
Ok, I double checked and this should indeed be the only issue.
I can reproduce the race and can confirm that a devm_remove_action_nowarn()
works, as long as we make it return the integer that indicates whether the
action has been removed already, such that we can handle it from the Rust side
accordingly.
Generally, I don't think it's a big deal, drivers usually hold the resources
they request anyways until remove(). But it's still an improvement, so I'll send
a patch for that.
>
> >
> > Could you do a custom devres_release here that mimick
> > `devm_remove_action` but omit the `WARN`? This way it allows the memory
> > behind DevresInner to be freed early without keeping it allocated until
> > the end of device lifetime.
> >
> > > + //
> > > + // SAFETY: When `drop` runs, it's guaranteed that nobody is accessing the revocable data
> > > + // anymore, hence it is safe not to wait for the grace period to finish.
> > > + unsafe { self.revoke_nosync() };
> > > + }
> > > +}
> > > diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
> > > index 6c836ab73771..2b61bf99d1ee 100644
> > > --- a/rust/kernel/lib.rs
> > > +++ b/rust/kernel/lib.rs
> > > @@ -41,6 +41,7 @@
> > > pub mod cred;
> > > pub mod device;
> > > pub mod device_id;
> > > +pub mod devres;
> > > pub mod driver;
> > > pub mod error;
> > > #[cfg(CONFIG_RUST_FW_LOADER_ABSTRACTIONS)]
> >
