On Tue, 9 Feb 2021 16:02:54 -0800
Ben Widawsky <ben.widawsky@xxxxxxxxx> wrote:
> From: Dan Williams <dan.j.williams@xxxxxxxxx>
>
> Create the /sys/bus/cxl hierarchy to enumerate:
>
> * Memory Devices (per-endpoint control devices)
>
> * Memory Address Space Devices (platform address ranges with
> interleaving, performance, and persistence attributes)
>
> * Memory Regions (active provisioned memory from an address space device
> that is in use as System RAM or delegated to libnvdimm as Persistent
> Memory regions).
>
> For now, only the per-endpoint control devices are registered on the
> 'cxl' bus. However, going forward it will provide a mechanism to
> coordinate cross-device interleave.
>
> Signed-off-by: Dan Williams <dan.j.williams@xxxxxxxxx>
> Signed-off-by: Ben Widawsky <ben.widawsky@xxxxxxxxx>
One stray header, and a request for a tiny bit of reordering to
make it easier to chase through creation and destruction.
Either way with the header move to earlier patch I'm fine with this one.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx>
> ---
> Documentation/ABI/testing/sysfs-bus-cxl | 26 ++
> .../driver-api/cxl/memory-devices.rst | 17 +
> drivers/cxl/Makefile | 3 +
> drivers/cxl/bus.c | 29 ++
> drivers/cxl/cxl.h | 4 +
> drivers/cxl/mem.c | 301 +++++++++++++++++-
> 6 files changed, 378 insertions(+), 2 deletions(-)
> create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
> create mode 100644 drivers/cxl/bus.c
>
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 745f5e0bfce3..b3c56fa6e126 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -3,6 +3,7 @@
>
> #ifndef __CXL_H__
> #define __CXL_H__
> +#include <linux/range.h>
Why is this coming in now? Feels like it should have been in earlier
patch that started using struct range
>
> #include <linux/bitfield.h>
> #include <linux/bitops.h>
> @@ -55,6 +56,7 @@
> (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \
> CXLMDEV_RESET_NEEDED_NOT)
>
> +struct cxl_memdev;
> /**
> * struct cxl_mem - A CXL memory device
> * @pdev: The PCI device associated with this CXL device.
> @@ -72,6 +74,7 @@
> struct cxl_mem {
> struct pci_dev *pdev;
> void __iomem *regs;
> + struct cxl_memdev *cxlmd;
>
> void __iomem *status_regs;
> void __iomem *mbox_regs;
> @@ -90,4 +93,5 @@ struct cxl_mem {
> } ram;
> };
>
> +extern struct bus_type cxl_bus_type;
> #endif /* __CXL_H__ */
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index 0a868a15badc..8bbd2495e237 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -1,11 +1,36 @@
>
> +
> +static void cxl_memdev_release(struct device *dev)
> +{
> + struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
> +
> + percpu_ref_exit(&cxlmd->ops_active);
> + ida_free(&cxl_memdev_ida, cxlmd->id);
> + kfree(cxlmd);
> +}
> +
...
> +static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
> +{
> + struct pci_dev *pdev = cxlm->pdev;
> + struct cxl_memdev *cxlmd;
> + struct device *dev;
> + struct cdev *cdev;
> + int rc;
> +
> + cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
> + if (!cxlmd)
> + return -ENOMEM;
> + init_completion(&cxlmd->ops_dead);
> +
> + /*
> + * @cxlm is deallocated when the driver unbinds so operations
> + * that are using it need to hold a live reference.
> + */
> + cxlmd->cxlm = cxlm;
> + rc = percpu_ref_init(&cxlmd->ops_active, cxlmdev_ops_active_release, 0,
> + GFP_KERNEL);
> + if (rc)
> + goto err_ref;
> +
> + rc = ida_alloc_range(&cxl_memdev_ida, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL);
> + if (rc < 0)
> + goto err_id;
> + cxlmd->id = rc;
> +
> + dev = &cxlmd->dev;
> + device_initialize(dev);
> + dev->parent = &pdev->dev;
> + dev->bus = &cxl_bus_type;
> + dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
> + dev->type = &cxl_memdev_type;
> + dev_set_name(dev, "mem%d", cxlmd->id);
> +
> + cdev = &cxlmd->cdev;
> + cdev_init(cdev, &cxl_memdev_fops);
> +
> + rc = cdev_device_add(cdev, dev);
> + if (rc)
> + goto err_add;
> +
> + return devm_add_action_or_reset(dev->parent, cxlmdev_unregister, cxlmd);
This had me scratching my head. The cxlmdev_unregister() if called normally
or in the _or_reset() results in
percpu_ref_kill(&cxlmd->ops_active);
cdev_device_del(&cxlmd->cdev, dev);
wait_for_completion(&cxlmd->ops_dead);
cxlmd->cxlm = NULL;
put_device(dev);
/* If last ref this will result in */
percpu_ref_exit(&cxlmd->ops_active);
ida_free(&cxl_memdev_ida, cxlmd->id);
kfree(cxlmd);
So it's doing all the correct things but not necessarily
in the obvious order.
For simplicity of review perhaps it's worth reordering probe a bit
to get the ida immediately after the cxlmd alloc and
for the cxlmdev_unregister() perhaps reorder the cdev_device_del()
before the percpu_ref_kill().
Trivial obvious as the ordering has no affect but makes it
easy for reviewers to tick off setup vs tear down parts.
> +
> +err_add:
> + ida_free(&cxl_memdev_ida, cxlmd->id);
> +err_id:
> + /*
> + * Theoretically userspace could have already entered the fops,
> + * so flush ops_active.
> + */
> + percpu_ref_kill(&cxlmd->ops_active);
> + wait_for_completion(&cxlmd->ops_dead);
> + percpu_ref_exit(&cxlmd->ops_active);
> +err_ref:
> + kfree(cxlmd);
> +
> + return rc;
> +}
> +
