On 22-02-01 15:49:41, Jonathan Cameron wrote:
> On Tue, 1 Feb 2022 07:31:54 -0800
> Ben Widawsky <ben.widawsky@xxxxxxxxx> wrote:
>
> > On 22-01-23 16:31:24, Dan Williams wrote:
> > > While CXL memory targets will have their own memory target node,
> > > individual memory devices may be affinitized like other PCI devices.
> > > Emit that attribute for memdevs.
> > >
> > > Signed-off-by: Dan Williams <dan.j.williams@xxxxxxxxx>
> >
> > This brings up an interesting question. Are all devices in a region affinitized
> > to the same NUMA node? I think they must be - at which point, should this
> > attribute be a part of a region, rather than a device?
>
> No particular reason why they should be in the same NUMA node
> in general. People occasionally do memory interleave across memory
> controllers on different CPU sockets (in extreme cases).
> Whilst, at the interleave set level, that will have a single numa
> domain, the individual devices making it up could be all over
> the place and it will depend on the configuration.
There's no such thing as a non-interleave set though. Everything is a region. A
x1 region is a region with one device.
>
> >
> > > ---
> > > Documentation/ABI/testing/sysfs-bus-cxl | 9 +++++++++
> > > drivers/cxl/core/memdev.c | 17 +++++++++++++++++
> > > tools/testing/cxl/test/cxl.c | 1 +
> > > 3 files changed, 27 insertions(+)
> > >
> > > diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl
> > > index 87c0e5e65322..0b51cfec0c66 100644
> > > --- a/Documentation/ABI/testing/sysfs-bus-cxl
> > > +++ b/Documentation/ABI/testing/sysfs-bus-cxl
> > > @@ -34,6 +34,15 @@ Description:
> > > capability. Mandatory for CXL devices, see CXL 2.0 8.1.12.2
> > > Memory Device PCIe Capabilities and Extended Capabilities.
> > >
> > > +What: /sys/bus/cxl/devices/memX/numa_node
> > > +Date: January, 2022
> > > +KernelVersion: v5.18
> > > +Contact: linux-cxl@xxxxxxxxxxxxxxx
> > > +Description:
> > > + (RO) If NUMA is enabled and the platform has affinitized the
> > > + host PCI device for this memory device, emit the CPU node
> > > + affinity for this device.
> > > +
> >
> > I think you'd want to say something about the device actively decoding. Perhaps
> > I'm mistaken though, can you affinitize without setting up HDM decoders for the
> > device?
>
> It's possible for PCI devices (up to a bug I should dig out the fix for)
> to be placed in their own NUMA domains, or gain them from the root ports / host
> bridges. The magic of generic initiators and fiddly ACPI DSDT files that
> the bios might want to create.
>
> >
> > > What: /sys/bus/cxl/devices/*/devtype
> > > Date: June, 2021
> > > KernelVersion: v5.14
> > > diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
> > > index 1e574b052583..b2773664e407 100644
> > > --- a/drivers/cxl/core/memdev.c
> > > +++ b/drivers/cxl/core/memdev.c
> > > @@ -99,11 +99,19 @@ static ssize_t serial_show(struct device *dev, struct device_attribute *attr,
> > > }
> > > static DEVICE_ATTR_RO(serial);
> > >
> > > +static ssize_t numa_node_show(struct device *dev, struct device_attribute *attr,
> > > + char *buf)
> > > +{
> > > + return sprintf(buf, "%d\n", dev_to_node(dev));
> > > +}
> > > +static DEVICE_ATTR_RO(numa_node);
> > > +
> > > static struct attribute *cxl_memdev_attributes[] = {
> > > &dev_attr_serial.attr,
> > > &dev_attr_firmware_version.attr,
> > > &dev_attr_payload_max.attr,
> > > &dev_attr_label_storage_size.attr,
> > > + &dev_attr_numa_node.attr,
> > > NULL,
> > > };
> > >
> > > @@ -117,8 +125,17 @@ static struct attribute *cxl_memdev_ram_attributes[] = {
> > > NULL,
> > > };
> > >
> > > +static umode_t cxl_memdev_visible(struct kobject *kobj, struct attribute *a,
> > > + int n)
> > > +{
> > > + if (!IS_ENABLED(CONFIG_NUMA) && a == &dev_attr_numa_node.attr)
> > > + return 0;
> > > + return a->mode;
> > > +}
> > > +
> > > static struct attribute_group cxl_memdev_attribute_group = {
> > > .attrs = cxl_memdev_attributes,
> > > + .is_visible = cxl_memdev_visible,
> > > };
> > >
> > > static struct attribute_group cxl_memdev_ram_attribute_group = {
> > > diff --git a/tools/testing/cxl/test/cxl.c b/tools/testing/cxl/test/cxl.c
> > > index 40ed567952e6..cd2f20f2707f 100644
> > > --- a/tools/testing/cxl/test/cxl.c
> > > +++ b/tools/testing/cxl/test/cxl.c
> > > @@ -583,6 +583,7 @@ static __init int cxl_test_init(void)
> > > if (!pdev)
> > > goto err_mem;
> > > pdev->dev.parent = &port->dev;
> > > + set_dev_node(&pdev->dev, i % 2);
> > >
> > > rc = platform_device_add(pdev);
> > > if (rc) {
> > >
>
