On Tue, 1 Feb 2022 07:31:54 -0800
Ben Widawsky <ben.widawsky@xxxxxxxxx> wrote:
> On 22-01-23 16:31:24, Dan Williams wrote:
> > While CXL memory targets will have their own memory target node,
> > individual memory devices may be affinitized like other PCI devices.
> > Emit that attribute for memdevs.
> >
> > Signed-off-by: Dan Williams <dan.j.williams@xxxxxxxxx>
>
> This brings up an interesting question. Are all devices in a region affinitized
> to the same NUMA node? I think they must be - at which point, should this
> attribute be a part of a region, rather than a device?
No particular reason why they should be in the same NUMA node
in general. People occasionally do memory interleave across memory
controllers on different CPU sockets (in extreme cases).
Whilst, at the interleave set level, that will have a single numa
domain, the individual devices making it up could be all over
the place and it will depend on the configuration.
>
> > ---
> > Documentation/ABI/testing/sysfs-bus-cxl | 9 +++++++++
> > drivers/cxl/core/memdev.c | 17 +++++++++++++++++
> > tools/testing/cxl/test/cxl.c | 1 +
> > 3 files changed, 27 insertions(+)
> >
> > diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl
> > index 87c0e5e65322..0b51cfec0c66 100644
> > --- a/Documentation/ABI/testing/sysfs-bus-cxl
> > +++ b/Documentation/ABI/testing/sysfs-bus-cxl
> > @@ -34,6 +34,15 @@ Description:
> > capability. Mandatory for CXL devices, see CXL 2.0 8.1.12.2
> > Memory Device PCIe Capabilities and Extended Capabilities.
> >
> > +What: /sys/bus/cxl/devices/memX/numa_node
> > +Date: January, 2022
> > +KernelVersion: v5.18
> > +Contact: linux-cxl@xxxxxxxxxxxxxxx
> > +Description:
> > + (RO) If NUMA is enabled and the platform has affinitized the
> > + host PCI device for this memory device, emit the CPU node
> > + affinity for this device.
> > +
>
> I think you'd want to say something about the device actively decoding. Perhaps
> I'm mistaken though, can you affinitize without setting up HDM decoders for the
> device?
It's possible for PCI devices (up to a bug I should dig out the fix for)
to be placed in their own NUMA domains, or gain them from the root ports / host
bridges. The magic of generic initiators and fiddly ACPI DSDT files that
the bios might want to create.
>
> > What: /sys/bus/cxl/devices/*/devtype
> > Date: June, 2021
> > KernelVersion: v5.14
> > diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c
> > index 1e574b052583..b2773664e407 100644
> > --- a/drivers/cxl/core/memdev.c
> > +++ b/drivers/cxl/core/memdev.c
> > @@ -99,11 +99,19 @@ static ssize_t serial_show(struct device *dev, struct device_attribute *attr,
> > }
> > static DEVICE_ATTR_RO(serial);
> >
> > +static ssize_t numa_node_show(struct device *dev, struct device_attribute *attr,
> > + char *buf)
> > +{
> > + return sprintf(buf, "%d\n", dev_to_node(dev));
> > +}
> > +static DEVICE_ATTR_RO(numa_node);
> > +
> > static struct attribute *cxl_memdev_attributes[] = {
> > &dev_attr_serial.attr,
> > &dev_attr_firmware_version.attr,
> > &dev_attr_payload_max.attr,
> > &dev_attr_label_storage_size.attr,
> > + &dev_attr_numa_node.attr,
> > NULL,
> > };
> >
> > @@ -117,8 +125,17 @@ static struct attribute *cxl_memdev_ram_attributes[] = {
> > NULL,
> > };
> >
> > +static umode_t cxl_memdev_visible(struct kobject *kobj, struct attribute *a,
> > + int n)
> > +{
> > + if (!IS_ENABLED(CONFIG_NUMA) && a == &dev_attr_numa_node.attr)
> > + return 0;
> > + return a->mode;
> > +}
> > +
> > static struct attribute_group cxl_memdev_attribute_group = {
> > .attrs = cxl_memdev_attributes,
> > + .is_visible = cxl_memdev_visible,
> > };
> >
> > static struct attribute_group cxl_memdev_ram_attribute_group = {
> > diff --git a/tools/testing/cxl/test/cxl.c b/tools/testing/cxl/test/cxl.c
> > index 40ed567952e6..cd2f20f2707f 100644
> > --- a/tools/testing/cxl/test/cxl.c
> > +++ b/tools/testing/cxl/test/cxl.c
> > @@ -583,6 +583,7 @@ static __init int cxl_test_init(void)
> > if (!pdev)
> > goto err_mem;
> > pdev->dev.parent = &port->dev;
> > + set_dev_node(&pdev->dev, i % 2);
> >
> > rc = platform_device_add(pdev);
> > if (rc) {
> >
