This patch series introduces the CXL region driver as well as associated APIs in CXL core. The region driver enables the creation of "regions" which is a concept defined by the CXL 2.0 specification [1]. Region verification and programming state are owned by the cxl_region driver (implemented in the cxl_region module). It relies on cxl_mem to determine if devices are CXL routed, and cxl_port to actually handle the programming of the HDM decoders. Much of the region driver is an implementation of algorithms described in the CXL Type 3 Memory Device Software Guide [2]. The region driver will be responsible for configuring regions found on persistent capacities in the Label Storage Area (LSA), it will also enumerate regions configured by BIOS, usually volatile capacities, and will allow for dynamic region creation (which can then be stored in the LSA). It is the primary consumer of the CXL Port [3] and CXL Mem drivers introduced previously [4]. Dan as reworked some of this which is required for this branch. A cached copy is included in the gitlab for this project [5]. Those patches will be posted shortly. The patches for the region driver could be squashed. They're broken out to aid review and because that's the order they were implemented in. My preference is to keep those as they are. Some things are still missing and will be worked on while these are reviewed (in priority order): 1. Connection to libnvdimm labels (No plan) 2. Volatile regions/LSA regions (Have a plan) 3. Switch ports (Have a plan) 4. Decoder programming restrictions (No plan). The one know restriction I've missed is to disallow programming HDM decoders that aren't in incremental system physical address ranges. 5. Stress testing Here is an example of output when programming a x2 interleave region: ./cxlctl create-region -i2 -n -a -s $((512<<20)) /sys/bus/cxl/devices/decoder0.0 [ 42.971496][ T644] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: -19 [ 42.972400][ T644] cxl_core:cxl_add_region:478: cxl region0.0:0: Added region0.0:0 to decoder0.0 [ 42.979388][ T644] cxl_core:cxl_commit_decoder:394: cxl_port port1: decoder1.0 [ 42.979388][ T644] Base 0x0000004c00000000 [ 42.979388][ T644] Size 536870912 [ 42.979388][ T644] IG 8 [ 42.979388][ T644] IW 2 [ 42.979388][ T644] TargetList: 0 1 -1 -1 -1 -1 -1 -1 [ 42.982410][ T644] cxl_core:cxl_commit_decoder:394: cxl_port endpoint3: decoder3.0 [ 42.982410][ T644] Base 0x0000004c00000000 [ 42.982410][ T644] Size 536870912 [ 42.982410][ T644] IG 8 [ 42.982410][ T644] IW 2 [ 42.982410][ T644] TargetList: -1 -1 -1 -1 -1 -1 -1 -1 [ 42.985427][ T644] cxl_core:cxl_commit_decoder:394: cxl_port endpoint2: decoder2.0 [ 42.985427][ T644] Base 0x0000004c00000000 [ 42.985427][ T644] Size 536870912 [ 42.985427][ T644] IG 8 [ 42.985427][ T644] IW 2 [ 42.985427][ T644] TargetList: -1 -1 -1 -1 -1 -1 -1 -1 [ 42.987937][ T644] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: 0 If you're wondering how I tested this, I've baked it into my cxlctl app [6] and lib [7]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl. To get the detailed errors, trace-cmd can be utilized. Until a region device exists, the region module will not be loaded, which means the region tracepoints will not exist. To get around this, modprobe cxl_region before anything. trace-cmd record -e cxl ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0 Branch can be found at gitlab [8]. --- [1]: https://www.computeexpresslink.org/download-the-specification [2]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide [3]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-9-ben.widawsky@xxxxxxxxx/ [4]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-17-ben.widawsky@xxxxxxxxx/ [5]: https://gitlab.com/bwidawsk/linux/-/commit/126793e22427f7975a8f2fca373764be78012e88 [6]: https://gitlab.com/bwidawsk-cxl/cxlctl [7]: https://gitlab.com/bwidawsk-cxl/cxl_rs [8]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_region --- Ben Widawsky (13): cxl/core: Rename find_cxl_port cxl/core: Track port depth cxl/region: Add region creation ABI cxl/region: Introduce concept of region configuration cxl/mem: Cache port created by the mem dev cxl/region: Introduce a cxl_region driver cxl/acpi: Handle address space allocation cxl/region: Address space allocation cxl/region: Implement XHB verification cxl/region: HB port config verification cxl/region: Add infrastructure for decoder programming cxl/region: Record host bridge target list cxl: Program decoders for regions .clang-format | 3 + Documentation/ABI/testing/sysfs-bus-cxl | 63 ++ .../driver-api/cxl/memory-devices.rst | 14 + drivers/cxl/Makefile | 2 + drivers/cxl/acpi.c | 30 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 4 + drivers/cxl/core/hdm.c | 198 +++++ drivers/cxl/core/port.c | 132 +++- drivers/cxl/core/region.c | 525 +++++++++++++ drivers/cxl/cxl.h | 32 + drivers/cxl/cxlmem.h | 9 + drivers/cxl/mem.c | 16 +- drivers/cxl/port.c | 42 +- drivers/cxl/region.c | 695 ++++++++++++++++++ drivers/cxl/region.h | 47 ++ drivers/cxl/trace.h | 54 ++ tools/testing/cxl/Kbuild | 1 + 18 files changed, 1849 insertions(+), 19 deletions(-) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.c create mode 100644 drivers/cxl/region.h create mode 100644 drivers/cxl/trace.h base-commit: 03716ce2db3c17ba38f26a88d75049c0472a629e prerequisite-patch-id: af6a0315e22bfc1099d4f58610b8b897e6e5a060 -- 2.34.1
