Hi all. I found some last minute bugs which I've fixed. Please ignore this series and wait until I send out v3. Thanks. Ben On 22-01-06 16:37:43, Ben Widawsky wrote: > This patch series introduces the CXL region driver as well as associated APIs in > CXL core. The region driver enables the creation of "regions" which is a concept > defined by the CXL 2.0 specification [1]. Region verification and programming > state are owned by the cxl_region driver (implemented in the cxl_region module). > It relies on cxl_mem to determine if devices are CXL routed, and cxl_port to > actually handle the programming of the HDM decoders. Much of the region driver > is an implementation of algorithms described in the CXL Type 3 Memory Device > Software Guide [2]. > > The region driver will be responsible for configuring regions found on > persistent capacities in the Label Storage Area (LSA), it will also enumerate > regions configured by BIOS, usually volatile capacities, and will allow for > dynamic region creation (which can then be stored in the LSA). It is the primary > consumer of the CXL Port [3] and CXL Mem drivers introduced previously [4]. Dan > as reworked some of this which is required for this branch. A cached copy is > included in the gitlab for this project [5]. Those patches will be posted > shortly. > > The patches for the region driver could be squashed. They're broken out to aid > review and because that's the order they were implemented in. My preference is > to keep those as they are. > > Some things are still missing and will be worked on while these are reviewed (in > priority order): > 1. Connection to libnvdimm labels (No plan) > 2. Volatile regions/LSA regions (Have a plan) > 3. Switch ports (Have a plan) > 4. Decoder programming restrictions (No plan). The one know restriction I've > missed is to disallow programming HDM decoders that aren't in incremental > system physical address ranges. > 5. Stress testing > > Here is an example of output when programming a x2 interleave region: > ./cxlctl create-region -i2 -n -a -s $((512<<20)) /sys/bus/cxl/devices/decoder0.0 > [ 42.971496][ T644] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: -19 > [ 42.972400][ T644] cxl_core:cxl_add_region:478: cxl region0.0:0: Added region0.0:0 to decoder0.0 > [ 42.979388][ T644] cxl_core:cxl_commit_decoder:394: cxl_port port1: decoder1.0 > [ 42.979388][ T644] Base 0x0000004c00000000 > [ 42.979388][ T644] Size 536870912 > [ 42.979388][ T644] IG 8 > [ 42.979388][ T644] IW 2 > [ 42.979388][ T644] TargetList: 0 1 -1 -1 -1 -1 -1 -1 > [ 42.982410][ T644] cxl_core:cxl_commit_decoder:394: cxl_port endpoint3: decoder3.0 > [ 42.982410][ T644] Base 0x0000004c00000000 > [ 42.982410][ T644] Size 536870912 > [ 42.982410][ T644] IG 8 > [ 42.982410][ T644] IW 2 > [ 42.982410][ T644] TargetList: -1 -1 -1 -1 -1 -1 -1 -1 > [ 42.985427][ T644] cxl_core:cxl_commit_decoder:394: cxl_port endpoint2: decoder2.0 > [ 42.985427][ T644] Base 0x0000004c00000000 > [ 42.985427][ T644] Size 536870912 > [ 42.985427][ T644] IG 8 > [ 42.985427][ T644] IW 2 > [ 42.985427][ T644] TargetList: -1 -1 -1 -1 -1 -1 -1 -1 > [ 42.987937][ T644] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: 0 > > If you're wondering how I tested this, I've baked it into my cxlctl app [6] and > lib [7]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl. > > To get the detailed errors, trace-cmd can be utilized. Until a region device > exists, the region module will not be loaded, which means the region tracepoints > will not exist. To get around this, modprobe cxl_region before anything. > > trace-cmd record -e cxl ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0 > > Branch can be found at gitlab [8]. > > --- > > [1]: https://www.computeexpresslink.org/download-the-specification > [2]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide > [3]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-9-ben.widawsky@xxxxxxxxx/ > [4]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-17-ben.widawsky@xxxxxxxxx/ > [5]: https://gitlab.com/bwidawsk/linux/-/commit/126793e22427f7975a8f2fca373764be78012e88 > [6]: https://gitlab.com/bwidawsk-cxl/cxlctl > [7]: https://gitlab.com/bwidawsk-cxl/cxl_rs > [8]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_region > > --- > > Ben Widawsky (13): > cxl/core: Rename find_cxl_port > cxl/core: Track port depth > cxl/region: Add region creation ABI > cxl/region: Introduce concept of region configuration > cxl/mem: Cache port created by the mem dev > cxl/region: Introduce a cxl_region driver > cxl/acpi: Handle address space allocation > cxl/region: Address space allocation > cxl/region: Implement XHB verification > cxl/region: HB port config verification > cxl/region: Add infrastructure for decoder programming > cxl/region: Record host bridge target list > cxl: Program decoders for regions > > .clang-format | 3 + > Documentation/ABI/testing/sysfs-bus-cxl | 63 ++ > .../driver-api/cxl/memory-devices.rst | 14 + > drivers/cxl/Makefile | 2 + > drivers/cxl/acpi.c | 30 + > drivers/cxl/core/Makefile | 1 + > drivers/cxl/core/core.h | 4 + > drivers/cxl/core/hdm.c | 198 +++++ > drivers/cxl/core/port.c | 132 +++- > drivers/cxl/core/region.c | 525 +++++++++++++ > drivers/cxl/cxl.h | 32 + > drivers/cxl/cxlmem.h | 9 + > drivers/cxl/mem.c | 16 +- > drivers/cxl/port.c | 42 +- > drivers/cxl/region.c | 695 ++++++++++++++++++ > drivers/cxl/region.h | 47 ++ > drivers/cxl/trace.h | 54 ++ > tools/testing/cxl/Kbuild | 1 + > 18 files changed, 1849 insertions(+), 19 deletions(-) > create mode 100644 drivers/cxl/core/region.c > create mode 100644 drivers/cxl/region.c > create mode 100644 drivers/cxl/region.h > create mode 100644 drivers/cxl/trace.h > > > base-commit: 03716ce2db3c17ba38f26a88d75049c0472a629e > prerequisite-patch-id: af6a0315e22bfc1099d4f58610b8b897e6e5a060 > -- > 2.34.1 >
