On Tue, 2020-12-08 at 16:24 -0800, Ben Widawsky wrote: > Changes since v1 [1] > > A few additions have been made: > - IOCTL (UAPI) interface has been added with commands > - Kernel docs have been created > - A new debug macro is introduced and sprinkled throughout. > > A deletion was made: > - Removal of the non-standard _OSC UUID. > > The detailed list of fixes is: > - fix cxl_register() no previous prototype warning (0day robot) > - s/REGLOG/REGLOC/ (Ben) > - Wait for doorbell on cxl_mem_mbox_get() and add comment on why (Ben) > - make "type-3" a proper adjective, add spec references, also did the same for > the Kconfig (Bjorn) > - align some defines (Bjorn) > - s/bar/BAR (Bjorn) > - rename cxl_bus_prepared() to cxl_bus_acquire() (Bjorn) > - move definition of struct cxl_mem to "cxl/mem: Map memory device registers" (Bjorn) > - use consistent hex/decimal (Bjorn) > - use consistent upper/lower hex values (Bjorn) > - get rid of READ_ONCE (Bjorn) > - add offsets to debug messages (Bjorn) > - cleanup SPDX comment style (Bjorn, Christoph) > - change errors returned by case (Bjorn, Dan) > - 80 character violation cleanups (Christoph) > - cleanup CXL_BUS_PROVIDER dependencies (Christoph, Randy) > - remove "raw" from mmio functions (Dan) > - rename PCI_DVSEC_VENDOR_CXL to add _ID (Jonathan) > - combine introduction of mbox infrastruct and cxl_mem_identify() (Jonathan) > - add ABI documentation for sysfs attributes (Jonathan) > - document scope of cxl_memdev_lock (Jonathan) > - rework cxl_register() to have devm semantics (reaction to comments about > cxl_mem_remove() and cxl_mem_add_memdev() semantics) (Jonathan) > - fix cxl_mem_exit() ordering (Jonathan) > - use GENMASK/GET_FIELD (Jonathan) > - fix and add comments for cap ids (Jonathan) > - use _OFFSET postfix in definitions (Jonathan) > - save pci_set_drvdata for later (Jonathan) There are a few more change credits for the acpi patches: - Remove unnecessary helpers and callbacks in drivers/cxl/acpi.c (Christoph, Jonathan) - Remove some unnecessary variable initalizations (Bjorn, Jonathan) - Convert to platform_driver (Rafael) > > [1]: https://lore.kernel.org/linux-cxl/20201111054356.793390-1-ben.widawsky@xxxxxxxxx/ > > --- > > Introduce support for “type-3” memory devices defined in the recently released > Compute Express Link (CXL) 2.0 specification[2]. Specifically, these are the > memory devices defined by section 8.2.8.5 of the CXL 2.0 spec. A reference > implementation emulating these devices has been submitted to the QEMU mailing > list and is available on gitlab [3]. “Type-3” is a CXL device that acts as a > memory expander for RAM or PMEM. It might be interleaved with other CXL devices > in a given physical address range. > > These changes allow for foundational enumeration of CXL 2.0 memory devices as > well as basic userspace interaction. The functionality present is: > - Initial driver bring-up > - Device enumeration and an initial sysfs representation > - Submit a basic firmware command via ‘mailbox’ to an emulated memory device > with non-volatile capacity. > - Provide an interface to send "raw" commands to the hardware. > > Some of the functionality that is still missing includes: > - Memory interleaving at the host bridge, root port, or switch level > - CXL 1.1 Root Complex Integrated Endpoint Support > - CXL 2.0 Hot plug support > - A bevy of supported device commands > > In addition to the core functionality of discovering the spec defined registers > and resources, introduce a CXL device model that will be the foundation for > translating CXL capabilities into existing Linux infrastructure for Persistent > Memory and other memory devices. For now, this only includes support for the > management command mailbox that type-3 devices surface. These control devices > fill the role of “DIMMs” / nmemX memory-devices in LIBNVDIMM terms. > > Now, while implementing the driver some feedback for the specification was > generated to cover perceived gaps and address conflicts. The feedback is > presented as a reference implementation in the driver and QEMU emulation. > Specifically the following concepts are original to the Linux implementation and > feedback / collaboration is requested to develop these into specification > proposals: > 1. Top level ACPI object (ACPI0017) > 2. HW imposed address space and interleave constraints > > ACPI0017 > -------- > Introduce a new ACPI namespace device with an _HID of ACPI0017. The purpose of > this object is twofold, support a legacy OS with a set of out-of-tree CXL > modules, and establish an attach point for a driver that knows about > interleaving. Both of these boil down to the same point, to centralize Operating > System support for resources described by the CXL Early Discovery Table (CEDT). > > The legacy OS problem stems from the spec's description of a host bridge, > ACPI0016 is denoted as the _HID for host bridges, with a _CID of PNP0A08. In a > CXL unaware version of Linux, the core ACPI subsystem will bind a driver to > PNP0A08 and preclude a CXL-aware driver from binding to ACPI0016. An ACPI0017 > device allows a standalone CXL-aware driver to register for handling / > coordinating CEDT and CXL-specific _OSC control. > > Similarly when managing interleaving there needs to be some management layer > above the ACPI0016 device that is capable of assembling leaf nodes into > interleave sets. As is the case with ACPI0012 that does this central > coordination for NFIT defined resources, ACPI0017 does the same for CEDT > described resources. > > Memory Windows > ------- > For CXL.mem capable platforms, there is a need for a mechanism for platform > firmware to make the Operating System aware of any restrictions that hardware > might have in address space. For example, in a system with 4 host bridges all > participating in an interleave set, the firmware needs to provide some > description of this. That information is missing from the CXL 2.0 spec as of > today and it also is not implemented in the driver. A variety of ACPI based > mechanisms, for example _CRS fields on the ACPI0017 device, were considered. > > Next steps after this basic foundation is expanded command support and LIBNVDIMM > integration. This is the initial “release early / release often” version of the > Linux CXL enabling. > > [2]: https://www.computeexpresslink.org/ > [3]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v2 > > > Ben Widawsky (10): > docs: cxl: Add basic documentation > cxl/mem: Map memory device registers > cxl/mem: Find device capabilities > cxl/mem: Implement polled mode mailbox > cxl/mem: Add basic IOCTL interface > cxl/mem: Add send command > cxl/mem: Add a "RAW" send command > cxl: Add basic debugging > MAINTAINERS: Add maintainers of the CXL driver > WIP/cxl/mem: Add get firmware for testing > > Dan Williams (2): > cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints > cxl/mem: Register CXL memX devices > > Vishal Verma (2): > cxl/acpi: Add an acpi_cxl module for the CXL interconnect > cxl/acpi: add OSC support > > Documentation/ABI/testing/sysfs-bus-cxl | 26 + > Documentation/cxl/index.rst | 12 + > Documentation/cxl/memory-devices.rst | 51 ++ > Documentation/index.rst | 1 + > MAINTAINERS | 10 + > drivers/Kconfig | 1 + > drivers/Makefile | 1 + > drivers/cxl/Kconfig | 58 ++ > drivers/cxl/Makefile | 9 + > drivers/cxl/acpi.c | 352 ++++++++ > drivers/cxl/acpi.h | 35 + > drivers/cxl/bus.c | 54 ++ > drivers/cxl/bus.h | 8 + > drivers/cxl/cxl.h | 188 +++++ > drivers/cxl/mem.c | 1022 +++++++++++++++++++++++ > drivers/cxl/pci.h | 34 + > include/acpi/actbl1.h | 51 ++ > include/uapi/linux/cxl_mem.h | 148 ++++ > 18 files changed, 2061 insertions(+) > create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl > create mode 100644 Documentation/cxl/index.rst > create mode 100644 Documentation/cxl/memory-devices.rst > create mode 100644 drivers/cxl/Kconfig > create mode 100644 drivers/cxl/Makefile > create mode 100644 drivers/cxl/acpi.c > create mode 100644 drivers/cxl/acpi.h > create mode 100644 drivers/cxl/bus.c > create mode 100644 drivers/cxl/bus.h > create mode 100644 drivers/cxl/cxl.h > create mode 100644 drivers/cxl/mem.c > create mode 100644 drivers/cxl/pci.h > create mode 100644 include/uapi/linux/cxl_mem.h >
