Jakub Kicinski wrote: [..] > I don't begrudge anyone building proprietary options, but leave > upstream out of it. So I am of 2 minds here. In general, how is upstream benefited by requiring every vendor command to be wrapped by a Linux command? Mind you, I am coming at this from the perspective of being a maintainer of a subsystem that does *not* allow unrestricted vendor commands. Since day one, the CXL subsystem has matched netdev's general sentiment and been more restrictive than NVMe. It places all vendor commands and even all yet-to-be-Linux-wrapped-standard-commands behind a CONFIG_CXL_MEM_RAW_COMMANDS option. That default-off option, when enabled, allows any command to be sent but it taints the kernel with a WARN(). CXL devices theoretically allow direct manipulation of system memory without IOMMU protection which is in contrast to NVMe which would need to work harder to violate kernel-lockdown protections. The expectation that I laid out here [1] is based on the observation that a significant portion of the vendor commands these devices support are for pre-release hardware qualification and debug flows. The recommendation to device vendors was "if you need wide distribution of kernels that allow unrestricted vendor passthrough, work with Linux distributions to enable this option in debug kernels, run those debug kernels for your pre-release hardware flows, ignore the warnings". 3 years on from that recommendation it seems no vendor has even needed that level of distribution help. I.e. checking a few distro kernels (Fedora, openSUSE) shows no uptake for CONFIG_CXL_MEM_RAW_COMMANDS=y in their debug builds. I can only assume that locally compiled custom kernel binaries are filling the need. So all seems quiet with current restriction for CXL endpoint vendor commands, but this stance was recently challenged in this thread [2] by CXL switch vendors with an assertion that fabric switch configuration has need for more and varied vendor flows than endpoint configuration. While I am not clear on the veracity of that claim, it at least challenged me to do the thought experiment of "what would it look like to relax the CXL command restriction?". Maybe we can come up with a community answer to the "so you want to build a userpace-to-device-firmware tunnel?" to at least get all the various concerns documented in one place, and provide guidance for how device vendors should navigate this space across subsystems. Between NVMe "allow all the things", CXL "allow all the things only after tainting the kernel", and the "never allow vendor passthrough" position (I am sure there are other nuanced positions) it at least seems useful to document the concerns. Here is a start for that guidance from the CXL perspective: * Integrity: Subsystem has a responsibility to meet kernel-lockdown expectations: Distros and system owners need to be assured that root's ability to modify the running kernel image are mitigated. For CXL there are 2 ways to do this, require Linux wrapper commands for all the low level commands (status quo), or a new trust the device to publish which commands have user data effects in something CXL calls the "Command Effects Log". In that "trust Command Effects" scenario the kernel still has no idea what the command is actually doing, but it can at least assert that the device does not claim that the command changes the contents of system-memory. Now, you might say, "the device can just lie", but that betrays a conceit of the kernel restriction. A device could lie that a Linux wrapped command when passed certain payloads does not in turn proxy to a restricted command. So at some point there is almost always an out-of-tree way to get around the kernel restriction, so the question is are we better off giving a blessed path or force vendors into ugly out-of-tree workarounds? * Introspection / validation: Subsystem community needs to be able to audit behavior after the fact. To me this means even if the kernel is letting a command through based on the stated Command Effect of "Configuration Change after Cold Reset" upstream community has a need to be able to read the vendor specification for that command. I.e. commands might be vendor-specific, but never vendor-private. I see this as similar to the requirement for open source userspace for sophisticated accelerators. * Collaboration: open standards support open driver maintenance. Without standards we end up with awkward situations like Confidential Computing where every vendor races to implement the same functionality in arbitrarily different and vendor specific ways. For CXL devices, and I believe the devices fwctl is targeting, there are a whole class of commands for vendor specific configuration and debug. Commands that the kernel really need not worry about. Some subsystems may want to allow high-performance science experiments like what NVMe allows, but it seems worth asking the question if standardizing device configuration and debug is really the best use of upstream's limited time? One of the release valves in the CXL space is openly specified commands with opaque payloads, like "Read Vendor Debug Log". That is clear what it does, likely a payload the kernel need never worry about, and the "Command Effects" is empty. However, going forward there is a new class of commands called "Set/Get Feature" that allow a wide range of vendor toggles to be deployed which will need an upstream response for the driver policy to vendor-specific "Features". So if fwctl, or something like it, can strike a balance of enforcing integrity and introspection while encouraging collaboration on the aspects that are worth upstream collaboration, I think that is a conversation worth having. [1]: http://lore.kernel.org/r/CAPcyv4gDShAYih5iWabKg_eTHhuHm54vEAei8ZkcmHnPp3B0cw@xxxxxxxxxxxxxx/ [2]: http://lore.kernel.org/r/20240321174423.00007e0d@xxxxxxxxxx
