Hello, I would like to initiate a discussion related to an unified infrastructure for ML workloads and user-space drivers. [PROBLEM STATEMENT] Last several years have revealed two important trends: (1) moving kernel-space functionality into user-space drivers (for example, SPDK, DPDK, ublk); (2) significant number of efforts of using ML models for various real life applications (for example, tuning kernel parameters, storage device failure prediction, fail slow drive detection, and so on). Both trends represent significant importance for the evolution of the Linux kernel. From one point of view, user-space drivers represent the way of decreasing the latency and improving the performance of operations. However, from another point of view, the approach of bypassing the Linux kernel introduces security and efficiency risks, potential synchronization issues of user-space threads, and breaking the Linux kernel architecture’s paradigm. Generally speaking, direct implementation of ML approaches in Linux kernel-space is very hard, inefficient, and problematic because of practical unavailability of floating point operations in the Linux kernel, and the computational power hungry nature of ML algorithms (especially, during training phase). It is possible to state that Linux kernel needs to introduce and to unify an infrastructure as for ML approaches as for user-space drivers. [WHY DO WE NEED ML in LINUX KERNEL?] Do we really need a ML infrastructure in the Linux kernel? First of all, it is really easy to imagine a lot of down to earth applications of ML algorithms for automation of routine operations during working with Linux kernel. Moreover, potentially, the ML subsystem could be used for automated research and statistics gathering on the whole fleet of running Linux kernels. Also, the ML subsystem is capable of writing documentation, tuning kernel parameters on the fly, kernel recompilation, and even automated reporting about bugs and crashes. Generally speaking, the ML subsystem potentially can extend the Linux kernel capabilities. The main question is how? [POTENTIAL INFRASTRUCTURE VISION] Technically speaking , both cases (user-space driver and ML subsystem) require a user-space functionality that can be considered as user-space extension of Linux kernel functionality. Such approach is similar to microkernel architecture by the minimal functionality on kernel side and the main functionality on user-space side with the mandatory minimization the number of context switches between kernel-space and user-space. The key responsibility of kernel-side agent (or subsystem) is the accounting of user-space extensions, synchronization of their access to shared resources or metadata on kernel side, statistics gathering and sharing it through the sysfs or specialized log file (likewise to syslog). For example, such specialized log file(s) can be used by ML user-space extensions for executing the ML algorithms with the goal of analyzing data and available statistics. Generally speaking, the main ML logic can be executed by extension(s) on the user-space side. This ML logic can elaborate some “recommendations”, for example, that can be shared with an ML agent on the kernel side. As a result, the kernel-space ML agent can check the shared “recommendations” and to apply the valid “recommendations” by means of Linux kernel tuning, recompilation, “hot” restart and so on. Technically speaking, the user-space driver requires pretty much the same architecture as the simple kernel-space agent/subsystem and user-space extension(s). The main functionality is on the user-space side and kernel-space side delivers only accounting the user-space extensions, allocating necessary resources, synchronizing access to shared resources, and gathering statistics. Generally speaking, such an approach implies the necessity of registering a specialized driver class that could represent an ML agent or user-space driver on kernel side. Then, it will be possible to use a modprobe-like model to create an instance of ML agent or user-space driver. Finally, we will have the kernel-space agent that is connected to the user-space extension. The key point here is that the user-space extension can directly communicate with a hardware device, but the kernel-space side can account for the activity of the user-space extension and allocates resources. It is possible to suggest an unified architecture of the kernel-side agent that will be specialized by the logic of user-space extension. But the logic of the kernel-space agent should be minimal, simple, and unified as much as possible. Technically speaking, the logic of kernel-space agent can be defined by the eBPF program and eBPF arena (or shared memory between kernel-space and user-space) can be used for interaction between the kernel-space agent and the user-space extension. And such interaction could be implemented through submission and completion queues, for example. As a summary, described architecture is capable of implementing ML infrastructure in Linux kernel and unification of user-space drivers architecture. Any opinion on this? How feasible could be such vision? Thanks, Slava.
