On Thu, Jan 16, 2025 at 07:49:49AM -0500, Theodore Ts'o wrote: > Historically, we have avoided adding tracepoints to the VFS because of > concerns that tracepoints would be considered a userspace-level > interface, and would therefore potentially constrain our ability to > improve an interface which has been extremely performance critical. Yes, the lack of tracepoints in the VFS is a fairly significant issue when it comes to runtime debugging of production systems... > I'd like to discuss whether in 2025, it's time to reconsider our > reticence in adding tracepoints in the VFS layer. First, while there > has been a single incident of a tracepoint being used by programs that > were distributed far and wide (powertop) such that we had to revert a > change to a tracepoint that broke it --- that was ***14** years ago, > in 2011. Yes, that was a big mistake in multiple ways. Firstly, the app using a tracepoint in this way. The second mistake was the response that "tracepoints should be stable API" based on the abuse of a single tracepoint. We had extensive tracepoint coverage in subsystems *before* this happened. In XFS, we had already converted hundreds of existing debug-build-only tracing calls to use tracepoints based on the understanding that tracepoints were *not* considered stable user interfaces. The fact that existing subsystem tracepoints already exposed the internal implementation of objects like struct inode, struct file, superblocks, etc simply wasn't considered when tracepoints were declared "stable". The fact is that it is simply not possible to maintain any sort of useful introspection with the tracepoint infrastructure without exposing internal implementation details that can change from kernel to kernel. > Across multiple other subsystems, many of > which have added an extensive number of tracepoints, there has been > only a single problem in over a decade, so I'd like to suggest that > this concern may have not have been as serious as we had first > thought. Yes, these subsystems still operate under the "tracepoints are not stable" understanding. The reality is that userspace has *never* been able to rely on tracepoints being stable across multiple kernel releases, regardless of what anyone else (including Linus) says is the policy. > I'd like to propose that we experiment with adding tracepoints in > early 2025, so that at the end of the year the year-end 2025 LTS > kernels will have tracepoints that we are confident will be fit for > purpose for BPF users. Why does BPF even need tracepoints? BPF code should be using kprobes to hook into the running kernel to monitor it, yes? Regardless of BPF, why not just send patches to add the tracepoints you want? -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
