Hi Dave, On Fri, Jan 17, 2025 at 08:18:37AM +1100, Dave Chinner wrote: > On Thu, Jan 16, 2025 at 07:49:49AM -0500, Theodore Ts'o wrote: > > Historically, we have avoided adding tracepoints to the VFS because of > > concerns that tracepoints would be considered a userspace-level > > interface, and would therefore potentially constrain our ability to > > improve an interface which has been extremely performance critical. > > Yes, the lack of tracepoints in the VFS is a fairly significant > issue when it comes to runtime debugging of production systems... > > > I'd like to discuss whether in 2025, it's time to reconsider our > > reticence in adding tracepoints in the VFS layer. First, while there > > has been a single incident of a tracepoint being used by programs that > > were distributed far and wide (powertop) such that we had to revert a > > change to a tracepoint that broke it --- that was ***14** years ago, > > in 2011. > > Yes, that was a big mistake in multiple ways. Firstly, the app using > a tracepoint in this way. The second mistake was the response that > "tracepoints should be stable API" based on the abuse of a single > tracepoint. > > We had extensive tracepoint coverage in subsystems *before* this > happened. In XFS, we had already converted hundreds of existing > debug-build-only tracing calls to use tracepoints based on the > understanding that tracepoints were *not* considered stable user > interfaces. > > The fact that existing subsystem tracepoints already exposed the > internal implementation of objects like struct inode, struct file, > superblocks, etc simply wasn't considered when tracepoints were > declared "stable". > > The fact is that it is simply not possible to maintain any sort of > useful introspection with the tracepoint infrastructure without > exposing internal implementation details that can change from kernel > to kernel. > > > Across multiple other subsystems, many of > > which have added an extensive number of tracepoints, there has been > > only a single problem in over a decade, so I'd like to suggest that > > this concern may have not have been as serious as we had first > > thought. > > Yes, these subsystems still operate under the "tracepoints are not > stable" understanding. The reality is that userspace has *never* > been able to rely on tracepoints being stable across multiple kernel > releases, regardless of what anyone else (including Linus) says is > the policy. As a (relatively) long time bpftrace developer, I've always been fairly consistent with users new to linux tracing that tracepoints are _not_ guaranteed to be stable and they exist on the stability spectrum somewhere between kprobes/fentry and uapi. IIRC from the cases I've seen where tracepoints shift, users just adjust their scripts. I don't remember having seen anyone both think that it's the kernel's fault and then go complain on list. I'm happy to adjust any of bpftrace's public facing docs to make that reality more clear if it'll help. > > > I'd like to propose that we experiment with adding tracepoints in > > early 2025, so that at the end of the year the year-end 2025 LTS > > kernels will have tracepoints that we are confident will be fit for > > purpose for BPF users. > > Why does BPF even need tracepoints? BPF code should be using kprobes > to hook into the running kernel to monitor it, yes? In addition to the points Andrii makes below, tracepoints also have a nice documenting property. They tend to get added to "places of interest". They're a great starting point for non kernel developers to dig into kernel internals. Often times tracepoint naming (as well as the exported fields) provide helpful hints. At least for me, if I'm mucking around new places (mostly net/) I'll tend to go look at the tracepoints to find the interesting codepaths. [..] Thanks, Daniel
