On 3/6/25 11:28 AM, Jeff Layton wrote: > On Thu, 2025-03-06 at 09:29 -0500, Chuck Lever wrote: >> On 3/6/25 7:38 AM, Jeff Layton wrote: >>> Sargun set up kprobes to add some of these tracepoints. Convert them to >>> simple static tracepoints. These are pretty sparse for now, but they >>> could be expanded in the future as needed. >> >> I have mixed feelings about this. To be very clear: I'm always up for better observability! The details of this patch are where I start to have some hesitation. >> - Probably tracepoints should replace the existing dprintk call sites. >> dprintk is kind of useless for heavy traffic. >> > > I'm fine with removing dprintks as we go. Removing them was controversial a few years ago when I first brought this up... I would very much like to see these call sites gone, even if we don't have immediate replacements in the form of trace points. >> - Seems like other existing tracepoints could report most of the same >> information. fh_verify, for example, has a tracepoint that reports >> the file handle. There's an svc proc tracepoint, and an NFSv4 COMPOUND >> tracepoint that can report XID and procedure. >> > > The problem there is the lack of context. Yes, I can see that > fh_verify() got called, but on a busy server it can be hard to tell why > it got called. I see things like the fh_verify() tracepoint working in > conjunction with these new tracepoints. IOW, you could match up the > xids and see which fh_verify() was called for which operation. If we're talking about NFSv3 only, sunrpc:svc_process records the XID, nfsd thread, NFSv3 procedure name, and NFSD namespace of each incoming RPC call. You also get the NFS client's IP address. You can also enable nfsd:nfsd_fh_verify to capture several of those items, plus the NFS file handle. The kernel process information will be identical for both the svc_proc and nfsd_fh_verify trace points -- that will tie the two records together so you can match an XID to an NFS procedure and its file handle argument. If you want to see a little more you can enable the function_graph plug-in for nfsd_dispatch(). Another approach is adding trace points in the XDR layer to capture all of the arguments of incoming RPC calls: https://web.git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=topic-xdr-tracepoints -- Chuck Lever
