Dear gentlemen, thanks again for your suggestions. I've given a fleeting try to both systemtap and lttng, in Debian Jessie using the pre-built = packaged versions of said probe tools. In both cases I pretty soon ended up at some error message, effectively instructing me to solve a kernel module version mismatch or tweak some .config entry in the running kernel to get the kernel-space tracer modules loaded. Which I could do, but don't really have enough time or motivation right now - maybe next time :-) Or perhaps I'm jusy lazy. Basically... these two tools need a kernel "helper", need debugging symbol information (quite obviously, to print function args) and are "out of tree", so they need some additional manual care, syncing of versions and whatnot - not a big surprise. Also, the fact that they need "debugging symbols" is a deterrent for me to use them on a customer's production system (perhaps as a last resort). The related quick follow-up reading has pointed me to some really cool tools I didn't know about yet, such as the Trace Compass. Wow, this could come in handy. :-) initially I thought "well there appears to be some kernel-space tracing framework, I used to read about it now and then at LWN" and upon a deeper dive I realize, that these are actually maybe *four* such frameworks or tools, living side by side, or sharing some common instrumentation snippets at the back end... That's Linux :-) History in the making. Never ending evolution. I'm getting off topic here. Essentially I'm writing this message to somehow "close this thread", although I don't yet have a resolution of the particular problem I'm facing. In reality it won't be me hunting the problem down on the live system, I'm trying to instruct someone "behind the scenes" :-) And although this is already way off topic in this "tracing" mailing list, I'm attaching a couple of snippets that I collected on the broader topic or task at hand, which is trying to pinpoint some anomaly in system timebase steering, looking almost like a leap second bug, or some such, happening semi-randomly at midnight on the edge of a new month. I don't want to be more specific - there are likely historical versions of Linux and NTPd in play, possibly some custom cron jobs in the OS, there is definitely proprietary software running in the system. I'm attaching my "snippets" "for the record" - for the benefit of people googling about this stuff in the future, be it tracing-related or timing-related. After some homework, I'm able to 1) trace events in the kernel 2) strace ntpd in the user space (as a key suspect) 3) increase the "log level" in ntpd, and 4) capture NTP traffic on Ethernet using tcpdump, all at the same time, which should make it easier to correlate the traces. I've added an hourly cron job to insert a "real time" timestamp into the kernel trace via 'date >trace_marker' and I've instructed strace (-tt) to include realtime time stamps too... Ultimately, as the kernel event trace and the user-space strace both log the same syscall occurrences from ntpd, they areeffectively a mirror copy of each other, which makes it easier to fix the kernel trace against wall time (and to look for discrepancies in the flow of the system wall time, compared to the monotonic clock used in the kernel trace). Thanks for your polite attention and for all your help :-) Have a nice weekend... Frank Rysanek
Attachment:
WPM$XT4E.PM$
Description: Mail message body
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: NTP_logging_howto.txt
Date: 28 Apr 2017, 16:12
Size: 2870 bytes.
Type: Text
Attachment:
NTP_logging_howto.txt
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: NTP_logging_internals.c
Date: 27 Apr 2017, 17:16
Size: 4632 bytes.
Type: Program-source
Attachment:
NTP_logging_internals.c
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: stop_tracing.sh
Date: 27 Apr 2017, 16:47
Size: 720 bytes.
Type: Unknown
Attachment:
stop_tracing.sh
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: strace-ntpd.sh
Date: 28 Apr 2017, 15:22
Size: 418 bytes.
Type: Unknown
Attachment:
strace-ntpd.sh
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: tcpdump_combo_startup.sh
Date: 27 Apr 2017, 18:21
Size: 410 bytes.
Type: Unknown
Attachment:
tcpdump_combo_startup.sh
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: trace_events.sh
Date: 27 Apr 2017, 16:42
Size: 955 bytes.
Type: Unknown
Attachment:
trace_events.sh
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: trace_functions.sh
Date: 27 Apr 2017, 16:48
Size: 766 bytes.
Type: Unknown
Attachment:
trace_functions.sh
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
---- File information -----------
File: watch_trace_output.sh
Date: 27 Apr 2017, 16:27
Size: 513 bytes.
Type: Unknown
Attachment:
watch_trace_output.sh
Description: Binary data
|