Thanks to David Galloway, there's a fresh version of sentry setup at
https://sentry.ceph.com. This should help us track teuthology failures
and make it easier to fix issues and keep the tests passing.
Sentry is a general tool for tracking errors, and it's hooked up to
teuthology. Whenever a job fails, teuthology reports the stacktrace and
a bunch of metadata to sentry (the metadata is customizable, so we can
add more fields easily [0]).
Sentry groups these failures ('events' in sentry terms) together into
a single 'issue', which you can search and filter by
suite/branch/os/etc.
You can login with your github account, add yourself to the ceph org,
and look around the sepia project to see all the failures from the
sepia lab. Pulpito also links to the sentry event directly for each
failure.
Each issue has a timeline of occurrences so it's easy to see if it's
recently introduced or a long-present intermittent failure. There's
also a chart of the distribution of the metadata, so you can see if
there's a common element like OS, branch, etc. This should help us
detect when an issue was introduced, and whether it is new or not.
The way sentry groups failures is also customizable - the default is
based on the stacktrace from teuthology, which is too coarse in some
cases [1]. There are likely more tweaks here that will make it easier
to use. A few ideas:
* report ceph crashes rather than the teuthology traceback when
there's a coredump
* parse test framework output and report the failed tests rather than
the command that ran them
* differentiate between different kinds of timeouts
* some automation with tracker tickets (you can link to redmine from
comments on sentry events, and vice versa today)
Check it out, and let us know if there are ways to improve it!
Josh
[0]
https://github.com/ceph/teuthology/blob/master/teuthology/run_tasks.py#L107-L144
[1] https://github.com/ceph/teuthology/pull/1593
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
