John Summerfield said the following on 06/04/2008 09:04 PM Pacific Time:
Jon Stanley wrote:
On Wed, Jun 4, 2008 at 9:27 PM, John Summerfield
<debian@xxxxxxxxxxxxxxxxxxxxxx> wrote:
I'd rather John spoke for himself.
Well with Will and I being John's cohorts in crime, I think we can
speak to what each of us wants. First, we want to measure the impact
that we're having in the project. This is just good sense, how do you
know that you're doing something that's worthwhile if you have no
metrics to back up your position? The second is, as Will said,
recognition and reward for top triagers. Triage is a thankless job,
so we need some way to recognize the rock stars of triage, and reward
them appropriately.
I've been in IT since it was called ADP/EDP, and remember punched cards,
paper tape and typewriter keyboards on large computers.
On occasion I've been hired to write programs, and one of the questions
much in the mind of managers is how to measure productivity.
A measure in one place I worked was "lines of code." Programmers writing
more lines of code was held to be more productive than those writing fewer.
This without any regard for complexity, or code quality (another thing
that's hard to quantify), or consumption of resources (such as _then_
very expensive computer time).
A number of LOC I've seen quoted is 100 per day, for a basically capable
programmer on not too-difficult code, and including documentation (to
standards I don't recall, but undoubtedly better than that which one
ordinarily sees in OS code).
At number of LOC was found to be fairly constant over a variety of
programming languages (but I suspect C wasn't included in the survey),
so writing in higher-level languages such as COBOL and PL/1, where
suitable, had obvious benefits in business applications than assembler,
and interpretive languages such as REXX (which probably didn't exist
then) produced even greater productivity benefits.
Now, writing computer programs is just one example of problem solving;
writing (books, stories etc) and triaging are other examples.
I don't think metrics that are easily measured are very useful in
measuring productivity.
If Will triaged 100 bugs in May, that's (I expect) quite good. However,
if 20 of them are misassigned or otherwise badly handled, I think I
would rather he handled 50 bugs and got 48 of them right.
A little while ago I filed a bug report against kernels 2.6.25, my
DC7700 wouldn't boot because the kernel couldn't find its disks.
The mkinitrd team said, "That's ours," without (I think, but I might be
defaming them unfairly) looking very closely at it. Had I left it at
that, it might still be on their list, but I went to some effort to
prove it wasn't mkinitrd and then changed the assignment.
Misassignment can be quite expensive, in terms of getting problems fixed.
I think that the idea of acknowledging good performance is excellent,
but that its implementation needs to be done carefully.
To achieve an award, I suggest these need to be considered. Change the
numbers to suit, I have no idea what absolute numbers are realistic.
1. Activity. People need to be more than casual workers. Maybe disposing
of 20 bugs in a month. If you don't do at least this amount of work, you
won't be recognised[1].
2. Accuracy. Allow maybe 5% error rate: one mistake in 20. If you don't
measure up on this, you are not eligible for recognition (but you might
attract some attention form a mentor).
3. Relative performance. Not the absolute best performance, this isn't
the Olympics, but compared with the worker's previous efforts. If Jon's
doing more, or making fewer mistakes than before, acknowledge that.
4. Knowing when to ask for advice. Jo says, "I've done this and that,
and I think .., what do you say?" This is way better than mishandling
the problem, and makes it into a learning experience. OTOH if Jo's only
had a quick look and done no analysis of the problem, then asking for
help is a distraction to others. Better to pass on that one, and go for
the next.
5. Mentoring. Jeff's been around for a while, maybe doesn't do much
triaging himself these days, but he's always ready to help people like
Jo. And he keeps an eye on newcomers, and those prone to making mistakes.
[1] Unless you're new. New workers should be publicly welcomed.
I might be using "mentor" a little loosely; those more experienced
should be expected to help others (and package maintainers probably
should be mentoring too). I don't mean a formal mentoring role such as
Debian has (but it might be a good idea).
I can see that some simple metrics might help, but be careful they arent
used too mechanistically.
John Speaks :)
Big picture right now is that we have no way to provide positive
feedback to those doing bug triage. On its face many would consider bug
triage to be a "miserable job"... I'm looking for ways to mitigate that:
http://poelcat.wordpress.com/2008/02/23/miserable/
I hope that providing information to the individuals and the wider
community can help alleviate Anonymity, Irrelevance, and Immeasurement.
Each person is motivated by different things. For me the biggest
detractor for getting involved in bug triage was not how my performance
would be measured, but that what I did wouldn't matter--thus for me I
want to charts of overall performance and how the efforts of the team
are paying off. Others might be competitive and want to see how they
rank against others. There are many motivators too.
My approach to these metrics is positive motivation, not issuing
accurate performance reviews to be considered for year-end compensation
increases :)
Thanks for sharing your ideas cautions about using metrics. The big
difference between your example of misguided metrics and our project
here is that there is NO "management". Management is all of us and what
WE decide we want things to be.
Here is the hard part, but the part that will help all of this
most--what can we tangibly start doing today to recognize the efforts of
bug triagers and chart our overall progress? I'm not talking about
t-shirts, candy bars and USB keys. Yes, we may have those from time to
time, but that is like trying to motivate employees soley by
money--which most studies have shown isn't a long term sustainable
motivator.
I believe to really scale we need to be able to collect motivational
information about our individual and group progress in an automated way
that can be reported in an automated way. So while I agree with your
ideas, a helpful and much needed next step would be to outline exactly
how we can practically start doing them in Fedora.
Thanks,
John
--
fedora-test-list mailing list
fedora-test-list@xxxxxxxxxx
To unsubscribe:
https://www.redhat.com/mailman/listinfo/fedora-test-list
