If it ain't broke? How much more evidence of being broke do we need?
The bug here is that the process is insufficiently robust under operator error.
That is broke.
The underlying problem here is the lack of auditability in the process.
There is a simple fix here, eliminate the dependency on the list ordering and the system does not have such a critical dependence on the operator.
Again nobody is claiming anything dishonest has happened here. The concern is that the accident could be repeated on purpose in the future to exclude undesirable candidates. Having spent part of last month watching this attempted in Alabama it is a real concern.
When something is broke admit the fact. Prattling on about not fixing what aint broke only makes people angry.
Sent from my GoodLink Wireless Handheld (www.good.com)
-----Original Message-----
From: John C Klensin [mailto:john-ietf@xxxxxxx]
Sent: Saturday, September 02, 2006 05:56 AM Pacific Standard Time
To: Ned Freed; Eastlake III Donald-LDE008
Cc: IETF-Discussion
Subject: RE: Now there seems to be lack of communicaiton here...
--On Friday, 01 September, 2006 15:07 -0700 Ned Freed
<ned.freed@xxxxxxxxxxx> wrote:
>> See below at @@@
>
>> -----Original Message-----
>> From: John C Klensin [mailto:john-ietf@xxxxxxx]
>> Sent: Friday, September 01, 2006 11:45 AM
>> To: Eastlake III Donald-LDE008; IETF-Discussion
>> Subject: RE: Now there seems to be lack of communicaiton
>> here...
>
>> --On Thursday, 31 August, 2006 17:30 -0400 Eastlake III
>> Donald-LDE008 <Donald.Eastlake@xxxxxxxxxxxx> wrote:
>
>> > John,
>> >
>> > If the selection method is random, it makes no difference
>> > whatsoever how the list of nomcom volunteers is ordered. It
>> > only matters that the
>...
>> @@@ I'm not sure why you agree with me and then say the
>> opposite. If it doesn't matter what the ordering of the
>> nomcom volunteer pool is then it doesn't matter what the
>> ordering of the nomcom volunteer pool is, and, in particular,
>> it doesn't matter how random or "biased" it is. The RFC 3797
>> algorithm takes random inputs and uses them to make successive
>> uniformly distributed non-repeating selections from a range
>> of integers. The sole purpose of the published ordered
>> volunteer pool list is to provide a pre-announced mapping
>> from those integers to the people who volunteered. If it
>> mattered what that mapping was, then you are not making
>> uniformly distributed random selections.
>
> John, I have to agree with Donald here. Nothing anyone has
> said has indicated that there is a flaw in the algorithmic
> part of the selection process. If something ain't broke don't
> fix it.
In my original note to Brian, I tried to ask a question
involving a statistical nicety. It has to do with preserving
the original randomness given a second draw from a rather small
population (the number of volunteers). As you and Donald
certainly know at least as well as I do, the key question about
pseudo-random number generators and processes is whether they
are good enough, not whether they produce randomness by all
possible measures. I agree (but unfortunately did not say so
explicitly) that the procedures defined in 3797 is more than
adequate if followed exactly. I had and have some doubts about
some of the variations 3797 permits if someone performs a
partial reset (as distinct from a full reset, which would
involve reopening the call for volunteers and maybe the Nomcom
Chair appointment).
However, I do not believe now, and have never believed, that
quibbles about statistics and randomness are the issue here. I
simply wanted to understand Brian's reasoning (and the reasoning
of anyone else who recommended re-drawing the Nomcom composition
after eliminating one name).
I see two, related, issues at the core of the actual problem
here:
(1) Whatever the reason, essentially eliminating the time
between when the volunteer pool was announced and the time the
Nomcom membership list was determined took a series of checks
out of the system, both with regard to eliminating individuals
who were not eligible and with regard permitting people who
volunteered and had been dropped to identify and question that
decision (or find lost email, etc.). We will never know
whether the numbers of the latter were large enough to have an
impact on the selection model, and that lack of knowledge is a
threat to the perceived integrity of the system.
(2) Because the notion of a reset after the Nomcom membership
list is selected permits someone, in principle, to examine the
membership list and say "no, I don't like that one, let's do it
over", a reset is an extremely bad idea if there is any possible
other alternative. To the extent to which a potential
candidate in this round, or anyone with very strong opinions
about a potential candidate, have an opportunity to examine the
first-selected list and then provide substantive input into
whether a reset should be done, we move beyond "extremely bad"
into something worse. I assume that everything have been done
with good faith here but, to paraphrase an off-list discussion,
at least some of the community will always have a nagging fear
that the reset was motivated by some members of the community
believing that a second draw would yield a more favorable Nomcom.
> Indeed, not only is the algorithmic part of the selection
> process fine, the non-algorithmic parts were well thought out
> and various contingencies were covered, including the specific
> one where disqualified people found their way onto the final
> qualified candidate list. The problem is quite simply that the
> nomcom chair failed to follow these recommendations.
yes
> It also appears we have dodged the bullet this time around.
> But this was mostly a matter of luck. We may not be so lucky
> the next time, and that means we need to act now to fix the
> problem that the nomcom chair has the dangerous and
> unnecessary ability to reset the selection process in cases
> where no reset should be allowed. (I have already described
> why there is real danger here - see my previous posting for
> specifics.) This absolutely must be attended to before the
> next nomcom.
I think we are in violent agreement.
> Now, various people have argued that the nomcom chair was
> within the rules to reset the process when the error came to
> light. I agree with this assessment, but this is really beside
> the point. The point is the nomcom chair should NOT have the
> reset option available in this circumstance, and that our
> current process is BROKEN in allowing this level of discretion.
>...
> I'm sorry, but I'm not so forgiving. The nomcom chair made a
> major blunder here, abetted by some exceptionally poor advice
> from the IETF chair. This blunder could have caused major
> damage to the organization. The combination of there being
> essentially no time before the second selection process and the
> fairly drastic nature of an ISOC appeal is the only - repeat
> ONLY - reason I for one didn't start the ball rolling.
>
> I've seen no indication that there's been any direct fallout
> from this, such as a lawsuit being filed. (Of course something
> along these lines could still happen.) But this does not mean
> that what has happened hasn't weakened the faith people have
> in the fairness of the process. I know for a fact that I'm
> much less sanguine about the process than I used to be, and I
> suspect I'm not alone in this.
Agreed. See above. My level of willingness to forgive was
motivated by the situation you described as having dodged the
bullet and because the Nomcom chair clearly has this discretion
under the current rules, not because I'm happy with the
situation.
> I also share your discomfort with the nomcom chair's decision
> to consult the IETF chair, although my discomfort falls short
> of wanting to see some formal rule against it happening.
Given this demonstration, I think some formal guidance that it
is inappropriate for the nomcom chair to seek procedural advice
from anyone who might be expected to be a candidate for
selection or re-selection. But I have no idea how to write a
formal rule that would work, even if it were desirable... and
I'm not convinced that it would be desirable.
>> But, if we are going to make sure this problem does not occur
>> in the future, I think we should make the procedure as
>> gaming-proof as possible. That, to me, implies two
>> requirements going forward:
>
>> (1) We get strong randomization of the selection process
>
>> @@@ We already have this. See RFC 3797.
>
> Yep. Again, if it ain't broke...
I do not disagree. Just pointing out that it needs to be, and
continue to be, part of the process.
>...
>> In addition, I am extremely concerned by hints on this list
>> that the Secretariat's checking procedures ruled people
>> ineligible to volunteer who had, in fact, attended the
>> correct number of meetings. That, it seems to me, is a much
>> larger threat to the integrity of the Nomcom model and
>> perceptions of trust in it than any issue that impacts a
>> single volunteer or even, within broad limits, the
>> randomization and Nomcom member selection processes.
>
>> @@@ Well, you are welcome to be as concerned as you like, but
>> we live in an imperfect world. As far as I know, there are
>> usually a few disputes about the volunteer list, usually in
>> connection with people whose eligibility the Secretariat
>> doubts. Sometimes they are determined to be correct and
>> sometimes the Secretariat is determined to be right.
>> Sometimes the volunteer is confused about the eligibility
>> requirements or about what their attendance actually was, or
>> has variant versions of their name, or has changed their
>> name, or ... But this sort of thing doesn't effect very many
>> people on the volunteer list and is almost always resolved
>> between the volunteer and the Secretariat before the list is
>> published.
>
> Nevertheless, I believe John is correct in saying that even if
> we eliminate the unnecessary and dangerous discretionary
> powers the nomcom chair has currently, there's still an issue
> when the list we use is subsequently found not to contain
> people it should. However, there doesn't appear to be a good
> way to address this problem should it arise. If someone has a
> novel suggestion for how to deal with this I'd love to hear
> it, but absent such a suggestion I'm at a loss as to how to
> deal with this one other than to say "yes, it's a potential
> issue, hopefully it won't happen".
Actually, there is a perfectly good way to address this issue,
at least at the 95% level. Suppose the list of volunteers were
made public significantly in advance of the selection of Nomcom
members so that people have the opportunity to detect and
protest inappropriate volunteers and so that people who are
dropped have an opportunity to protest. Then these things can
be straightened out before the Nomcom is selected, eliminating
later second-guessing and debates about what actions various
problems should cause. Hmm. That is already specified in
3797: we don't need to fix anything, just follow our own,
existing, rules.
john
_______________________________________________
Ietf@xxxxxxxx
https://www1.ietf.org/mailman/listinfo/ietf
_______________________________________________ Ietf@xxxxxxxx https://www1.ietf.org/mailman/listinfo/ietf