Re: Review of: draft-resnick-on-consensus-05

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Finally back to this original review.

On 10/6/13 7:03 PM, Dave Crocker wrote:
In terms of philosophy and desirable practice, the draft discusses an extremely appealing model and generally explains its nature and practice well. However the draft tends to confuse what is (or has been) with what should be. This includes confusing the IETF's historical sense of what 'rough consensus' means, with what the draft suggests it mean. As a consequence, it sometimes merely explains what is preferred, rather than the problems with what is common practice. (Sometimes. At other times, it nicely makes its case.) Also it sometimes does not lay out the context for its assertions.

In effect, the draft is offering fundamental enhancements to existing IETF practice, rather than merely clarifying the ways things are "supposed to be".

I think this is a fair criticism and I will do a grand editing pass to try to tease the two apart: Where the draft is actually describing real historical practices vs. where it is describing a theory that does not comport with our practices. I know of several places beyond what you've mentioned that make the same conflation, and I think it's well worth making it clear. There are some instances where "what is (or has been)" is actually mushy: That is, people haven't really thought about it in depth. In those cases, I've definitely done a bit of saying what I think folks would mean if they were actually asked about it. (See more on this point in the discussion of the intro.) But with all of these, I think I should be more explicit about which is which.

By way of properly setting context, the draft should cite RFC 1603/3934

You mean 2418. But yes, I think you're right.

Working Group Guidelines and Procedures. Its section 3.3 has text that describes long-standing community views on the meaning of rough consensus, which don't quite align with what the draft puts forward:

               In general, the dominant view of the
   working group shall prevail.  (However, it must be noted that
   "dominance" is not to be determined on the basis of volume or
   persistence, but rather a more general sense of agreement.) Consensus
   can be determined by a show of hands, humming, or any other means on
   which the WG agrees (by rough consensus, of course).  Note that 51%
   of the working group does not qualify as "rough consensus" and 99% is
   better than rough.  It is up to the Chair to determine if rough
   consensus has been reached.

Yup. I think this document has to cite that statement in 2418, and it has to disagree with it, at least to the extent of claiming it is an over-simplification. I had intended to do so and simply forgot. That 51%/99% statement actually leads down the path that the document wants to discourage.

The disparity between that document's text and what this draft describes should not be a problem for what the draft is recommending, but merely requires changing the explanatory tone of the draft and how it makes its case.
[...]
The changes being suggested to the draft are substantial, but will not affect the essential points for which the draft is lobbying.

I agree, though I think the changes are actually pretty straightforward. Maybe substantial in number of actual edits, but not in essential content.

Abstract

   a "majority rule-> a "simple majority rule"

Majorities come in different forms or degrees and the fact that 'rough consensus' is often taken to mean 67% or 75%, as a rule of thumb can make this confusing.

From what you've written, your basic point seems to be that 51% isn't enough; it's worth making that explicit.

Setting aside the possible confusion in British usage, I don't think you understood my intention here. And since you and Christopher and Jaap ended up in the same place, I'm happy to say that it's my fault for not being clear enough.

I intended that the abstract simply summarize the document's *purpose*, not its conclusions. I was using it like the abstracts that appear on scientific papers. Of course, the abstracts on our technical specifications usually give away the punch line, which is what we're used to seeing. I'll have a go at an edit, at least to clarify what I'm doing.

For example, "voting" can be done with models that give quite a bit of power to minorities. So, you need an explicit, direct, early statement that asserts the nature and importance of "rough consensus" factoring in minority views.

This is an abstract, not a place to put early statements. I think the kind of statement you're looking for goes in the intro, which I hopefully address below.


1.  Introduction
[...]
That is, our credo is that we don't let a single individual make the

As the draft phrases it, that isn't really true. We let individuals make decisions all the time. What I believe the aphorism is saying -- and what reflects our community belief -- is: We do not assign to any one individual the authority to /dictate/ a decision. ("make" is too soft a word for this point, IMO.)

Fair enough.

   decisions, nor do we let the majority dictate decisions, nor do we

I believe the model being pursued in this draft -- of the nature and essential importance of properly dealing with minority views -- is hugely powerful and appropriate. But I believe it has never been a prevalent view in the community, including back when the aphorism was seeking to summarize the community view.

Yes, I think this all goes to your earlier point of cleanly distinguishing between historical view and new theory.

The language I remember from the 90s primarily distinguished "strongly dominant majority" from 51% or 67% or 75%. That is, it was about imprecise nature of the required clear and strong majority, rather than the moderating role of the minority. Very occasionally -- but only occasionally -- we would use language that was along the lines of "strongly dominant view, with no vigorous and coherent opposition".

When I started having conversations about the draft with folks, especially people who've been around quite a while, I was struck by the number of times the discussion started with, "No, that's not what we meant", but ended in, "Oh, yes, in those extreme cases, that's exactly what we meant." That is, I think the "51% is not enough, but 99% is more than enough" view was certainly in the front of people's minds, but when probed about the implications of that, almost everyone came to the same conclusion as the document: 51/99 is often a good rule of thumb if one wanted to guess about the consensus based on numbers of people, but it's still reasonable to call certain outcomes (like the ones described in the document) "rough consensus" even if the views of a huge majority might not win the day.

But it is certainly fair to say that in the 90s, few of us would have independently articulated the idea that a 90% majority might not be rough consensus.

That latter part probably should be seen as a crude version of what the draft is seeking to discuss, but I believe it was never consistently voiced or understood.

In other words, I think the role of minority views that the draft is putting forward is not a mere clarification and refinement to the existing IETF model, but rather is a significant enhancement. So its premise about the potential importance of minority views does have some historical hooks in the IETF philosophy, but not as much as the draft seems to assert.

Yep. I hope I can edit to make that clearer.

   allow decisions to be made in a vacuum without practical experience.

Again, this statement goes beyond actual practice. The language here prohibits decisions without experience and of course, the real IETF has always indulged in some decision-making based on theory that lacked practice. We /prefer/ experience. We /value/ the input it provides. But we are perfectly capable of making decision in a vacuum.

Remember that I'm saying what the "credo" says, not how tightly we adhere to it. How well we stick to it is talked about later. I think we do take as a principle that we want to base decisions on running code. We certainly don't always do so.

   Instead, decisions are made by (more or less) consent of all
   participants, and the actual products of engineering trump
   theoretical designs.

"Consent of all"? Hell no. Not even close, as your following text already note.

Again, here it is stating what the credo says, not the specifics of what we do (or even mean to do) in practice.

[Compare: Person A: "The US Constitution says, 'Congress shall make no law...abridging the freedom of speech.' Our credo is that the government will not limit speech." Person B: "But that's not true. The government limits all sorts of speech." Person A: "Never said it didn't. I said what the credo was. I will talk at length in a moment about how the government does and does not adhere to the credo, and how some of that is fine and still inline with our overall principles, and how some of it is problematic, and how they can do things that are less problematic."]

This is the Introduction. It's setting up the bigger picture. It's not the place to say, "The details are complicated"; of course they are.

   Our ideal is full consensus, but we don't
   require it: Full consensus would allow a single intransigent person

     Full consensus would allow
     ->
     Requiring full consensus gives each person a veto, by allowing

Right. I'm inclined to skip the "veto" language, but I will definitely edit.

   who simply keeps saying "No!" to stop the process cold.  We only
   require rough consensus: If the chair of a working group determines

Again, this seems to conflate two points. Both are extremely important. Both need to be discussed. But not intermixed.

One is that we do not require everyone to agree, as long as there is a strongly dominant group that do agree.

The other is that all concerns must be diligently considered, but do not have to be thoroughly resolved. 'Diligently considered' is an imposition on the entire group but it is mandatory. 'Not thoroughly resolved' is acceptable when the strongly dominant group agree that it is acceptable.

All true, but that discussion is not appropriate to the Introduction. The distinction between the two points is discussed at length later.

   that a technical issue brought forward by an objector has been truly
   considered by the working group, and the working group has made an
   informed decision that the objection has been answered or is not
   enough of a technical problem to prevent moving forward, the chair
   can declare that there is rough consensus to go forward, the
   objection notwithstanding.  To reinforce that we do not vote, we have

"To reinforce" switches from discussing the handling of minority concerns about voting. At a minimum, break the paragraph.

Done.

   also adopted the tradition of "humming": When (for example) the chair
   of the working group wants to get a "sense of the room", instead of a
   show of hands, the chair asks for each side to hum for or against a
   question.

The draft has historically drawn a strong distinction between hands and hums. It's an interesting distinction, and well might be useful, but I believe it has not been a universally-shared distinction in the community. It essentially relies on a theoretical analysis of differences in psychological effects, but there's no objective basis for validating the one really is superior to the other.

And the draft says as much. See section 4, paragraphs 3-5. This is discussed at length. I don't see what's missing from here.

   More and more
   frequently, people walk away from working groups, thinking that
   "consensus" has created a document with horrible compromises to
   satisfy everyone's pet peeve instead of doing "the right thing".
   None of these things are indicators of a rough consensus process
   being used, and are likely due to some basic misperceptions.

Is the issue their perception, or the fact that horrible compromises are actually being done? From the text, I can't tell what point is intended.

Here, it's only about the perception. I talk about the real problems with compromises at the end of section 2.

     it is not intended to dictate ->  it does not dictate

Will fix.

2.  Lack of disagreement is more important than agreement

The substance of this section has an important focus and makes useful comments.

However the title could be taken to cover such things as failure to gain affirmative support.

I don't intend any of my section titles to be complete and precise. If folks really don't like that, I can go about changing them, but I'd really rather not.

... Consensus is not when everyone is

par break at start of sentence.

Sure.

And possibly move this and the remaining sentences higher up, since they voice the essential point of the section. In other words, try to start with the premise so that folks are oriented, and then explain it.

I know this is apropos of a conversation we're having offline, but I do believe that people can get through the example of the problem before we tell them what the solution is.

Here's a suggestion for some re-casting to this separate paragraph, to help the reader bit, with the draft's logic and meaning:

Engineering always involves a set of tradeoffs. The process of gaining agreement about tradeoffs is the essence of a good rough consensus effort, and requires distinguishing the group debate and compromise needed to _come to_ consensus, the chair's process of _determining_ (or assessing) consensus, and the final point of the group's _having_ consensus on it choices.

Did you intend for that to replace something or just to insert it? I think it works well inserted, but if you meant something to be deleted, please clarify.

   However, there is another sense of "compromise" that involves
   compromising between people, not engineering principles.

This seems to argue that political compromise is never acceptable. That's not realistic. Folks often can "live with" unresolved concerns. The meta-issue is distinguishing disagreements about key points versus disagreement about minor ones. Political compromise about a flat vs. hierarchical namespace is a rather different kettle of importance than between specifying a syntactic separator of comma vs. semicolon. (No matter how much prettier I might think commas are, and superior for good visual presentation...)

So, the "live with" case is mentioned above, but I think it needs clarification in the previous paragraph. But I *do* want to argue that political compromise *is* never acceptable. It's one thing for me to conclude that I can live with semicolons over commas even though my preference is for commas (i.e., that it is a reasonable, though not preferred, engineering choice); that's just fine. It's quite different for me to conclude that the rest of these people are idiots for not choosing commas, but I'm going to put up with the semicolons because I want to make people happy so they'll do a hierarchical namespace, or because I've simply given up on trying to make this a good document. If it's a serious issue, giving up or trading is bad. If it's an "I can live with it" issue, then you're simply agreeing that the choice is acceptable.

And I'm not saying that the bad sort of compromise won't ever happen (we are humans, after all), or that there are no borderline cases (Is this me "living with it" or capitulating?), but I do want to say that participants have to decide whether they're making a reasonable engineering compromise or they are capitulating (or horse trading), and not do it if it's the latter.

And all that said, I need to clarify this.

   Note that this portrays rough consensus as an "exception".

I don't understand this first sentence. How is it an exception? Even with the following text, I don't see the basis for claiming that it might/would be seen as exceptional.

Maybe I need a different word. I didn't mean "exceptional" in the sense of "extraordinary" or "atypical". I simply meant it in the "exception handling" sense: We work toward full agreement, but if there's still disagreement, we undertake an alternative path. I'm open to suggestions for text.

   Now, a conclusion of having only rough consensus relies heavily on
   the good judgement of the consensus caller.  The group must truly
   consider and weigh an issue before the objection can be dismissed as
   being "in the rough".  The chair of the working group in one of these

In spite of being appealing to use here, I think that the phrase "in the rough" is actually distracting

As I mentioned earlier, I think it is worth defining as the golf term, since it is used quite frequently among IETFers nowadays.

   cases is going to have to decide that not only has the working group
   taken the objection seriously, but that it has fully examined the
   ramifications of not making a change to accommodate it, and that the
   outcome does constitute a failure to meet the technical requirements
   of the work.  In order to do this, the chair will need to have a good
   idea of the purpose and architecture of the work being done and use
   their own technical judgement to make sure that the solution meets
   those requirements.  What can't happen is that the chair bases their
   decision solely on hearing a large number of voices simply saying,
   "The objection isn't valid."  That would simply be to take a vote.  A
   valid justification needs to me made.

This begs the question of who determines that a justification is valid. I'll suggest that it mostly should be the working group, with the chair as fall-back or safety valve, but not default.

Remember, this is precisely the case where the working group claims to have considered the objection and found it to be wanting, but the objector still insists that the objection is valid. The rest of the working group can make the case that it has properly considered the objection, but it can't make the final call on whether it has; only the chair can do that.

By the way, the draft seems to think that it's always the chair who leads discussion and debate; it often/mostly isn't. Rather it's typically the authors/editors, with a chair serving to moderate impasses.

During my large edit, I'll have a look at this. My intention wasn't to say that the chair *leads* discussion. Indeed, beyond authors/editors, quite often it is someone else in the group who takes up the mantle of analyzing the discussion and proposing paths forward. But the chair in the end is responsible for making the final call as to the consensus of the group.

   A good outcome is for the
   objector to understand the decision taken and accept the outcome,
   even though their particular issue is not being accommodated in the
   final product.  Remember, if the objector feels that the issue is so

par break at Remember.

Sure.

   So, why do we hum?

This section reads nicely and sounds quite good and very possibly might be something that we /should/ do. However it has little to do with common IETF practice.

Yep. This goes back to your main critique. So really the above should say, "So why should we hum?", or something to that effect. Wilco.

Even better is that I think the draft has buried it's more important point, namely that the /form of the question/ asked when seeking a hum is the really hard part of the effort, and the most important part. Some forms invite constructive discussion and some close it off. Some are fair and balanced (...) and some are highly biasing.

I'll try to expand on that.

   But couldn't the above could have been done with a show of hands

This paragraph goes back to the hum-vs-hand methodology issue. It's important, but it's out of place here, since this section is about the /role/ of a hum, not its method. Discuss methodology separately, possibly up where it is first introduced.

I disagree with you on this one. The previous discussion was in the Introduction, which says, "Something's screwy here. What's up with that?" Here, I am trying to say, "Here's what we ought to be doing. Hums are good. Hands work too, but have their weaknesses." I think it works together well.

"the chair wants to choose the path that gets to the least objections fastest" sounds like an extremely appealing expedient, by virtue of being almost mechanical, but numbers often don't matter. A single objection that observes (and substantiates) that a choice will not or cannot be used in the real world will outweigh a much larger number of trivial objections.


Perhaps merely qualifying the statement a bit will suffice: the chair wants to choose the path that gets to the least number of significant objections fastest". Hmmm.

Yeah, this gets talked two paragraphs previously, but I think it's fair to reiterate here. I think the better fix is to add to the below sentence:

   There is always a chance
   that this could be misleading, because some people are more willing
   to hum loudly than others (just by dint of personality), but keep in
   mind that taking the hum is to figure out how to start the
   conversation.

Add after the parenthetical: "or that one of the quieter hums actually turn outs to be a showstopper that makes the original choice impossible,"


   Of course, a chair could get the temperature of the room by doing a
   show of hands too, and knowing who specifically feels one way or
   another can help a good chair guide the subsequent conversation.

And possibly place the minority under group pressure.

Sometimes a hum can do exactly the same.

   Sometimes, they simply mean they are making a call _of_ the
   consensus; that is, they are declaring the consensus that has, in
   their view, been reached when the discussion has reached an end.
   That's a fine thing and what chairs are supposed to do: They are
   "calling" the consensus.  Sometimes, when a chair says that they are
   We often hear chairs say that they are making a "consensus call".

Might be worth tossing in the word "confirm" or "confirming existing consensus" here, to leave no doubt about what "calling" means.

Yeah, or use the word "stating" or "expressing". I'll work on that.

   So in a large working group with over 100 active participants and
   broad agreement to go forward with a particular protocol, if a few
   participants say, "This protocol is going to cause congestion on the
   network, and it has no mechanism to back off when congestion occurs;
   we object to going forward without such a mechanism in place", and
   the objection is met with silence on the mailing list, there is no

The problem here is who is the authority to determine that the objection needs to hold sway. It's easy to lay this on the chair, but that makes the mechanisms fragile; it's too easy for one person to get this sort of thing wrong.

The case we're talking about here is an objection being met with silence. In that case, the presumption (i.e., not the conclusion) is, "Real objection; needs to be dealt with." The only way that the chair can "get this sort of thing wrong" is to at that point silently decide that it's not an objection worth addressing. The chair can say to the group, "I've seen this objection, but I don't think it's valid. Can someone confirm my view that it's not valid, or can the objector or someone else please explain why it is?" But this is only about whether a completely dropped-on-the-floor objection is OK. It's not.

Did I miss your point?



7.  Five people for and one hundred people against might still be rough
    consensus
...


   Now the worst case scenario happens.  The objector, still unhappy
   that their preferred solution was not chosen, recruits one hundred
   people, maybe a few who were silent participants in the working group
   already, but mostly people who work at the same company as the
   objector who never participated before.  The objector gets them all
   to post a message to the list saying, "I believe we should go with
   the new elegant algorithm in section Z instead of the current one.
   It is more elegant, and works better on our hardware."  The chair
   sees these dozens of messages coming in and posts a query to each of
   them: "We discussed this on the list, and we seemed to have consensus
   that, given the inherent risk of a new algorithm, and the widespread
   deployment of this current one, it's better to stick with the current
   one.  Do you have further information that indicates something
   different?"  And in reply the chair gets utter silence.  These
   posters to the list (say some of whom were from the company sales and
   marketing department) thought that they were simply voting and have
   no answer to give.  At that point, it is within bounds for the chair
   to say, "We have objections, but the objections have been
   sufficiently answered, and the objectors seem uninterested in
   participating in the discussion.  Albeit rough in the extreme, there
   is rough consensus to go with the current solution."

There's an interesting methodological point buried quite deep in this example, and probably worth surfacing sooner and more directly:

Besides being careful about how the question for a consensus check is phrased, it is sometimes advisable to seek to discover what the responses mean.

In the example here, the question is whether those responding were sufficiently well-informed to be able to justify their position.

I'll take an edit pass. Yes, that is exactly what I was trying to get at (among other things).

   There is no doubt that this is the degenerate case and a clear
   indication of something pathological.  But this is precisely what
   rough consensus is designed to guard against: vote stuffing.  In the

Sorry, no. It's not 'designed' for that at all, at least not in typical practice in the IETF. It probably /should/ be, and the draft does a pretty good job of explaining how it /can/ be, but there is nothing in the "design" of the IETF's consensus process that has a history of mitigating against the problem the draft discusses here, and the problem isn't mentioned in formal IETF documents.

Yes, fair criticism. "But this is precisely what rough consensus is ideally suited to guard against:"

Excellent comments. Thanks.

On to the next set....

pr

--
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478





[Index of Archives]     [IETF Annoucements]     [IETF]     [IP Storage]     [Yosemite News]     [Linux SCTP]     [Linux Newbies]     [Fedora Users]