An IETF hum is better used as a method for building a community consensus, by showing outliers the strength of other opinions.
As such it is important theatre. As a method of assessing opinions, it isn't very good, for many of the reasons cited.
Larry
Dave Crocker <dhc@xxxxxxxxxxxx> wrote:
Folks,
An IETF hum is a method of surveying a group for its views.
Unfortunately the hums that were taken at the end of this week's IAB
plenary do not permit any meaningful interpretation.
Here's why...
Surveys are extremely sensitive to the phrasing of the questions, the
phrasing and range of the response choices, the sequencing of the
questions, and the context of the asking. Get any of these wrong and
you can get the wrong information, or even just the appearance of
information -- that is, misunderstandings -- but nothing actually useful.
A common response to such a concern is "well, at least we'll get some
answers", but that's like saying "well, at least we'll get some noise."
The fact that the noise is misunderstood to be signal does not
actually make it signal.
The different phrasings of a question can produce very different
understandings by responders. The challenge is to formulate a question
that is likely to be interpreted similarly amongst responders (and the
person asking.) It's also a challenge to ask a question that captures
something that is actually meaningful (and was intended) rather than
merely sounding good.
The offered response choices can bias the responses. A set of choices
like (Good, Excellent) obviously leaves out (Bad, Don't Care, Don't
Know.) Or they can have bias in their phrasing by making some choices
more or less appealing (Could be better, Excellent), rather than
equivalent vocabulary in tone (Bad, Good). So it's a challenge to make
sure that choices cover the proper range and with equanimity to the
alternative choices.
A sequence of questions also needs to be carefully orchestrated. For
example today's questions took as a given that surveillance is an
attack. Due diligence might expect establishing that relationship
explicitly. And yes, it is possible that some IETF attendees do not see
it as an attack. Another example of sequencing is dealing with
subtleties and complexities. For example some anti-surveillance
mechanisms are certain to defeat popular operational management
mechanisms. Do we care about the tradeoffs?
Lastly, environmental context can encourage or discourage candor.
Examples include the genders of the asker and respondent, any
relationship they might have, or the presence of others. Would you
really provide candid answers about possible problems with your sex life
when being asked with your partner present? Amongst a group of
co-workers? Your parents?
The hums asked at the plenary were problematic along each of these lines.
The first question was theatre, essentially making the context
political. By way of example, note the difference between what was asked:
The IETF is willing to respond to the pervasive surveillance attack?
which has loaded language with 'pervasive' and 'attack', versus a more
neutral and purely technical question meant to cover the same basic concern:
The IETF is willing to improve its specifications to be more
resistant to surveillance?
But this isn't exactly a balanced question either. By that, I mean that
the answer really is already known. A good question is one that has a
chance of getting some support for each choice. So perhaps a better
example would be:
The IETF is willing to require adding resistance to surveillance
to all of its protocols?
The questions typically also did not offer "don't know" or "don't care"
choices. Some folk probably knew that they don't know enough yet,
limiting their ability to support the kinds of questions being asked.
The IETF's doing anything privacy-related that is useful is going to
require considering tradeoffs and some of those tradeoffs might reduce
the utility of a service. So the actual choices that will be made might
turn out to be quite different from what was implied by the dominant
answers to the plenary questions.
And lastly, consider carefully the context of the room and ask whether
everyone actually felt completely free to give a "no" hum to the initial
questions. I suggest that the emotions of the room created a strong
bias against no's. Maybe not for you. Maybe not for me. But probably
for many of the folk sitting near you.
We now find ourselves with a set of hums that appears to establish a
direction but which can't survive even basic analysis, as the later
postings on the ietf mailing list demonstrate.
Here's what I suggest: A single, simple, conceptual question that
supplies all of the 'guidance' we can legitimately offer, at this stage:
The IETF needs to press for careful attention to privacy
concerns in its work, including protection against surveillance.
[ ] No
[ ] Yes
[ ] Don't Yet Know
[ ] Don't Care
d/
--
Dave Crocker
Brandenburg InternetWorking
bbiw.net
An IETF hum is a method of surveying a group for its views.
Unfortunately the hums that were taken at the end of this week's IAB
plenary do not permit any meaningful interpretation.
Here's why...
Surveys are extremely sensitive to the phrasing of the questions, the
phrasing and range of the response choices, the sequencing of the
questions, and the context of the asking. Get any of these wrong and
you can get the wrong information, or even just the appearance of
information -- that is, misunderstandings -- but nothing actually useful.
A common response to such a concern is "well, at least we'll get some
answers", but that's like saying "well, at least we'll get some noise."
The fact that the noise is misunderstood to be signal does not
actually make it signal.
The different phrasings of a question can produce very different
understandings by responders. The challenge is to formulate a question
that is likely to be interpreted similarly amongst responders (and the
person asking.) It's also a challenge to ask a question that captures
something that is actually meaningful (and was intended) rather than
merely sounding good.
The offered response choices can bias the responses. A set of choices
like (Good, Excellent) obviously leaves out (Bad, Don't Care, Don't
Know.) Or they can have bias in their phrasing by making some choices
more or less appealing (Could be better, Excellent), rather than
equivalent vocabulary in tone (Bad, Good). So it's a challenge to make
sure that choices cover the proper range and with equanimity to the
alternative choices.
A sequence of questions also needs to be carefully orchestrated. For
example today's questions took as a given that surveillance is an
attack. Due diligence might expect establishing that relationship
explicitly. And yes, it is possible that some IETF attendees do not see
it as an attack. Another example of sequencing is dealing with
subtleties and complexities. For example some anti-surveillance
mechanisms are certain to defeat popular operational management
mechanisms. Do we care about the tradeoffs?
Lastly, environmental context can encourage or discourage candor.
Examples include the genders of the asker and respondent, any
relationship they might have, or the presence of others. Would you
really provide candid answers about possible problems with your sex life
when being asked with your partner present? Amongst a group of
co-workers? Your parents?
The hums asked at the plenary were problematic along each of these lines.
The first question was theatre, essentially making the context
political. By way of example, note the difference between what was asked:
The IETF is willing to respond to the pervasive surveillance attack?
which has loaded language with 'pervasive' and 'attack', versus a more
neutral and purely technical question meant to cover the same basic concern:
The IETF is willing to improve its specifications to be more
resistant to surveillance?
But this isn't exactly a balanced question either. By that, I mean that
the answer really is already known. A good question is one that has a
chance of getting some support for each choice. So perhaps a better
example would be:
The IETF is willing to require adding resistance to surveillance
to all of its protocols?
The questions typically also did not offer "don't know" or "don't care"
choices. Some folk probably knew that they don't know enough yet,
limiting their ability to support the kinds of questions being asked.
The IETF's doing anything privacy-related that is useful is going to
require considering tradeoffs and some of those tradeoffs might reduce
the utility of a service. So the actual choices that will be made might
turn out to be quite different from what was implied by the dominant
answers to the plenary questions.
And lastly, consider carefully the context of the room and ask whether
everyone actually felt completely free to give a "no" hum to the initial
questions. I suggest that the emotions of the room created a strong
bias against no's. Maybe not for you. Maybe not for me. But probably
for many of the folk sitting near you.
We now find ourselves with a set of hums that appears to establish a
direction but which can't survive even basic analysis, as the later
postings on the ietf mailing list demonstrate.
Here's what I suggest: A single, simple, conceptual question that
supplies all of the 'guidance' we can legitimately offer, at this stage:
The IETF needs to press for careful attention to privacy
concerns in its work, including protection against surveillance.
[ ] No
[ ] Yes
[ ] Don't Yet Know
[ ] Don't Care
d/
--
Dave Crocker
Brandenburg InternetWorking
bbiw.net