> > There are quite a few really good ideas for improvements to IETF productivity. > > The problem with taking a particular suggestion and then adding others to it > > will be that nothing gets considered in detail and nothing gets done. > > you say that like it's a bad thing. > > not to pick on you personally, but since you brought this up - it's > hard to escape the impression that you'd rather have one half-baked > idea adopted without much discussion, than to have serious discussion > about what the real problems are and how to solve them. > > come to think of it, that resembles a lot of what passes for > 'engineering' of IETF protocols. > > Keith Let me follow up to my own message here because I think I responded a bit too quickly, and because I think my original reply didn't really address the point I was trying to make. In particular I didn't really want to single out Dave - he just happened to be the person who sent the message that triggered this response. I think that Dave's message reflects a common frustration in IETF that we talk a lot about particular problems and never seem to do anything about them. When people express that frustration, they often seem to think that the solution to this frustration is to do something rather than just talk about it. In other words, they prefer experimentation to analysis. I share the frustration, but have some doubts about the solution. There are circumstances where experimentation is appropriate. Offhand, these seem to include: (a) the cost of failure of the experiment is small relative to the potential gain, (b) the problem being addressed is so poorly understood or so complex that analysis isn't an option, and (c) the results of the experiment can be evaluated with a reasonable degree of objectivity to inform a decision about whether to do things that way in the future. Even when those circumstances are met, the experiment is usually more valuable if it is carefully designed based on such undertstanding and analysis as is available. What really bothers me is the apparent popularity of a mindset, in a group of people that claims to be doing engineering, that we should just try something without really thinking about it, and without a good way to evaluate the experiment objectively. The fundamental assumption of engineering is that you can make better (more effective, reliable, and cost-effective) solutions to problems if you (a) first understand what problem you are trying to solve, and (b) analyze your proposed solutions (and choose and/or refine them based on analysis) before building them. More and more in IETF we seem to be insisting that we (a) artifically and prematurely limit the scope of an engineering effort to a narrow solution space, (b) decide on that solution without doing any analysis, and (c) let the marketplace or the Internet community pay to conduct very expensive and poorly designed experiments without any good way to evaulate the results or much of an ability to change direction based on what is learned. Examples that pop into my head of this include DKIM, IMA, ZEROCONF. I'm not claiming that this is a complete or representative sample, these are just three things that popped into my head in three seconds. (and hopefully a diverse enough sample that nobody thinks I'm picking on him personally) Of course our management and process issues are different than our protocols, and it would make some sense for us to attack those problems differently. It would be understandable if, out of familiarity, we tried to apply engineering techniques to solving our management and process problems...and it would be understandable if we found that it didn't work well because we didn't know how to do the right kinds of analysis. But what seems to be the case instead is that we hardly use engineering discipline in protocol design. We guess about what will make a good protocol design, and guess about what will improve our management and process also. If it seems like we make good progress on the former and poor progress on the latter, perhaps it's because for management and process issues that affect all of us, we have no good way to artifically narrow the scope of the discussion (and marginalize the nay-sayers) in order to get the apperance of agreement. I'm all for experimentation about how we run meetings as long as we take reasonable care in designing the experiments, identify effective ways to evaluate the results of the experiments, and leave ourselves room to adopt a different course if we don't like the results. It helps if we have controls also. For instance, if we wanted to experiment with allowing vendors to pay for greater exposure, we could do this at one meeting per year for a couple of years and see how we like it in comparison with other meetings. I'm much more concerned about how we design our protocols. Our longstanding habit of marginalizing dissenting voices in order to get agreement within a working group - our failure to resolve basic design conflicts before settling on designs - has contributed to a fragmented and conflicted Internet architecture (if one can call it an architecture). But there's a catch-22 here: I don't see how we can do better protocol design without some substantial changes in how working groups operate, and more crucially - changes to the habits and assumptions made by IETF participants. And those very habits and assumptions make it inherently difficult to evaluate possible new ways of doing things. We cannot rationally evaluate ANY new proposals (technical, managerial, or process) that potentially affect a large or diverse group, in an environment where - Participants expect (and demand) to be able to talk incessantly and without any structure to the discussion, thereby making progress difficult. They demand this at least in part because there is no other way to get issues considered than to badger people about them. - Because of the volume of talk, there tends to be a limited attention span on the part of participants. - The most effective way to achieve either real progress or the apperance of progress is to limit the discussion scope, often in such a way as to exclude important considerations. - Because of this tendency to make decisions prematurely and without analysis or adequate review, every new idea proposed becomes a threat. Rather than carefully considering new ideas and refining some of those that seem to be promising, we either adopt them without much refinement or scuttle them. The discussion of such ideas tends to happen at a level, and in an environment, where detailed understanding is not possible. My question is - do others see this as a problem, and (without trying to propose a concrete solution that will be seen as a threat) is there a shared sense that this is a problem and general willingness to try new ways of conducting our discussions? Keith _______________________________________________ Ietf@xxxxxxxx https://www1.ietf.org/mailman/listinfo/ietf