--On Wednesday, November 4, 2020 21:10 +1300 IETF Executive Director <exec-director@xxxxxxxx> wrote: > Hi > > ** Final reminder ** > > The IETF is conducting a survey to help us understand what > tools and formats you use for authoring IETF Internet Drafts > (I-Ds), how you use them and what more you need from them. > The data from this survey is vital for our future planning of > tools investment. >... > Your email address has been selected because you have been > listed as the author or submitter of an I-D in the last five > years. The survey is anonymous - we only record the data you > provide and do not record your IP address or any details of > your computer. Jay, I filled the survey out last night. Partially prompted by other recent events and discussions, I have several comments and thoughts that I think deserve community, as well as LLC, attention. So, while I would normally send a note like this off list (as I have done in the past), I think this one needs to be public. In no particular order: (1) The experience would presumably be different for those for whom the number of I-Ds or RFCs is small, but it took me close to 45 minutes to make even the crudest of estimates for the first page. Just as some people (not a random sample of the community) find my long analytical notes discouraging and won't read to the end (or even start), surveys that require that sort of time investment will discourage some people (not a random sample of the community) from completing them and may even discourage them from opening your surveys in the future. That means non-representative results and a survey that requires great care to interpret especially in a way that does not just reinforce prior biases or assumptions. (2) Again probably different for those with small numbers of I-Ds and, especially, RFCs but for those of us with large numbers of the latter, the survey is not, as claimed above, anonymous. You don't need to collect IP addresses or data about computers - the number of people who are authors or co-authors of 51+ (if I remember correctly) RFCs is small enough that examination of answers to a few questions after that would make identifying us rather easy. (3) Our rules (or lack thereof) about RFC and I-D "authorship" sometimes don't predict well to actual construction of documents. There are exceptions, but the typical number of people who actually interact with the tools and key or copy words into their machine-usable representations is often one independent of the number of listed authors. When you include someone in a survey like this who has not actually interacted with a document or tools at a technical level, you get weird results if they complete the survey and an artificially low response rate if they do not. At the other extreme, many of us have helped relative newcomers by providing outline I-Ds in xml2rfc form based on their notes but are not listed as authors or even contributors -- efforts that don't count for survey purposes (I would not be surprised to learn that similar outlines have been provided in MS Word). And, fwiw, the larger amount of understanding required of a newcomer to flesh out an outline in xml2rfc v3 as compared to v2 is one objection to the new format that has not been discussed enough. (4) The question and response structure of the survey instrument is poorly designed (assuming you are after confident and maximum interpretability). I have many examples but, to keep this note from getting longer, let me stop at three: (4.a) While I imagine most of us could figure it out, putting a "If you have any questions about this survey then please contact..." comment after the survey is completed is fairly pointless. To be useful, such a statement should appear on the first page and/or the cover letter suggesting that people take the survey, not after it is done and it is too late to ask questions about the survey itself. Although I'm confident that was just an oversight, it seems to me sufficiently obvious to suggest that not enough thinking went into the survey design. (4.b) A couple of decades ago (before xml2rfc existed), I used to do most of my draft and RFC preparation in plain text, with one or two in nroff, and one I recall going in and out of WordPerfect or MS Word before being edited in plain text. I don't intend to go back there and other than giving me a bit of extra perspective on those types of tools and methods, it is not clear to me that history that far back is relevant to what you are trying to learn. However,except for that first question about date ranges the survey doesn't say "within the past X" years. But your scales in that part of the survey are "Always/ Often / ... / Never". So if, for example, I constructed an I-D in 1998 using nroff obtained from the RFC Editor because I was trying to revise or update an earlier document, I can't answer "never" to a "used nroff" question even though I never used any of the templates or specialized tools listed, nor can I answer "always" to use of xml2rfc (even in the last decade) but pushing me into "often" does not give you the information that "almost always" might. Either the choice of scale labels is unfortunate or the questions need to be more refined. (4.c) Then there are questions about how satisfied I am with xml2rfc as an authoring tool, with a scale from "very satisfied" to "very dissatisfied". Reasonable question except that there are some of us who are "very" (or closer than "nearly") satisfied with v2 (perhaps for substantive reasons and perhaps because we are just used to it) and very dissatisfied with v3. I assume there are those who are "very" satisfied with v3 but hate v2. But the questions fail to allow the respondent to differentiate so responses to any statement made about satisfaction with xml2rfc (by implication, both versions) are going to be so hard to interpret accurately as to be nearly useless. I'm not going to go on further except to say that, for those of us who are trying to do technical work in the IETF, surveys like this (and at least some of the "Consultations" you have posted) take time away from that available to do that work. That might be worthwhile if the surveys/ consultations are infrequent and designed well and carefully enough to inspire confidence in their interpretation and use. But, even then, I suggest that at current levels they may be interfering with progress on standards-related activities, which is one thing the LLC, at least IMO, should absolutely not be doing. thanks, john
