Al, > [acm] I agree with your observation that there are cases where trial duration > should be increased to accommodate the encountered in the DUT, but not as a > mandate for all testing. I have four factors in mind: > 1. Some of the virtual network DUTs we are testing now have very small buffers, > and the B2B stream of frames is quite short -- less than 2000 frames@10GE in > some cases -- so 2 seconds fully sufficient. > 2. The trial duration is a factor in total test duration, where each trial is one step in > the Binary Search. We need to manage the tension between the time needed to > reach a search result and confidence that we have depleted the queues. > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is present. > 4. The current text says "at least 2 seconds". > > So I suggest adding the following text: > > The duration of the trial MUST be at least 2 seconds, to allow DUT > buffers to deplete. When RFC2544 Latency measurements indicate that > large buffers are present in the DUT, the trial duration SHOULD be > increased to ensure that buffer depletion takes place, without unduly > extending the total test time. The overall approach of collecting evidence that there is a problem before increasing the 2 second minimum duration is fine, but the details appear to need more attention: 1. Obtaining the RFC 2544 Latency measurements would need to be added to Section 4 (Prerequisites) of this draft to ensure that the buffer size information is available. 2. I did not see any requirements in the RFC 2544 Latency test (Section 26.2) to deplete buffers. Did I miss something? 3. I would think that the trial duration MUST be increased, not just SHOULD be increased if there is evidence of large buffer size, as buffer depletion appears to be a necessary characteristic of this B2B measurement. It also looks like the link to the entire slide deck didn't make it into my original review correctly - that slide deck is at: http://www.taht.net/~d/lca_tcp3.odp . In addition to slide 6 from this slide deck (Figure 1 in the APNIC blog), slide 14 is also relevant to this discussion. Thanks, --David > -----Original Message----- > From: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx> > Sent: Tuesday, November 24, 2020 11:11 AM > To: Black, David; tsv-art@xxxxxxxx > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last-call@xxxxxxxx > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > [EXTERNAL EMAIL] > > Hi David, Thanks for your review and comment! > > Please see a proposed resolution below, [acm] Al > > > -----Original Message----- > > From: David Black via Datatracker [mailto:noreply@xxxxxxxx] > > Sent: Tuesday, November 24, 2020 10:27 AM > > To: tsv-art@xxxxxxxx > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > > call@xxxxxxxx > > Subject: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > > Reviewer: David Black > > Review result: Ready with Issues > > > ... > > > > This draft updates the back-to-back frame testing procedure in RFC > > 2544 to take account of experience. > > > > The draft is in good shape, with one notable exception in Section 5.2: > > > > The duration of the trial MUST be at least 2 seconds, to allow DUT > > buffers to deplete. > > > > That duration of 2 seconds has been carried forward from RFC 2544 > > without change. A 2 second duration may have been sufficient to > > deplete buffers in 1999, but that is no longer reliably the case. For > > example, on-site measurement of the network for the 2020 Linux > > Conference in Australia indicated a at least 1.6 seconds of buffering, > > as indicated by Figure 1 at > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-yK2DOVVn0ecoYjf7mFEABLmlwDk$ , > > which is slide 6 in from the complete slide deck at: > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0-yK2DOVVn0ecoYjf7mFEABLmlwDk$ > > . Experience with bufferbloat suggests that one network device was > > primarily responsible. Also, see slide 14 in that slide deck. > > > > That 1.6 seconds measured on an actual network is entirely too close > > to 2 seconds for confidence that buffers will be depleted in any tested device. > > Hence, the 2 second minimum duration ought to be increased by at least > > a factor of 10. I'd suggest changing it to 30 seconds or 60 seconds > > as convenient round numbers, and providing the rationale that > > increased buffering in WiFi devices, e.g., home "routers," as > > indicated by experience with bufferbloat measurements, is the reason for the > increased duration. > > > [acm] > [acm] I agree with your observation that there are cases where trial duration > should be increased to accommodate the encountered in the DUT, but not as a > mandate for all testing. I have four factors in mind: > 1. Some of the virtual network DUTs we are testing now have very small buffers, > and the B2B stream of frames is quite short -- less than 2000 frames@10GE in > some cases -- so 2 seconds fully sufficient. > 2. The trial duration is a factor in total test duration, where each trial is one step in > the Binary Search. We need to manage the tension between the time needed to > reach a search result and confidence that we have depleted the queues. > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is present. > 4. The current text says "at least 2 seconds". > > So I suggest adding the following text: > > The duration of the trial MUST be at least 2 seconds, to allow DUT > buffers to deplete. When RFC2544 Latency measurements indicate that > large buffers are present in the DUT, the trial duration SHOULD be > increased to ensure that buffer depletion takes place, without unduly > extending the total test time. > > I hope this suggestion resolves your issue; thanks for highlighting it! > Al -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
