Al, > I clarified: > The duration of the trial MUST include least 2 seconds in addition to the time > required to send and receive each burst of frames, to ensure that DUT buffers to > deplete. > > and I'll add: > The upper search limit for the time to send each burst MUST be configurable as > high as 30 seconds (buffer time results reported at the configured upper limit are > likely invalid, and the test MUST be repeated with a higher search limit). I think this will work - please make sure to address the scenario in which the search algorithm hits the upper search limit without causing any frame loss (e.g., upper search limit configured to 2 seconds for a bizarre DUT that has 10 seconds of buffering). Thanks, --David > -----Original Message----- > From: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx> > Sent: Wednesday, November 25, 2020 1:20 PM > To: Black, David; tsv-art@xxxxxxxx > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last-call@xxxxxxxx > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > [EXTERNAL EMAIL] > > David, > > Allow me to points more clearly: > > - I'm relying on the Binary Search aspect of the procedure to measure unexpected > large buffer times; searches are an integral part of the procedure. > > - These are benchmarking measurements in an isolated test environment, where > we will control every source of variability possible to foster repeatability. > > - The trial duration has two components: time to conduct sending + receiving, plus > additional waiting time for buffer depletion. The test equipment orchestrates this > easily, because it is a single device. > > I clarified: > The duration of the trial MUST include least 2 seconds in addition to the time > required to send and receive each burst of frames, to ensure that DUT buffers to > deplete. > > and I'll add: > The upper search limit for the time to send each burst MUST be configurable as > high as 30 seconds (buffer time results reported at the configured upper limit are > likely invalid, and the test MUST be repeated with a higher search limit). > > Al > > > > -----Original Message----- > > From: Black, David [mailto:David.Black@xxxxxxxx] > > Sent: Wednesday, November 25, 2020 9:35 AM > > To: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx>; tsv-art@xxxxxxxx > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > > call@xxxxxxxx; Black, David <David.Black@xxxxxxxx> > > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > > Al, > > > > We appear to not be communicating on a key aspect of the concern: > > > > > A burst of 1.7 seconds falls well within the coverage of a 2 second > > > trial to count the frames and determine that loss has occurred. We > > > don't need a big margin time- wise. Also, the trial duration > > > increases with the search parameters: IOW, if the tester is still > > > sending frames after 2 seconds, then obviously we can't allow the trial to > terminate prematurely. > > > > The 1.6 seconds value is *not* an upper bound - it was measured in > > practice on an operational conference network as part of what appears > > to have been a just-in-time slide preparation exercise. Hence that > > 1.6 seconds of buffering ought to be viewed as being somewhere in the > > midst of the expected distribution of buffer times. For that reason, > > the upper bound on test duration needs to be set with a safety factor > > to ensure that even if a bizarre DUT has 10 seconds of buffering, the > > B2B test depletes that buffering so that the B2B test is testing the > > forwarding path as intended, not the buffering capacity. > > > > > I want to avoid over-reacting to the buffer bloat case. We all agree > > > that bloated buffers are bad, and knowing a DUT has buffering >=1 > > > second may be enough. I certainly do not think that trial durations > > > for the B2B Frame benchmark need to be as long as you originally > > > suggested, where the overwhelming number of test conditions can use a <<1 > second burst and 2 second buffer depletion time: > > > > I agree with this approach provided that there is a test procedure (in > > the B2B test and/or its prerequisites) that will discover the behavior > > of the bizarre DUT with 10 seconds of buffering and adjust the B2B > > test time to ensure that this excessive buffering is depleted. > > > > > I hope the approach of the single sentence above works for you. > > > > Based on this discussion, in addition to that sentence (which I think > > has to contain a "MUST" for situations in which excessive buffering is > > found), I would like to see a brief explanation of how the procedures > > in the draft, along with additional test procedures pulled in from RFC > > 2544 or elsewhere, avoid a 2 second test duration on a DUT with 10 > > seconds of buffering. > > > > Thanks, --David > > > > > -----Original Message----- > > > From: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx> > > > Sent: Wednesday, November 25, 2020 12:00 AM > > > To: Black, David; tsv-art@xxxxxxxx > > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > > call@xxxxxxxx > > > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > > > > > > > [EXTERNAL EMAIL] > > > > > > Hi David, some more clarity below. > > > Al > > > > > > > > > > -----Original Message----- > > > > From: Black, David [mailto:David.Black@xxxxxxxx] > > > > Sent: Tuesday, November 24, 2020 3:02 PM > > > > To: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx>; tsv-art@xxxxxxxx > > > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > > > > call@xxxxxxxx; Black, David <David.Black@xxxxxxxx> > > > > Subject: RE: Tsvart last call review of > > > > draft-ietf-bmwg-b2b-frame-03 > > > > > > > > Al, > > > > > > > > > [acm] I agree with your observation that there are cases where > > > > > trial > > > > duration > > > > > should be increased to accommodate the encountered in the DUT, > > > > > but not > > > > as a > > > > > mandate for all testing. I have four factors in mind: > > > > > 1. Some of the virtual network DUTs we are testing now have very > > > > > small > > > > buffers, > > > > > and the B2B stream of frames is quite short -- less than 2000 > > > > frames@10GE in > > > > > some cases -- so 2 seconds fully sufficient. > > > > > 2. The trial duration is a factor in total test duration, where > > > > > each > > > > trial is one step in > > > > > the Binary Search. We need to manage the tension between the > > > > > time needed > > > > to > > > > > reach a search result and confidence that we have depleted the > > queues. > > > > > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is > > > > present. > > > > > 4. The current text says "at least 2 seconds". > > > > > > > > > > So I suggest adding the following text: > > > > > > > > > > The duration of the trial MUST be at least 2 seconds, to > > > > > allow > > DUT > > > > > buffers to deplete. When RFC2544 Latency measurements > > > > > indicate > > that > > > > > large buffers are present in the DUT, the trial duration > > > > > SHOULD > > be > > > > > increased to ensure that buffer depletion takes place, > > > > > without > > unduly > > > > > extending the total test time. > > > > > > > > The overall approach of collecting evidence that there is a > > > > problem before increasing the 2 second minimum duration is fine, > > > > but the details appear to need more attention: > > > > > > > > 1. Obtaining the RFC 2544 Latency measurements would need to be > > > > added to Section 4 (Prerequisites) of this draft to ensure that > > > > the buffer size information is available. > > > [acm] > > > Section 26.2 Latency is a very primitive procedure by today's standards. > > It is > > > common for testers to measure delay on every packet, and report a > > > true > > average. > > > This is enough to alert the operator to the presence of bufferbloat, > > > but > > there's no > > > assurance that the Latency benchmark measures the extent of the > > > buffer > > time. > > > > > > > > > > > 2. I did not see any requirements in the RFC 2544 Latency test > > > > (Section > > > > 26.2) to deplete buffers. Did I miss something? > > > [acm] > > > It's an overall requirement in Section 23, after running the trial: > > > > > > d) Wait for two seconds for any residual frames to be received. > > > > > > > > > > > 3. I would think that the trial duration MUST be increased, not > > > > just SHOULD be increased if there is evidence of large buffer > > > > size, as buffer depletion appears to be a necessary characteristic > > > > of this B2B measurement. > > > [acm] > > > > > > We need to keep in mind that the sentence(s) in this discussion > > > pertain > > to the trials > > > of the B2B Frame Benchmark testing, in the section that begins: > > > > > > Each trial in the test requires the tester to send a burst of > > > frames > > (after idle time) > > > with the minimum inter-frame gap, and to count the corresponding > > > frames forwarded by the DUT. > > > > > > Let's say we have a DUT that offers 1.6 seconds of buffering. The > > > Binary > > Search will > > > increase the length of the burst until the buffer drops frames, and > > > then > > try to find > > > the longest burst where frame loss is zero. This simple procedure > > > always > > waits the > > > minimum trial duration for frames to exit the DUT. > > > > > > A burst of 1.7 seconds falls well within the coverage of a 2 second > > trial to count the > > > frames and determine that loss has occurred. We don't need a big > > > margin > > time- > > > wise. Also, the trial duration increases with the search parameters: > > IOW, if the > > > tester is still sending frames after 2 seconds, then obviously we > > > can't > > allow the trial > > > to terminate prematurely. > > > > > > So, I suggest we use this wording instead: > > > > > > The duration of the trial MUST be at least 2 seconds in > > > addition > > to > > > the time to send each burst of frames, to allow DUT buffers > > > to > > deplete. > > > > > > and we establish an adaptive waiting time for extreme cases, > > > consistent > > with > > > RFC2544 section 23 item d). For a DUT with 1.6 second buffers, the > > > Trial > > duration > > > would be 3.6 seconds. > > > > > > BTW, one tester issue we are trying to fix here is the case when the > > frame header > > > processing rate is sufficiently high that no buffer limit is > > > measureable > > at the large > > > frame sizes, and we implore people not to waste time with the B2B > > > Frame Benchmark at those frame sizes. One commercial tester sent > > > bursts up to > > 30 > > > seconds in length, and then happily reported the impossible result > > > of 30 > > second > > > buffers. > > > > > > I want to avoid over-reacting to the buffer bloat case. We all agree > > that bloated > > > buffers are bad, and knowing a DUT has buffering >=1 second may be > > enough. I > > > certainly do not think that trial durations for the B2B Frame > > > benchmark > > need to be > > > as long as you originally suggested, where the overwhelming number > > > of > > test > > > conditions can use a <<1 second burst and 2 second buffer depletion > > time: > > > > > > > > > Hence, the 2 second minimum duration ought to be increased by > > > > > > at least a factor of 10. I'd suggest changing it to 30 > > > > > > seconds or 60 > > seconds... > > > > > > I hope the approach of the single sentence above works for you. > > > > > > > > > > > > > > > > > It also looks like the link to the entire slide deck didn't make > > > > it into my original review correctly - that slide deck is at: > > > > https://urldefense.com/v3/__http://www.taht.net/*d/lca_tcp3.odp__;fg!! > > > > BhdT !0oWabwvUrtVlGQY7ZKnFnMa- > > > un_e95tfCJcRKIWG7wECiJkVISb4sjd9Salwehw$ > > > > . In addition to slide 6 from this slide deck (Figure 1 in the > > > > APNIC blog), slide 14 is also relevant to this discussion. > > > > > > > > Thanks, --David > > > > > > > > > -----Original Message----- > > > > > From: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx> > > > > > Sent: Tuesday, November 24, 2020 11:11 AM > > > > > To: Black, David; tsv-art@xxxxxxxx > > > > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > > > > call@xxxxxxxx > > > > > Subject: RE: Tsvart last call review of > > > > > draft-ietf-bmwg-b2b-frame-03 > > > > > > > > > > > > > > > [EXTERNAL EMAIL] > > > > > > > > > > Hi David, Thanks for your review and comment! > > > > > > > > > > Please see a proposed resolution below, [acm] Al > > > > > > > > > > > -----Original Message----- > > > > > > From: David Black via Datatracker [mailto:noreply@xxxxxxxx] > > > > > > Sent: Tuesday, November 24, 2020 10:27 AM > > > > > > To: tsv-art@xxxxxxxx > > > > > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; > > > > > > last- call@xxxxxxxx > > > > > > Subject: Tsvart last call review of > > > > > > draft-ietf-bmwg-b2b-frame-03 > > > > > > > > > > > > Reviewer: David Black > > > > > > Review result: Ready with Issues > > > > > > > > > > > ... > > > > > > > > > > > > This draft updates the back-to-back frame testing procedure in > > > > > > RFC > > > > > > 2544 to take account of experience. > > > > > > > > > > > > The draft is in good shape, with one notable exception in > > > > > > Section > > 5.2: > > > > > > > > > > > > The duration of the trial MUST be at least 2 seconds, to > > > > > > allow > > DUT > > > > > > buffers to deplete. > > > > > > > > > > > > That duration of 2 seconds has been carried forward from RFC > > > > > > 2544 without change. A 2 second duration may have been > > > > > > sufficient to deplete buffers in 1999, but that is no longer reliably the case. > > > > > > For example, on-site measurement of the network for the 2020 > > > > > > Linux Conference in Australia indicated a at least 1.6 seconds > > > > > > of buffering, as indicated by Figure 1 at > > > > > > > > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/buff > > > > erbl > > > > oat- > > > > may-be-solved-but-its-not-over- > > > > yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0- > > > > yK2DOVVn0ecoYjf7mFEABLmlwDk$ , > > > > > > which is slide 6 in from the complete slide deck at: > > > > > > > > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/buff > > > > erbl > > > > oat- > > > > may-be-solved-but-its-not-over- > > > > yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0- > > > > yK2DOVVn0ecoYjf7mFEABLmlwDk$ > > > > > > . Experience with bufferbloat suggests that one network > > > > > > device was primarily responsible. Also, see slide 14 in that slide deck. > > > > > > > > > > > > That 1.6 seconds measured on an actual network is entirely too > > > > > > close to 2 seconds for confidence that buffers will be > > > > > > depleted in any > > > > tested device. > > > > > > Hence, the 2 second minimum duration ought to be increased by > > > > > > at least a factor of 10. I'd suggest changing it to 30 > > > > > > seconds or 60 seconds as convenient round numbers, and > > > > > > providing the rationale that increased buffering in WiFi > > > > > > devices, e.g., home "routers," as indicated by experience with > > > > > > bufferbloat measurements, is the reason > > > > for the > > > > > increased duration. > > > > > > > > > > > [acm] > > > > > [acm] I agree with your observation that there are cases where > > > > > trial > > > > duration > > > > > should be increased to accommodate the encountered in the DUT, > > > > > but not > > > > as a > > > > > mandate for all testing. I have four factors in mind: > > > > > 1. Some of the virtual network DUTs we are testing now have very > > > > > small > > > > buffers, > > > > > and the B2B stream of frames is quite short -- less than 2000 > > > > frames@10GE in > > > > > some cases -- so 2 seconds fully sufficient. > > > > > 2. The trial duration is a factor in total test duration, where > > > > > each > > > > trial is one step in > > > > > the Binary Search. We need to manage the tension between the > > > > > time needed > > > > to > > > > > reach a search result and confidence that we have depleted the > > queues. > > > > > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is > > > > present. > > > > > 4. The current text says "at least 2 seconds". > > > > > > > > > > So I suggest adding the following text: > > > > > > > > > > The duration of the trial MUST be at least 2 seconds, to > > > > > allow > > DUT > > > > > buffers to deplete. When RFC2544 Latency measurements > > > > > indicate > > that > > > > > large buffers are present in the DUT, the trial duration > > > > > SHOULD > > be > > > > > increased to ensure that buffer depletion takes place, > > > > > without > > > > unduly > > > > > extending the total test time. > > > > > > > > > > I hope this suggestion resolves your issue; thanks for > > > > > highlighting > > it! > > > > > Al -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
