Hi David, some more clarity below. Al > -----Original Message----- > From: Black, David [mailto:David.Black@xxxxxxxx] > Sent: Tuesday, November 24, 2020 3:02 PM > To: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx>; tsv-art@xxxxxxxx > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > call@xxxxxxxx; Black, David <David.Black@xxxxxxxx> > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > Al, > > > [acm] I agree with your observation that there are cases where trial > duration > > should be increased to accommodate the encountered in the DUT, but not > as a > > mandate for all testing. I have four factors in mind: > > 1. Some of the virtual network DUTs we are testing now have very small > buffers, > > and the B2B stream of frames is quite short -- less than 2000 > frames@10GE in > > some cases -- so 2 seconds fully sufficient. > > 2. The trial duration is a factor in total test duration, where each > trial is one step in > > the Binary Search. We need to manage the tension between the time needed > to > > reach a search result and confidence that we have depleted the queues. > > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is > present. > > 4. The current text says "at least 2 seconds". > > > > So I suggest adding the following text: > > > > The duration of the trial MUST be at least 2 seconds, to allow DUT > > buffers to deplete. When RFC2544 Latency measurements indicate that > > large buffers are present in the DUT, the trial duration SHOULD be > > increased to ensure that buffer depletion takes place, without unduly > > extending the total test time. > > The overall approach of collecting evidence that there is a problem before > increasing the 2 second minimum duration is fine, but the details appear > to need more attention: > > 1. Obtaining the RFC 2544 Latency measurements would need to be added to > Section 4 (Prerequisites) of this draft to ensure that the buffer size > information is available. [acm] Section 26.2 Latency is a very primitive procedure by today's standards. It is common for testers to measure delay on every packet, and report a true average. This is enough to alert the operator to the presence of bufferbloat, but there's no assurance that the Latency benchmark measures the extent of the buffer time. > > 2. I did not see any requirements in the RFC 2544 Latency test (Section > 26.2) to deplete buffers. Did I miss something? [acm] It's an overall requirement in Section 23, after running the trial: d) Wait for two seconds for any residual frames to be received. > > 3. I would think that the trial duration MUST be increased, not just > SHOULD be increased if there is evidence of large buffer size, as buffer > depletion appears to be a necessary characteristic of this B2B > measurement. [acm] We need to keep in mind that the sentence(s) in this discussion pertain to the trials of the B2B Frame Benchmark testing, in the section that begins: Each trial in the test requires the tester to send a burst of frames (after idle time) with the minimum inter-frame gap, and to count the corresponding frames forwarded by the DUT. Let's say we have a DUT that offers 1.6 seconds of buffering. The Binary Search will increase the length of the burst until the buffer drops frames, and then try to find the longest burst where frame loss is zero. This simple procedure always waits the minimum trial duration for frames to exit the DUT. A burst of 1.7 seconds falls well within the coverage of a 2 second trial to count the frames and determine that loss has occurred. We don't need a big margin time-wise. Also, the trial duration increases with the search parameters: IOW, if the tester is still sending frames after 2 seconds, then obviously we can't allow the trial to terminate prematurely. So, I suggest we use this wording instead: The duration of the trial MUST be at least 2 seconds in addition to the time to send each burst of frames, to allow DUT buffers to deplete. and we establish an adaptive waiting time for extreme cases, consistent with RFC2544 section 23 item d). For a DUT with 1.6 second buffers, the Trial duration would be 3.6 seconds. BTW, one tester issue we are trying to fix here is the case when the frame header processing rate is sufficiently high that no buffer limit is measureable at the large frame sizes, and we implore people not to waste time with the B2B Frame Benchmark at those frame sizes. One commercial tester sent bursts up to 30 seconds in length, and then happily reported the impossible result of 30 second buffers. I want to avoid over-reacting to the buffer bloat case. We all agree that bloated buffers are bad, and knowing a DUT has buffering >=1 second may be enough. I certainly do not think that trial durations for the B2B Frame benchmark need to be as long as you originally suggested, where the overwhelming number of test conditions can use a <<1 second burst and 2 second buffer depletion time: > > > Hence, the 2 second minimum duration ought to be increased by at least > > > a factor of 10. I'd suggest changing it to 30 seconds or 60 seconds... I hope the approach of the single sentence above works for you. > > It also looks like the link to the entire slide deck didn't make it into > my original review correctly - that slide deck is at: > https://urldefense.com/v3/__http://www.taht.net/*d/lca_tcp3.odp__;fg!!BhdT > !0oWabwvUrtVlGQY7ZKnFnMa-un_e95tfCJcRKIWG7wECiJkVISb4sjd9Salwehw$ . In > addition to slide 6 from this slide deck (Figure 1 in the APNIC blog), > slide 14 is also relevant to this discussion. > > Thanks, --David > > > -----Original Message----- > > From: MORTON, ALFRED C (AL) <acm@xxxxxxxxxxxxxxxx> > > Sent: Tuesday, November 24, 2020 11:11 AM > > To: Black, David; tsv-art@xxxxxxxx > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > call@xxxxxxxx > > Subject: RE: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > > > > [EXTERNAL EMAIL] > > > > Hi David, Thanks for your review and comment! > > > > Please see a proposed resolution below, [acm] Al > > > > > -----Original Message----- > > > From: David Black via Datatracker [mailto:noreply@xxxxxxxx] > > > Sent: Tuesday, November 24, 2020 10:27 AM > > > To: tsv-art@xxxxxxxx > > > Cc: bmwg@xxxxxxxx; draft-ietf-bmwg-b2b-frame.all@xxxxxxxx; last- > > > call@xxxxxxxx > > > Subject: Tsvart last call review of draft-ietf-bmwg-b2b-frame-03 > > > > > > Reviewer: David Black > > > Review result: Ready with Issues > > > > > ... > > > > > > This draft updates the back-to-back frame testing procedure in RFC > > > 2544 to take account of experience. > > > > > > The draft is in good shape, with one notable exception in Section 5.2: > > > > > > The duration of the trial MUST be at least 2 seconds, to allow DUT > > > buffers to deplete. > > > > > > That duration of 2 seconds has been carried forward from RFC 2544 > > > without change. A 2 second duration may have been sufficient to > > > deplete buffers in 1999, but that is no longer reliably the case. For > > > example, on-site measurement of the network for the 2020 Linux > > > Conference in Australia indicated a at least 1.6 seconds of buffering, > > > as indicated by Figure 1 at > > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat- > may-be-solved-but-its-not-over- > yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0- > yK2DOVVn0ecoYjf7mFEABLmlwDk$ , > > > which is slide 6 in from the complete slide deck at: > > > > https://urldefense.com/v3/__https://blog.apnic.net/2020/01/22/bufferbloat- > may-be-solved-but-its-not-over- > yet/__;!!BhdT!wOuQE0NajXs4dT7tdIMVQU5FFpb0JiU0- > yK2DOVVn0ecoYjf7mFEABLmlwDk$ > > > . Experience with bufferbloat suggests that one network device was > > > primarily responsible. Also, see slide 14 in that slide deck. > > > > > > That 1.6 seconds measured on an actual network is entirely too close > > > to 2 seconds for confidence that buffers will be depleted in any > tested device. > > > Hence, the 2 second minimum duration ought to be increased by at least > > > a factor of 10. I'd suggest changing it to 30 seconds or 60 seconds > > > as convenient round numbers, and providing the rationale that > > > increased buffering in WiFi devices, e.g., home "routers," as > > > indicated by experience with bufferbloat measurements, is the reason > for the > > increased duration. > > > > > [acm] > > [acm] I agree with your observation that there are cases where trial > duration > > should be increased to accommodate the encountered in the DUT, but not > as a > > mandate for all testing. I have four factors in mind: > > 1. Some of the virtual network DUTs we are testing now have very small > buffers, > > and the B2B stream of frames is quite short -- less than 2000 > frames@10GE in > > some cases -- so 2 seconds fully sufficient. > > 2. The trial duration is a factor in total test duration, where each > trial is one step in > > the Binary Search. We need to manage the tension between the time needed > to > > reach a search result and confidence that we have depleted the queues. > > 3. The RFC 2544 Latency benchmark will tell us if bufferbloat is > present. > > 4. The current text says "at least 2 seconds". > > > > So I suggest adding the following text: > > > > The duration of the trial MUST be at least 2 seconds, to allow DUT > > buffers to deplete. When RFC2544 Latency measurements indicate that > > large buffers are present in the DUT, the trial duration SHOULD be > > increased to ensure that buffer depletion takes place, without > unduly > > extending the total test time. > > > > I hope this suggestion resolves your issue; thanks for highlighting it! > > Al -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
