Hi Tommy and BMWG Subscribers,
Happy New Year! Apologies for the late response due to the
Christmas break.
Tommy, thank you for your review of the next-generation firewall
benchmarking draft and for your comments submitted on Dec 17.
In testing, it is important to document any tiny detail of the
test setup and environment. The goal is to ensure reproducibility
and comparability of results. An under-specification of details
was one of the issues with RFC3511. Configurable TCP parameters
can highly influence benchmarking test results. With the goal to
achieve reproducible test results we decided to specify TCP
parameter values, by the way aligned with defaults of most of
today's operating systems.
Using normative language is the only way from the editors' point
of view to achieve a level-playing field for all users (vendors,
network operators, end users) to always achieve comparable results
when testing the same system/device. In the whole review (more
than 20 drafts over three years in total), the normative language
was extensively reviewed and found adequate.
Ensuring that the test bed is up to the task (enabling performance benchmarking of a firewall) and not hampered by its own limitations is another important step (section 4.1). The editors confirm that it is normatively required for the test bed to be adequate for the performance tests. If you find that sub-par test beds with inherent issues in throughput and latency should be permitted, please explain why.
Thank you for noticing the incorrect long name for QUIC in section 4.3.1.3. We were unaware that the long name has been omitted by IETF policy. Thus, we would edit the respective sentence and remove the long name. Furthermore, could you please point us to the RFC specifying that HTTP/3 is by definition equal to HTTP over QUIC? There was a 2018 informal IETF statement, but we could not find this mentioned as normative language in RFC9000.
We will make an editorial change for clarification, adding HTTP/3
to the server side (section 4.3.2.3).
The benchmarking methodology could be complemented with synthetic loss/delay, but this is out of scope of this draft. In fact, adding synthetic frame loss in TCP test environments often creates results that are quite cumbersome to interpret. We would not recommend it.
The choice of TCP as a transport protocol has been made
deliberately, representing what the editors and contributors
deemed realistic test methodology for NGFWs. Contributions for
other transports are welcome in BMWG I would guess; since this
would require another 20-30 pages from the point of view of the
editors, probably a fresh start with a new draft, complementing
the upcoming RFC would suit the purpose best.
Regarding the use of "throughput" in section 7.7 and elsewhere,
it is defined as "Inspected Throughput" in section 6.3.
Throughput is defined on layer 2 because that is the overwhelming
informal understanding when qualifying a switch, router, or
firewall. We had quite extensive discussions on this topic; you
may want to consult with the IETF mailing list archive and WG
session recordings.
Best regards, Carsten
Reviewer: Tommy Pauly Review result: On the Right Track This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC tsv-art@xxxxxxxx if you reply to or forward this review. This document is a useful update to previous work, and it’s good to see this methodology being refreshed. Thanks to the authors and the WG for their work on this! >From a transport perspective, I’m concerned that some things are over-specified (details of TCP implementations) and others are underspecified (how throughput is measured, how loss and delay are tested). There are some nods towards non-TCP transports (UDP port configuration and HTTP/3 tests), but these are inconsistent. I’d like to see transports (TCP/UDP/QUIC/other) be treated more consistently throughout the document, particularly since non-TCP traffic will become increasingly relevant for the devices these tests are targeting. Section 4.1 says: “Testbed configuration MUST ensure that any performance implications that are discovered during the benchmark testing aren't due to the inherent physical network limitations such as the number of physical links and forwarding performance capabilities (throughput and latency) of the network devices in the testbed.” This seems like a hard thing to make normative. It’s not about interoperability, but rather talking about the necessary conditions to get useful measurements. I’d suggest a non-normative phrasing that explains the consequence of not checking this. Overall, I suggest reviewing your use of normative language, and prefer to use non-normative language when you are just describing facts rather than necessary actions on the behalf on an implementation or test setup. Also in section 4.1: can (DUT/SUT) be defined in the sentence earlier, when the terms are first introduced? It would be good to explain/define “fail-open” behavior. I assume I know what you mean, but there can be various types of failing open. In the criteria to match in Figure 3, the Transport layer discusses filtering on TCP/UDP ports, and the IP layer discusses filtering on addresses. There is no mention, however, of how alternate transport IP protocols (like SCTP, not TCP or UDP) would be filtered, or how such a security device may have separate rules for different transport protocols that run over UDP (like QUIC and SCTP-over-UDP) could be treated. This may not be standard practice, but it may be interesting and useful to point to. The client configuration section 4.3.1.1 details TCP stack configuration, but does not address other transports. Discussing QUIC seems like it will be relevant soon. Overall, for this section, I am struck that there’s a lot of detail that seems over-specified, with lots of normative language. For example, the TCP connection MUST end with a three- or four-way handshake. What if there’s a RST? I don’t understand what we’re requiring of these TCP implementations apart from being a functional and compliant TCP implementation. How much of this is actually required? Also, RFC 793 is not provided as a link, but is just a text reference. In Section 4.3.1.3, there are problems with the references to QUIC. QUIC is not an acronym for “Quick UDP Internet Connections”; that should be removed. Also, the text contrasts HTTP/2 and QUIC. If you are comparing to HTTP/3, please reference the draft (and soon-to-be RFC) for HTTP/3, not QUIC. It also doesn’t make sense to say “if you are testing HTTP/3, you MUST use QUIC”. HTTP/3 is definitionally HTTP over QUIC, so this normative requirement is unnecessary—just as it would be unnecessary to say that HTTP/2 MUST run over TCP. The client sections mentions HTTP/3, but that is not mentioned for the server (Section 4.3.2.3). Should the tests include results when including synthetic loss or delay to better emulate realistic network conditions? I understand many of the recommendations to remove uncontrolled delay and loss (in section 5), but various transport properties will change with loss and delay (and congestion, and buffering), which will influence performance of the DUT. Ideally, these DUTs would perform well in those conditions as well. Section 6.3 is again quite TCP-centric, and should be expanded for other transports. For metrics like “HTTP throughput” (section 7.7), how is throughput being rigorously defined? Is this measuring TCP bytes, TLS stream bytes, or actual application payload bytes?
-- Carsten Rossenhövel Managing Director, EANTC AG (European Advanced Networking Test Center) Salzufer 14, 10587 Berlin, Germany office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721 cross@xxxxxxxx, https://www.eantc.de Place of Business/Sitz der Gesellschaft: Berlin, Germany Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany EU VAT No: DE812824025
-- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
