Re: [Last-Call] Tsvart last call review of draft-ietf-bmwg-ngfw-performance-12

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Tommy and BMWG Subscribers,

Happy New Year!  Apologies for the late response due to the Christmas break. 

Tommy, thank you for your review of the next-generation firewall benchmarking draft and for your comments submitted on Dec 17. 

In testing, it is important to document any tiny detail of the test setup and environment.  The goal is to ensure reproducibility and comparability of results.  An under-specification of details was one of the issues with RFC3511.  Configurable TCP parameters can highly influence benchmarking test results.  With the goal to achieve reproducible test results we decided to specify TCP parameter values, by the way aligned with defaults of most of today's operating systems. 

Using normative language is the only way from the editors' point of view to achieve a level-playing field for all users (vendors, network operators, end users) to always achieve comparable results when testing the same system/device.  In the whole review (more than 20 drafts over three years in total), the normative language was extensively reviewed and found adequate.

Ensuring that the test bed is up to the task (enabling performance benchmarking of a firewall) and not hampered by its own limitations is another important step (section 4.1).  The editors confirm that it is normatively required for the test bed to be adequate for the performance tests.  If you find that sub-par test beds with inherent issues in throughput and latency should be permitted, please explain why.

Thank you for noticing the incorrect long name for QUIC in section 4.3.1.3.  We were unaware that the long name has been omitted by IETF policy.  Thus, we would edit the respective sentence and remove the long name.  Furthermore, could you please point us to the RFC specifying that HTTP/3 is by definition equal to HTTP over QUIC?  There was a 2018 informal IETF statement, but we could not find this mentioned as normative language in RFC9000.

We will make an editorial change for clarification, adding HTTP/3 to the server side (section 4.3.2.3).

The benchmarking methodology could be complemented with synthetic loss/delay, but this is out of scope of this draft.  In fact, adding synthetic frame loss in TCP test environments often creates results that are quite cumbersome to interpret.  We would not recommend it.

The choice of TCP as a transport protocol has been made deliberately, representing what the editors and contributors deemed realistic test methodology for NGFWs.  Contributions for other transports are welcome in BMWG I would guess; since this would require another 20-30 pages from the point of view of the editors, probably a fresh start with a new draft, complementing the upcoming RFC would suit the purpose best.

Regarding the use of "throughput" in section 7.7 and elsewhere, it is defined as "Inspected Throughput" in section 6.3.  Throughput is defined on layer 2 because that is the overwhelming informal understanding when qualifying a switch, router, or firewall.  We had quite extensive discussions on this topic; you may want to consult with the IETF mailing list archive and WG session recordings. 

Best regards, Carsten


Am 12/17/2021 um 6:18 PM schrieb Tommy Pauly via Datatracker:
Reviewer: Tommy Pauly
Review result: On the Right Track

This document has been reviewed as part of the transport area review team's
ongoing effort to review key IETF documents. These comments were written
primarily for the transport area directors, but are copied to the document's
authors and WG to allow them to address any issues raised and also to the IETF
discussion list for information.

When done at the time of IETF Last Call, the authors should consider this
review as part of the last-call comments they receive. Please always CC
tsv-art@xxxxxxxx if you reply to or forward this review.

This document is a useful update to previous work, and it’s good to see this
methodology being refreshed. Thanks to the authors and the WG for their work on
this!

>From a transport perspective, I’m concerned that some things are over-specified
(details of TCP implementations) and others are underspecified (how throughput
is measured, how loss and delay are tested). There are some nods towards
non-TCP transports (UDP port configuration and HTTP/3 tests), but these are
inconsistent. I’d like to see transports (TCP/UDP/QUIC/other) be treated more
consistently throughout the document, particularly since non-TCP traffic will
become increasingly relevant for the devices these tests are targeting.

Section 4.1 says: “Testbed configuration MUST ensure that any performance
implications that are discovered during the benchmark testing aren't due to the
inherent physical network limitations such as the number of physical links and
forwarding performance capabilities (throughput and latency) of the network
devices in the testbed.”

This seems like a hard thing to make normative. It’s not about
interoperability, but rather talking about the necessary conditions to get
useful measurements. I’d suggest a non-normative phrasing that explains the
consequence of not checking this.

Overall, I suggest reviewing your use of normative language, and prefer to use
non-normative language when you are just describing facts rather than necessary
actions on the behalf on an implementation or test setup.

Also in section 4.1: can (DUT/SUT) be defined in the sentence earlier, when the
terms are first introduced?

It would be good to explain/define “fail-open” behavior. I assume I know what
you mean, but there can be various types of failing open.

In the criteria to match in Figure 3, the Transport layer discusses filtering
on TCP/UDP ports, and the IP layer discusses filtering on addresses. There is
no mention, however, of how alternate transport IP protocols (like SCTP, not
TCP or UDP) would be filtered, or how such a security device may have separate
rules for different transport protocols that run over UDP (like QUIC and
SCTP-over-UDP) could be treated. This may not be standard practice, but it may
be interesting and useful to point to.

The client configuration section 4.3.1.1 details TCP stack configuration, but
does not address other transports. Discussing QUIC seems like it will be
relevant soon.

Overall, for this section, I am struck that there’s a lot of detail that seems
over-specified, with lots of normative language. For example, the TCP
connection MUST end with a three- or four-way handshake. What if there’s a RST?
I don’t understand what we’re requiring of these TCP implementations apart from
being a functional and compliant TCP implementation. How much of this is
actually required? Also, RFC 793 is not provided as a link, but is just a text
reference.

In Section 4.3.1.3, there are problems with the references to QUIC. QUIC is not
an acronym for “Quick UDP Internet Connections”; that should be removed. Also,
the text contrasts HTTP/2 and QUIC. If you are comparing to HTTP/3, please
reference the draft (and soon-to-be RFC) for HTTP/3, not QUIC. It also doesn’t
make sense to say “if you are testing HTTP/3, you MUST use QUIC”. HTTP/3 is
definitionally HTTP over QUIC, so this normative requirement is
unnecessary—just as it would be unnecessary to say that HTTP/2 MUST run over
TCP.

The client sections mentions HTTP/3, but that is not mentioned for the server
(Section 4.3.2.3).

Should the tests include results when including synthetic loss or delay to
better emulate realistic network conditions? I understand many of the
recommendations to remove uncontrolled delay and loss (in section 5), but
various transport properties will change with loss and delay (and congestion,
and buffering), which will influence performance of the DUT. Ideally, these
DUTs would perform well in those conditions as well.

Section 6.3 is again quite TCP-centric, and should be expanded for other
transports.

For metrics like “HTTP throughput” (section 7.7), how is throughput being
rigorously defined? Is this measuring TCP bytes, TLS stream bytes, or actual
application payload bytes?


-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
cross@xxxxxxxx, https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025
-- 
last-call mailing list
last-call@xxxxxxxx
https://www.ietf.org/mailman/listinfo/last-call

[Index of Archives]     [IETF Annoucements]     [IETF]     [IP Storage]     [Yosemite News]     [Linux SCTP]     [Linux Newbies]     [Mhonarc]     [Fedora Users]

  Powered by Linux