Reviewer: Bob Briscoe Review result: Ready with Issues This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC tsv-art@xxxxxxxx if you reply to or forward this review. Last call review of https://www.ietf.org/archive/id/draft-ietf-teas-rfc3272bis-24.html The draft gives a taxonomy of traffic engineering (TE) practices, techniques and systems, including a review of recent IETF TE projects. It is a bis to RFC3272, which it obsoletes. This review categorizes comments into * SERIOUS - PLS FIX (2) * COMMENTS * NITS IMO, the comments and the nits all need to be fixed as well, but it all depends how much effort you want to put into perfecting this. I haven't pointed out nits already raised by other reviewers. And it might be hard to believe, but I have tried to limit the number of comments I've raised! ================================================================================ ==SERIOUS - PLS FIX== 1. The "Congestion Problem" >From a transport area perspective, there is one particiularly glaring omission. A very large majority of Internet traffic is either capacity-seeking or at least adaptive to available capacity. This traffic intentionally induces congestion at the path bottleneck, which is 'good congestion', because it maximizes capacity utilization and minimizes completion time. So when this draft repeatedly says that the goal of TE is to combat "The Congestion Problem", it needs to explain why one part of the IETF is trying to induce congestion while another (this draft) is trying to combat it. The explanation is that most network operators design their networks with one node per-customer (or per-customer-site) as the path bottleneck (or two nodes, if dual-homed). Then this node (typically the multi-service edge or equivalent) is where operators can focus deployment of traffic management and control functions including service differentiation, while other nodes can be overprovisioned so that they either do not need these functions at all, or they only need much-simplified functions that complement the primary controls at the edge node. Once this context has been explained, the goal of TE is indeed to /avoid/ congestion at all these other nodes, while the goal of endpoints is to /induce/ congestion at their bottleneck node (but only when they have something to send or receive - the rest of the time they are idle). Each occurrence of 'congestion problem' will then need to be qualified. Eg: * "Clearly, congestion is highly undesirable." * "Congestion is one of the most significant problems in an operational IP context." * "If traffic from a source to a destination exceeds the capacity of a link along the shortest path, the link (and hence the shortest path) becomes congested while a longer path between these two nodes may be under-utilized." [Given latency is important to many / most applications, if throughput is sufficient, it would be wrong to 'solve' this 'problem' by using the longer path. The solution would be to minimize the delay that results from congestion by using the latest queue management techniques.] 1.1 Definition of Congestion Quoting 3 instances of similar wording: * "...traffic incident on the resource exceeds its output capacity over an interval of time." * "...for an interval of time, the arrival rate of packets exceeds the output capacity of the resource." * "...sustained overload over an interval of time." These statements define just the extreme of congestion, not all congestion. If the input exceeded the output for a sustained interval of time, the queue would very quickly overflow the buffer and there would be sustained high levels of loss. That is sustained overload, not just 'congestion'. Congestion controlled traffic ensures that the input and the output are roughly matched on average, alternating which is highest at a fast timescale. This is congestion, even though the average queue remains stable. The more congestion controlled /flows/ there are, the faster the queue cycles, so the loss fraction is higher. But input and output are still matched on average (contrary to the definition given). Even if there is a high proportion of unresponsive traffic alongside the congestion controlled traffic, input will not exceed output over a sustained interval unless the sum of the unresponsive traffic /alone/ exceeds capacity. The normal percentage of unresponsive traffic in today's Internet is probably under 1% (since QUIC became prevalent, it has not been so easy to measure how much UDP traffic is unresponsive). ================================================================================ ==COMMENTS== GENERAL COMMENTS 2. Current Practice? §5.1 on "IETF projects related to TE" needs to say that not all these practices have been widely adopted in practice, and some are still too immature to know whether they will be adopted. Otherwise, on reaching §8 "Contemporary TE practices", an obvious question arises: "Well, what were the previous 56 pages about?" There are sometimes hints in the wording when a technology is not yet adopted or deployed, e.g. the use of phrases like 'can be useful', 'may be used', but I guess it's hard to pass judgement on likely deployment. Content Distribution (§5.2) is the only TE technique included that is not within the IETF project section. I suggest the following should also be included: * ECMP. This is described rather disparagingly in §6.2 under routing recommendations as if it is not good enough. However it is widely used with n-stage Clos topologies (with n=2 or higher), precisely because it is considered good enough (i.e. cost-effective) by many major operators. §8 says "Service providers apply many of the TE mechanisms described in this document..." I think it also needs to be said that many service providers do not. SECTION-BY-SECTION COMMENTS §1. Intro "...a preponderance of Internet traffic tends to originate in one autonomous system and terminate in another," This assertion (inherited from RFC3272) needs an up-to-date reference. I thought the opposite had been true for a decade or more, but I have no hard measurement evidence other than Arbor's study in 2010 (Craig Labovitz et al, ACM CCR), which found that the majority of inter-domain traffic was flowing to CDNs, and I figured one could assume that most CDN content would then be served multiple times intra-domain. §1.1 What is Internet TE? It might be useful to give examples of practices that are /not/ TE. For instance, "Other functions that regulate the flow of traffic through the network" surely includes endpoint congestion control algorithms (CCAs), which I don't think this definition intended to include. I was surprised to find that capacity planning is categorized as TE - I thought TE was what you do within capacity contraints. Similarly, I didn't think queue management and scheduling would be categorized as TE. Given the document doesn't discuss different types of scheduling at all, and its discussion of AQM is outdated (see later), it doesn't seem wise to have cast the scope so wide. Also, although these definitions are preceded by 'In this document' it might be worth saying what some other definitions of TE exclude. For instance, until reading this draft, I thought TE was defined as: "Traffic engineering (TE) is a process whereby a network operator can engineer the paths used to carry traffic flows that vary from those chosen automatically by the routing protocol(s) in use in that same network." [Tom Nadeau, "Offline Traffic Engineering", MPLS Network Management (2003)]. §1.2 Components of Traffic Engineering This new section seems at odds with the very similar existing section "Traffic Engineering Process Models" (§3) and particularly §3.1, which opens with "The key components of the traffic engineering process model are as follows." Why was it necessary to add §1.2 with similar scope, but different content? Unlike §3 it doesn't list 'Measurement' or 'Analysis' as a component of TE, which surely can't be correct. And it implies that resource reservation is the only path steering approach. "Examples of resources are bandwidth, buffers, and queues,..." A queue is not a resource. The buffer is the resource, and the queue uses it. "...rate-shaping mechanisms that are typically supported via queuing." In aggregates, using queuing for rate shaping has become less acceptable than using dropping nowadays. On a related note, traffic mapping is first mentioned in §6.3. Should it not be mentioned earlier as one of the components of TE? §1.3 Scope It seemed odd to call the subject of the draft 'Internet TE', then define the scope as intra-domain, given Internet means Internetwork. It might be worth explaining that this is intended to mean 'TE of the Internet service' not 'TE of the Internet'. The draft focuses nearly exclusively on MPLS as the transport technology (Ethernet transport is mentioned a couple of times, but only in passing). It might be worth saying this in the scope section. §2.4.1 Combating the Congestion Problem Under short timescale, a lengthy passage about AQM is provided (and no other examples of short timescale technologies are given). This seems out of place at this point, where no other technology is described in such depth. It would seem more consistent to have a subsection on AQM in §5, and refer forward to it from here. Having said that, the text on AQM is outdated in three respects: RED, TCP and LQD. a) RFC7567 (which is a BCP and cited in this section) effectively deprecates RED, as follows: "With an appropriate set of parameters, RED is an effective algorithm. However, dynamically predicting this set of parameters was found to be difficult. As a result, RED has not been enabled by default, and its present use in the Internet is limited. Other AQM algorithms have been developed since RFC 2309 was published, some of which are self-tuning within a range of applicability. Hence, while this memo continues to recommend the deployment of AQM, it no longer recommends that RED or any other specific algorithm is used by default. It instead provides recommendations on IETF processes for the selection of appropriate algorithms, and especially that a recommended algorithm is able to automate any required tuning for common deployment scenarios." However, it is understandable that this draft needs to introduce RED, because it is the basis of WRED, which this text is working towards introducing as part of Diffserv AF. So it would be perhaps best to say sthg like: "RFC7567 recommends self-tuning AQM algorithms like those that the IETF has since published [RFC8290, RFC8033, RFC8034, RFC9332], but RED is still appropriate for links with stable bandwidth, if configured carefully." b) TCP is no longer an appropriate byword for 'responsive traffic', and UDP is no longer a byword for unresponsive traffic, both given the growing prevalence of QUIC over UDP (and of SCTP, DCCP). Pls search the draft for multiple occurrences. c) Even at the time RFC3272 was written, it says LQD was theoretical. If a deployed ref would be preferred, I suggest AFD, which was implemented by cisco, at least, and is still available AFAICT. Pan, R.; Breslau, L.; Prabhaker, B. & Shenker, S. "Approximate Fairness Through Differential Dropping" ACM SIGCOMM Computer Communication Review, 2003, 33, 23-40 §5.1. Overview of IETF Projects Related to Traffic Engineering After §5.1.1 on Intserv (or after the section on scalability within it), it would surely be worth mentioning Pre-Congestion Notification (PCN) https://datatracker.ietf.org/wg/pcn/documents/ , which solves the scaling problems of Intserv by using measurement-based admission control (and flow termination to handle failures) between edge-nodes. Nodes between the edges of the internetwork have no per-flow operations and the edge nodes can use RSVP per-flow or per-aggregate. It was implemented by a number of equipment vendors. Also, in §5.1.5 on DETNET, it would be worth mentioning that DETNET would suffer from the same scaling problems described in the Intserv section, but DETNET's domain of applicability is considered small enough for this to be acceptable. §5.1.1.2. Differentiated Services "The Diffserv model deals with traffic management issues on a per hop basis. ... Other TE capabilities, such as capacity management (including routing control), are also required in order to deliver acceptable service quality in Diffserv networks" The above-quoted para is problematic: i) Diffserv is not solely per-hop (which contradicts the complementary mix of domain edge and per-hop functions explained 2 paras earlier). ii) Routing control is not /required/ to deliver acceptable service quality - other techniques, e.g. liberal provisioning, can preserve service after shortest path reroutes around failures. §5.1.1.4. (Layer-4) Transport-Based TE When the draft says no support for ATSSS splitting has yet been developed for QUIC, it would be worth explaining why (e2e cryptographic protection), and possibly referencing multipath QUIC [draft-ietf-quic-multipath]. It seems rather odd to say so much about QUIC (which ATSSS does not support) and so little about MPTCP (which ATSSS does use). §5.1.2.3. Network Slicing This section seems out of scope or at best aspirational - should it be deleted? It admits itself that "IETF network slices are not, of themselves, TE constructs. However, a network operator that offers IETF network slices is likely to use many TE tools in order to manage their network and provide the services." Further, it doesn't point to any work on how this might be done, particularly what information visibility would be necessary to coordinate multiple slices. §5.1.3.7. Flow Measurement The RTFM WG concluded in Oct 2000. Should the draft not discuss IPFIX (the open standards development of Cisco's Netflow), which ran from 2001-2015? See the comparison with RTFM at https://datatracker.ietf.org/doc/html/rfc5472#section-3.6 Since that comparison, IPFIX was developed a lot further too; see https://datatracker.ietf.org/group/ipfix/documents/ . §5.1.3.8. Endpoint Congestion Management (and the omission of a section on multipath L4 transports) §2.3.1 says endpoint congestion control is not in primary scope. But, surely, if the draft includes this fairly outdated example of purely endpoint coordination across flows, there should be a full sub-section on multipath transport protocols, which are currently used by in-network control as well as endpoint control (rather than subordinating multipath within the section on ATSSS, which is just one in-network example of the use of multipath transports)? Then the ATSSS section could cross-refer to the new multipath subsection instead of having to include it. The idea of multipath L4 transport was originally developed as an improvement over existing TE techniques, whether deployed solely on endpoints, or with in-network control. At minimum, the original rationale for adding a multipath capability to L4 transport protocols should be referenced: Wischik, D.; Handley, M. & Bagnulo Braun, M. The Resource Pooling Principle SIGCOMM Comput. Commun. Rev., ACM, 2008, 38, 47-52 When I was in BT, an ex-colleague, the late Peter Key, calculated that in-network traffic engineering would become redundant, if at least about 6% of traffic used multipath at L4. (6% is from memory 'cos I can't find his paper on it - pls don't quote it.) §6.1. Generic Non-functional Recommendations Stability: I suspect this para might be talking about flap, but I don't believe I've seen the problem spelled out, so perhaps it should be here. That is, when endpoint CCAs (or TE systems in neighbouring domains) interact with TE such that, when the TE of one domain moves an aggregate, CCAs rapidly restore the original imbalance, possibly causing the TE system to flap. §9. Security Considerations Shouldn't it be said here that external control interfaces (e.g. ALTO and the other approaches in §5.1.2) have to trade off providing flexibility to customers with opening up control of a network's internals to potentially malicious actors. General Comment Jitter? <RANT> In lists of important traffic characteristic (as in the definition of QoS) pls consider replacing 'jitter' with '99th percentile delay' or another high percentile e.g. 99.9th. Jitter was only relevant when many end devices were analogue. Once the vast majority of devices have memory buffers, the only relevant delay metrics characterize the tail of the distribution. In contrast, jitter is overwhelmingly driven by the shape of the body of the delay distribution, which bears no relation to the tail. Because jitter does not and cannot characterize the seriousness of the actual delay that a buffered receiver will play out, it just confuses everyone into seeing problems where there are none, and missing where the real problems are. </RANT> ================================================================================ ==NITS== §1.1 What is Internet TE? "...utilizing network resources without waste": too strong; how about "without excessive waste"? "while reacting to the real-time statistical behavior of traffic" -> "while reacting to statistical measures of the real-time behavior of traffic" (There's a similar problem in §5.1.2.2. with the phrase "statistical packet bandwidth") in the later case -> latter §1.2 Components of Traffic Engineering Standard TE solutions -> TE solutions §1.4. Terminology Please include explanations of the following terms, which have confusable alternative meanings: * 'global' is invariably used (17 occurrences) in the sense of domain-wide (although it clearly means globe-wide in phrases like 'the global Internet infrastructure', "global network provider" and "globally interconnected network"). The phrase 'global synchronization' is an exception to both cases. * 'end-to-end' is used in the sense of edge-to-edge, contrary to the common IETF (or at least transport and application area) use of the phrase meaning application endpoint to application endpoint. * 'transport' is generally used to mean below IP, as one would expect in a TE document. But it is also used to describe L4 protocols, e.g. the title and content of §5.1.1.4. "Transport-Based TE" and in §5.1.3.8. "Endpoint Congestion Management". I suggest 'layer-4 transport' is used in these latter cases. And some comments on existing definitions: * Effective bandwidth - although the definition given is not incorrect it is not as precise as the mathematical definition, so perhaps a reference should be added, e.g. F. Kelly. Notes on effective bandwidths. In Stochastic Networks: Theory and Applications. Oxford University Press, 1996. * Hot-spot - better defined as an element or sub-system in a considerably higher state of congestion /than others/. * Inter-domain is defined, but intra-domain is not (perhaps both are obvious and neither is needed?). * Offline/online TE are defined as "exists outside of/within the network", but surely they're defined by when they operate, not where (e.g. online TE can be located outside the network, e.g. SDN). * Traffic flow: It defines flow as between 2 endpoints, and says a common classification for a flow is a 5-tuple. However, wherever 'flow' is used in the body of the document, I'm pretty sure it is more likely to be intended to mean an aggregate flow. So the way this definition is worded has great potential for confusion. Flow-size distribution. On a related point, it would be useful to explain that the very large majority of 5-tuple flows are very brief (mice - indeed single-packet flows massively predominate). So a number of TE techniques are designed to shift elephant flows around, or to shift aggregates likely to contain elephants. The practicality of numerous technologies described in this draft depends heavily on the definition of 'flow'. * southbound - perhaps needs a definition? §2.2 Network Domain Context "This requirement is clarified in [RFC2475] which also provides an architecture for Differentiated Services (Diffserv)." Suggest 'also' is removed. §2.4 Solution Context "A collection of online and possibly offline tools and mechanisms for measurement, characterization, modeling, and control of traffic, and control over the placement and allocation of network resources, as well as control over the mapping or distribution of traffic onto the infrastructure." This list needs to be split, 'cos measurement cannot be offline. §2.4.1 Combating the Congestion Problem "Many of these adaptive schemes rely on measurement systems." -> "These adaptive schemes rely on measurement systems." [How could an adaptive scheme not rely on measurement?] "RED provides congestion avoidance which is not worse than traditional Tail-Drop (TD) queue management." not worse -> better [I don't think the intention was to damn with faint praise]. "RED reduces the possibility of global synchronization where retransmission bursts become synchronized across the whole network" Global synchronization only means synchronization between all the flows sharing the same bottleneck, not the whole network. Also it's primarily about synchronization of the sawtoothing window variations of each flow; retransmissions will not synchronize unless the paths all have the same RTT. "All the policies described above for the long and medium time scales can be categorized as being reactive." Given the shorter the timescale, the more it is likely that a solution will be reactive, this odd choice of sentence conveys the opposite impression, even though it is strictly not incorrect. For instance, long timescale activities like capacity expansion certainly /can/ be reactive in theory, but normally they are preventative. §2.5 Implementation and Operational Context "The operational context of Internet TE is characterized by constant changes that occur at multiple levels of abstraction." I think this intended to say multiple levels of granularity [something can be described or modelled at a level of abstraction, but surely real changes do not occur at a level of abstraction]. Similarly, in § 3.1 "Measurement in support of the TE function can occur at different levels of abstraction." -> granularity §4.1 Time-Dependent Versus State-Dependent Versus Event-Dependent "learning models, as in the success- to-the-top (STT) method" [needs a ref & perhaps a brief description] "a fully functional TE system is likely to use all aspects of time-dependent, state-dependent, and event-dependent methodologies as described in Section 4.1." [Shouldn't this point be made in §4.1?] §4.3.2. Considerations for Software Defined Networking "...SDN control plane often determines the end-to-end path ..." [end-to-end implies more than intra-domain - seems too strong] §4.6. Open-Loop Versus Closed-Loop "feedback information may be in the form of historical information" [Surely that would not be described as closed loop?] §5.1.2.1. Application-Layer Traffic Optimization "...allows a network to publish its network information such as network locations, costs between them at configurable granularities, and end-host properties to network applications." This sentence is hard to parse. Perhaps change the order and/or flag the items in the list with a), b), c). §5.1.2.2. Network Virtualization and Abstraction "statistical packet bandwidth" [does this mean some form of effective bandwidth?] §5.1.3.2. RSVP "RSVP has been extended to reserve resources for aggregation of flows" [Cite RFC3175?] §5.1.3.4. RSVP-TE "...the paths of LSPs" -> LSPs [the P already stands for path] "To determine the paths for P2MP LSPs, selection of the branch points (based on capabilities, network state, and policies) is key." [This problem is left dangling. Is there at least a reference giving a solution?] §5.1.3.5. Generalized MPLS (GMPLS) "TE extensions to MPLS (see Section 5.1.3.3)." -> 5.1.3.4 "These additions impact basic LSP properties: ..." [Again, this problem is left dangling. Is there at least a reference giving solutions to the ensuing list of apparently fundamental problems?] §5.1.3.12. Segment Routing "...global context (network wide)" [this potentially gives the impression of an Internet-wide lookup capability] "BIER-TE does not of itself offer traffic engineering..." -> BIER-TE does not offer a complete traffic engineering system... [Rather misleading - perhaps better would be to move the sentence from the next para here: "...steers the traffic within the network and forms an element of a traffic engineering system."] §6. Recommendations for Internet Traffic Engineering The order of some of the sub-sections seems odd (e.g. 'measurement' and 'traffic mapping' after 'routing') and contrary to more logical ordering of the TE process model in §3. §6.1. Generic Non-functional Recommendations "...a TE system should remain functional as the network expands with regard to the number of routers and links, and with respect to the traffic volume." [traffic volume -> number of flows (I don't believe the same number of flows but carrying more volume would impact TE scaling at all)] §6.4 Measurement Recommendations hot-spot -> hot spot [consistent with 2 other occurrences] §6.5. Policing, Planning, and Access Control "This is a simple way to check that the actual traffic volumes are consistent with the planned volumes." [check -> enforce/ensure] §6.6. Network Survivability "Network capacity reserved in one layer to provide protection and restoration is not available to carry traffic in a higher layer: it is not visible as spare capacity in the higher layer." [Unclear whether these are statements of fact or recommendations. Perhaps 'is' -> 'should be' (twice)] §7. Inter-Domain Considerations a are -> are "it is generally considered inadvisable for one domain to permit a control process from another domain to influence the routing and management of traffic in its network." [Surely this is inescapable, if TE in one domains moves traffic to a different ingress of the next domain. Or is this intended to mean 'Don't open up an explicit control interface for other domains"?] Could mention that L4 multipath transport protocols (whether controlled by endpoints or in-network ) were designed to shift traffic between domains (and they are doing so). §8. Overview of Contemporary TE Practices in Operational IP Networks This section presents rather a large wall of unbroken text. Navigation markers would be useful, perhaps based on the list in the 2nd para (altho I couldn't see them all in the text). §13. Informative References [Floyd94] -> [RFC3168] §A.2 This Document §5.1.3.4 & §5.1.3.13 are missing. General (multiple sections) I found a number of cases of repetition - probably just a symptom of age (of the draft, not the editor ;) * The definition of congestion was given 3 times, as already listed earlier; Repeated explanation of the distinction between reactive and proactive/preventative: * "...can be both pro-active and reactive. In the pro-active case, the TE control system takes preventive action..." * Reactive Versus Preventive Congestion Management Schemes (the main bullet item on this distinction) * Network performance optimization can be corrective or perfective. In corrective optimization,... In perfective... * Prescriptive TE can be further categorized as either corrective or perfective. Corrective TE prescribes ... Perfective TE... (BTW, no hyphen in proactive.) Extensions to link-state routing protocols are repeatedly listed and explained: * "Examples of protocol extensions used to advertise network link state information are defined in [RFC5305], [RFC6119], [RFC7471], [RFC8570], and [RFC8571]." * "taking into consideration the prevailing network state as advertised by IGP extension for IS-IS in [RFC5305], for OSPFV2 in [RFC3630], and for OSPFv3 in [RFC5329]" * "[RFC5305] describes the extensions to the Intermediate System to Intermediate System (IS-IS) protocol to support TE, similarly [RFC3630] specifies TE extensions for OSPFv2, and [RFC5329] has the same description for OSPFv3." * "A number of enhancements to the link state IGPs allow them to distribute additional state information required for constraint-based routing. The extensions to OSPF are described in [RFC3630], and to IS-IS in..." I found the number of sentences that gratuitously contained 'X or not X' started to irritate. A taxonomy will naturally divide practice into alternatives, but there were a number of cases where such phrases were not used to divide up the taxonomy, but seemed to be just woffle that could be removed completely without subtracting anything. Some examples (some are only marginally gratuitous): Some TE solutions rely on these elements to a lesser or greater extent. This determination may be made at a very coarse or very fine level. Metrics that provide quantitative or qualitative measures may allow the settings of the traffic control mechanisms to be manipulated by external or internal entities Delivery requirements of a specific set of packets may be specified explicitly or implicitly. derivation of solutions which may be implicitly or explicitly formulated This process model may be enacted explicitly or implicitly An SLA may explicitly or implicitly specify a Traffic Conditioning Agreement using a set of shared or dedicated network resources -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
