I would put each of my comments in the etherpad into one of three buckets: observations and commentary, minor points and clarifications, and points that I think require more discussion. I'm only really going to address the last bucket: the others I raised but don't feel particularly strongly about, at least at this late stage in the document's life.
(section 3.2 of [RFC7525]), essentially preventing the negotiation
process resulting in fallback to the use of clear text. In other
cases, some service providers have relied on middle boxes having
access to clear text for the purposes of load balancing, monitoring
for attack traffic, meeting regulatory requirements, or for other
purposes. These middle box implementations, whether performing
functions considered legitimate by the IETF or not, have been
impacted by increases in encrypted traffic. Only methods keeping
with the goal of balancing network management and PM mitigation in
[RFC7258] should be considered in solution work resulting from this
document.
KR> I feel like this section could be better organized by:
* Moving the examples to 1.1 as a bulleted list of sample situations in which network operators attempted to and/or succeeded in defeating encryption to preserve existing operational mechanisms, or in which performance suffered for users (whether of the encrypted flows or of other flows impacted by encrypted flows).
KM> Interesting point, but we'd need more examples. I'll think about this more and chat with Al in case he has ideas. For now, I went with Brandon's easier suggestion, but moving to this would be nice for the document readers.
AM> Although I see how these examples could be part of the background, I think those who will
eventually remove their objections will prefer the reduced emphasis on these examples where
they are (in section 2). In one view, the entire memo is background, since nothing new is proposed.
KR>
* Using this section as an introduction to the methodology for cataloging operational mechanisms depending on cleartext traffic monitoring, with the various caveats on what will be considered (e.g., only mechanisms required heretofore for operability), and for describing the approach to seeking mitigations and/or substitutions.
KM> Hmm, interesting point. I'll have to think about this more as it could be alot of work at this stage.
AM> Unfortunately, we've already implemented many AD-level suggestions on the organization of Section 2.
We're at the stage of *what can everybody live with", and re-re-re-org falls out now, IMO.
This is a reasonable objection, but I am mostly concerned about satisfying the target audience. ISTM that audience is something like "IETF participants who are skeptical about, or ignorant of, the operational difficulties posed by widespread encryption of flows". If the document meets your goals as-is, then further refinement is unnecessary. Does it? Or is it impossible to know at this point?
AM> Also, the neutral exposition that we've been asked to provide a million times actually
comes from multiple perspectives expressed in contributions that we would combine
in a balanced way, without value judgements (no good or bad).
Where we lack balance, we lack specific contributions.
"Neutral" meaning "without advocacy"? It's a fine line. The document still has a thesis, which (if I am reading it correctly) is to inform the community of current operational practices that are encumbered in some way by encryption. To wit, q( It provides network operators' perspectives about the motivations and objectives of those practices as well as effects anticipated by operators as use of encryption increases. ) Without that thesis, which some might interpret as advocacy, it's not clear the document would have enough focus to be useful. I'd like folks to see this doc as *at worst* devil's advocacy, and preferably as a challenge to come up with better arguments and to find alternative methods for dealing with network management problems.
heuristics grows, and accuracy suffers. For example, the traffic
patterns between server and browser are dependent on browser supplier
and version, even when the sessions use the same server application
(e.g., web e-mail access). It remains to be seen whether more
complex inferences can be mastered to produce the same monitoring
accuracy.
KR> This might be too formal of an approach for this doc, but it might be possible to construct a taxonomy of layers of metadata made unavailable by encryption at each layer to show the completeness/comprehensiveness of the survey. So, for instance:
* Protocol and port number are still available as a way of characterizing traffic over the public internet even if the payload is encrypted, but this information is lost if (e.g.) the traffic is traversing an IPsec tunnel or if radically different kinds of traffic all use port 443/tcp without any other way to distinguish between them.
* TCP is open to optimization/measurement even if using TLS, except when tunneled encrypted: congestion signals (like rexmits) previously transparent to the middlebox, for instance, are then lost.
* Encrypting the payload defeats attempts to survey traffic by user agent (if there's no other way to distinguish, e.g., by fingerprinting).
KM> I think this would be a really helpful follow on document. I'd be willing to work on it if you're game. I've been thining about something similar, specific to TLS, but should be broadened.
I'm afraid this topic might be more of a research project than it initially appears, but it's probably a reasonable exercise to see what we can produce for a plaintext protocol vs. that same protocol over TLS. Ping me.
It is important to note that the push for encryption by application
providers has been motivated by the application of the described
techniques. Some application providers have noted degraded
performance and/or user experience when network-based optimization or
enhancement of their traffic has occurred, and such cases may result
in additional operator troubleshooting, as well.
KR> Observation: additionally, I think you'll encounter the argument that the responsibility for diagnosing bad interactions between applications and networks falls on the application owner rather than the network operator. Basically, I feel like the desire among protocol designers is for operators to provide a pipe with certain key characteristics that interact well with established transport protocol mechanisms, and otherwise to leave the traffic alone and let the application developers do what they want to within the expected constraints. If that's infeasible (e.g., in edge cases, or with respect to new technologies that interact badly with existing transports, such as the loss=congestion assumption of TCP that interacts badly with wifi), that's precisely the case needs to be made by this document.
KM> We have encountered this argument already. It's a tough one as SPs have the SLAs with customers, so they are the first call. Many don't know how to get in touch with APP providers. I understand the application developers perspecive, but also see that there has to be some ability to troubleshoot. Sure, providers could wrap the protocols for transport to provide some way of measuring, but information is lost. IPv6 with flow identifiers is another way to do it, but you might not be able to prioritize a call or protocol that has little tolerance for delay over one that does for instance. And I realize that app providers just want all traffic to have the same priority, but emergency calls are important.
BW> I think the point made by the document is correct though: operators are nearly always the first call, not the application provider.
KM> We were asked to remove text that said that. I agree that it is the case as the providers have the SLAs and you don't typlically have a number for App providers.
BW> The operators are looking for ways to demonstrate that they did not cause the problem (or determine that they did) for efficient hand-off to the correct party for resolution. There are certainly problems an approach that changes the behavior of the protocol, but it's difficult to argue with the diagnostic need.
AM> Using Netflix as an example, the first source of problem they mention is the network when
addressing the question "Why doesn't Netflix work?":
"If Netflix isn’t working, you may be experiencing a network connectivity issue, an issue with your device, or an issue with your Netflix app or account."
from https://help.netflix.com/en/node/461?ui_action=kb-article- popular-categories
They previously had even stronger wording, something like "First, make sure your network connection meets the Netflix requirements ... URL"
One of the causes of re-buffering are CDN-related pauses when accessing the next segment: completely hidden from users so far.
Additional frequent cause: the unlicensed WiFi network owned and operated by the customer.
Another way to look at this strategy: App providers are transferring as much overhead cost to the network operators as possible
(troubleshooting customer problems is expensive - rolling a truck negates months of revenue), while preserving as
much value/control/revenue as they can for themselves. The greed-thingy plays poorly over time.
A user-focused strategy would be to form partnerships for troubleshooting of shared customers, but that might result in exposing
the real causes and some would rather hide for now, it seems.
I agree with all the points you've made. How do we square reality (users blame network operators) with the current approach to protocol design at the IETF (keep the network dumb)? I feel like creating a conversation about this apparent cognitive dissonance will be one of the most important outcomes of publishing this document.
I have no doubt the conflict will resolve itself somehow: CDNs, for instance, act as an intelligent overlay over dumb networks and can therefore provide the most consistent user experience when deeply deployed into carrier networks the structure of which they have intimate knowledge. Is the right solution to continue to effectively delegate this responsibility by encouraging breaking of connections at the edge, or should the IETF be trying to optimize the end-to-end performance of its protocols on the public internet?
Anyway, I digress. This isn't a conversation I'm proposing you have in this document; just that the doc should raise these kinds of questions in the reader.
packet is able to provide stateless load balancing. This ability
confers great reliability and scaleability advantages even if the
flow remains in a single POP, because the load balancing system is
not required to keep state of each flow. Even more importantly,
there's no requirement to continuously synchronize such state among
the pool of load balancers.
KR> An important point is that an integrated load balancer repurposing limited existing bits in transport flow state must maintain and synchronize per-flow state occasionally: using the sequence number as a cookie only works for so long given that there aren't that many bits available to divide across a pool of machines.
KM> I added in this point, but have to check back on flow of text.
I checked the wording in -14.I'm going to propose slightly different language:
q( This ability
confers great reliability and scalability advantages even if the
flow remains in a single POP, because the load balancing system is
not required to keep state of each flow. There is value even when the
confers great reliability and scalability advantages even if the
flow remains in a single POP, because the load balancing system is
not required to keep state of each flow. There is value even when the
repurposed bits are strictly insufficient for encoding all state: an integrated load balancer repurposing
limited existing bits in transport flow state must still maintain and
synchronize per-flow state occasionally (using the sequence number as
a cookie only works for so long given that there aren't that many
bits available to divide across a pool of machines), but there is no longer
limited existing bits in transport flow state must still maintain and
synchronize per-flow state occasionally (using the sequence number as
a cookie only works for so long given that there aren't that many
bits available to divide across a pool of machines), but there is no longer
a requirement for such synchronization to be continuous or instantaneous. )
KR> A dedicated mechanism for storing load balancer state, such as QUIC's proposed connection ID, is strictly better from the load balancer's point of view, and is probably even better from a privacy perspective than bolting it on to an unrelated transport signal because it can be tightly controlled by one of the endpoints and rotated to avoid roving client linkability: in other words, being a specific, separate signal, it can be governed in a way that is finely targeted at that specific use-case. (I'm thinking the advantages of separate mechanisms belongs in a different part of the doc; this section is more like the problem statement than the solution statement.)
KM> This (above) needs to be reworded to be neutral and this does go towards solution space, which we were trying to avoid. How about:
Another possibility is a dedicated mechanism for storing load balancer state, such as QUIC's proposed connection ID to provide visibility to the load balancer. An identifier could be used for tracking purposes, but this may provide an option that is an improvement from bolting it on to an unrelated transport signal. This method allows for tight control by one of the endpoints and can be rotated to avoid roving client linkability: in other words, being a specific, separate signal, it can be governed in a way that is finely targeted at that specific use-case.
SGTM. Maybe s/improvement from bolting it on to/improvement compared to co-opting/.
In future Network Function Virtualization (NFV) architectures, load
balancing functions are likely to be more prevalent (deployed at
locations throughout operators' networks)[. NFV environments will
require some type of identifier (IPv6 flow identifiers, the Proposed
QUIC connection ID, etc.) for managing]
traffic using encrypted tunnels.[ The shift to increased encryption
will have an impact to visibility of flow information and will require
adjustments to perform similar load balancing functions within an NFV.]
KR> I'm not sure what architecture this paragraph is discussing: are you talking about encrypted tunnels between NFV nodes? Is this something obvious to people involved in NFV? A diagram (or informational reference) would be helpeful to me here.
KM> I see your point, the langauage here could be more clear. Do the above adjustments (ed: in []) help?
I am still a little confused. Is the idea that load balancing in NFV environments has a unique need for stateless (or reduced-state) load balancing that other applications don't have? I'm having a hard time wrapping my head around why that would be the case. Or is the point here just to highlight NFV as just another use case to consider?
2.2.2. Differential Treatment based on Deep Packet Inspection (DPI)
...
These effects and potential alternative solutions have been discussed
at the accord BoF [ACCORD] at IETF95.
KR> This section is labeled DPI, but really, the underlying issue is what you stated in the first paragraph: different kinds of traffic have different QoS needs, yet a network provider can't rely on a voluntary signal from an untrusted device to decide on QoS or every packet is simply going to be marked "high importance" and so we're back to treating all traffic equivalently. I'd argue against one of the memes I heard at the accord BoF, that it's down to latency vs. throughput, by pointing out that some applications (e.g., live video with low hand-wave latency) need both.
Even after reading this, I'm still skeptical of the need for any more granularity than flow, and using AQM on a per-flow (e.g., 5-tuple) or flow-aggregate (some subset of the 5-tuple) to prevent an application or user from consuming resources unfairly. What, for instance, prevents a carrier from privileging VoIP traffic by looking at endpoints? Would there be a way for someone else to masquerade non-VoIP traffic as VoIP traffic given this kind of setup? This is the kind of question that I need answered by this doc.
BW> It might be useful to note in this section that QUIC and H2 both combine multiple micro-flows, possibly of different types, within a single encrypted transport-layer flow. They share this with IPsec tunnels and the like. IOW, the increased use of encrypted aggregating encapsulation can hide even the the most basic representation of a flow from the differentiated service element. This same concern applies to load balancing elements discussed in section 2.2.1.
Good point: an example of this is sending both Youtube and search responses over the same QUIC or H2 connection, with no way for the network to throttle one without throttling the other.
AM> We were asked not to refer to QUIC, for various reasons (e.g., still under development).
There will always be areas where network can make the best decision, because of the
information available to the network operators (and the lack of that same info at end-points).
When network resources are constrained, only the network can manage priorities.
This has been organized according to applications that can be identified, but there
can be other solutions requiring cooperation between user devices and the network
according to subscription to a special service (QCI above).
Got it. Is DPI the right framing for this, or is something more generic (e.g., "content-aware traffic management") what is really required? E.g., the network doesn't necessarily need to know which video you're watching, only that it is video, and maybe what the available bitrates are and associated quality.
An application-type-aware network edge (middlebox) can further
control pacing, limit simultaneous HD videos, or prioritize active
videos against new videos, etc.
KR> Observation: This subsection provides the first really compelling argument I've seen for exposing flow metadata to the path. On long paths, physics gets in the way of tight control feedback loops. If nothing else, this should provide motivation for protocol designers and operators to break down the characteristics of different kinds of flows, determine where control points are needed in each of them, and figure out how to implement those.
I think there is this conceit among protocol designers that quality problems can all be solved at the endpoints without any cooperation from path elements; the really killer arguments are examples of where that cannot possibly be the case. ECN is a great example of this, and is a signal explicitly targeted at middleboxes with opt-in by the endpoints: it allows a middlebox to report congestion without dropping packets, which produces measurably better QoS for the user.
KM> Ack, thanks. You're not looking for additional text here, is that right? If so, what are you thinking should be added?
No, just an observation that this was one of the more thought-provoking sections of the doc for me.
Alternate approaches such as blind caches [I-D.thomson-http-bc] are
being explored to allow caching of encrypted content; however, they
still need to intercept the end-to-end transport connection.
KM> [s/need to intercept the end-to-end transport connection/require cooperation between the content owners/CDNs and blind caches and fall outside the scope of what is covered in this document/
SGTM.
2.2.6. Content Compression
In addition to caching, various applications exist to provide data
compression in order to conserve the life of the user's mobile data
plan and optimize delivery over the mobile link. The compression
proxy access can be built into a specific user level application,
such as a browser, or it can be available to all applications using a
system level application. The primary method is for the mobile
application to connect to a centralized server as a proxy, with the
data channel between the client application and the server using
compression to minimize bandwidth utilization. The effectiveness of
such systems depends on the server having access to unencrypted data
flows.
KR> Observation: given the side channels exposed by data compression that is blind to content, the inability to compress arbitrary payloads is likely to be regarded as a feature of encryption. (Though I recognize this is a catalog, not an endorsement.) Furthermore, in most cases eliminating compression is still 2-competitive with compression, so I'm not sure it's a really compelling use-case.
BW> Per-object content compression might not be a compelling use case here. Aggregated data stream content compressions that spans objects and data sources is compelling, though. If there is a network element close to the receiver that sees all content destined for the receiver and can treat it all as part of a unified compression scheme (e.g., through the use of a shared segment store) will often be much more effective at providing data off-load.
KM> Thanks, we'll add this text (modified) to make those helpful points clear.
How about:
Aggregated data stream content compression that spans objects and data sources that can be treated as part of a unified compression scheme (e.g., through the use of a shared segment store) is often effective at providing data offload when there is a network element close to the receiver that has access to see all the content.
Sounds good. This is general enough to cover the case of networks with limited uplinks wanting to cache content that is conceptually shared (e.g., VOD) but delivered independently to end users via individual TLS connections.
KR> It might be worth discussing the typical opt-in strategy for these things in the presence of TLS, adding a new intercept CA to willing clients, which has the downside that it potentially exposes every https connection to an active MitM.
BW> +1
KM> OK, we hadn't done that before since the option doesn't change, but you make a good point, so I'll add in text. Thanks.
I added the following:
This method is also used by other types of network providers enabling
traffic inspection, but not modification.</t>
<t>Content filtering via a proxy can also utilize an intercepting
certificate where the client's session is terminated at the proxy
enabling for cleartext inspection of the traffic. A new session
is created from the intercepting device to the client's
destination, this is an opt-in strategy for the client. Changes to
TLSv1.3 do not impact this more invasive method of interception, where
this has the potential to expose every HTTPS session to an active
man in the middle (MitM). </t>
Mostly sounds good. Is there a reason to mention TLS 1.3 specifically here?
KR> Random comment: especially with respect to government content filtering, I'm worried that the IETF's current approach of playing chicken with regulators on end-to-end encryption is going to result in normalization of intercept CAs, which will be strictly worse than a compromise solution in which a subset of traffic can be inspected (but not modified) with the user's knowledge and consent (e.g., distinct optics in the browser). I wouldn't like either outcome, frankly, but it would be nice if we had a game plan for what to do for user privacy if intercept CAs become a requirement for using the web in large parts of the world (something we might be one "crisis" away from), and an honest evaluation of the alternatives. Fundamentally, I don't like it when discussion gets shut down because people want to bury their heads in the sand in the name of ideology.</rant>
BW> +1. I also note that this concern applies to some of the other performance related use cases too.
KM> I think the real argument here is a control one between the application and management folks and not security/privacy even though that's what is often discussed. This is all about control.
Right, but the core issue being addressed by this document is that measures intended for reasons of privacy and security (encryption) are impacting something over which there is much less consensus (content-aware flow management and path intelligence). I'm not proposing any language here, only pontificating that the purity approach might backfire, and I'm not sure we have a backup plan.
In addition, mobile network operator often sell tariffs that allow
free-data access to certain sites, known as 'zero rating'. A session
to visit such a site incurs no additional cost or data usage to the
user. This feature is impacted if encryption hides the details of
the content domain from the network.
KR> There's the related issue that zero-rating by-implementation typically applies only to direct connections to a particular endpoint (e.g., by IP): if a user accidentally tunnels traffic from Spotify through a corporate VPN, that traffic won't be zero-rated, encrypted tunnel or not. (This goes back to the taxonomy of metadata layers comment I made near the top.) Carriers aren't going to trust e.g., a Host header for zero-rating, because that provides a simple way to tunnel traffic for free: consequently, determination of zero-rating will always involve some hard-to-impersonate credential, like an IP address or server certificate in the public trust web.
KM> Not sure what to add here, any ideas, AL?
I think the only change I'd make here is to change "content domain" to "content origin", because domain implies hostname where the origin is often an IP.
When RTSP stream content is encrypted, the 5-tuple information within
the payload is not visible to these ALG implementations, and
therefore they cannot provision their associated middelboxes with
that information.
KR> I would argue that this is a protocol design issue. This was originally a problem with firewalls and NATs, with content inspection as a hack to work around the protocol/network impedance mismatch. I'm not the only one who would argue the right solution today is to design protocols to not require linkage across connections by middleboxes that do basic filtering.
KM> I think we are in agreement here for solution direction, but the document specificly tries to avoid solutions. This example has been raised in the IESG by Warren and the apps side hadn't considered his view of it previously. It would be good for protocols to have these considerations in their designs, they were mostly thinking it didn't matter and were end-to-end. But poor video streaming sessions are an issue. Not sure we should add any text here???
This was just another random observation.
Data center operators may also maintain packet recordings in order to
be able to investigate attacks, breach of internal processes, etc.
In some industries, organizations may be legally required to maintain
such information for compliance purposes. Investigations of this
KR> I think you'll get a "[citation needed]" from folks on the TLS mailing list.
KM> I suspect this is one you have that recorded text, you have to maintain it for chain of custody with investigation handling. I'll have to figure out if there is anything that would require the capture, I suspect not, but could be wrong.
Just making the point that this has been contended several times on various mailing lists and at meetings, so it would be nice to get the oft-cited cases documented somewhere as an informational reference.
There are use cases where DAR or disk-level
encryption is required. Examples include preventing exposure of data
if physical disks are stolen or lost.
KR> I don't see these last two sentences are relevant, as they have nothing to do with the network flows.
KM> I'm happy to remove. DO they help a reader who is not familiar with the technology to understand the layers of encryption used at all or is it better to remove the sentence?
I'm not sure. I'm actually a bit confused about this section in general. It seems to be discussing monitoring of data during transport to/from the storage cluster in the same paragraph as encryption of data at rest, but I'm not sure what point it's trying to make. Is it that operators have a threat model that doesn't include the network connection between the storage cluster and the client, but which does include exfiltration of the disks in the cluster?
Security monitoring in the enterprise may also be performed at the
endpoint with numerous current solutions that mitigate the same
problems as some of the above mentioned solutions. Since the
software agents operate on the device, they are able to monitor
traffic before it is encrypted, monitor for behavior changes, and
lock down devices to use only the expected set of applications.
Session encryption does not affect these solutions. Some might argue
that scaling is an issue in the enterprise, but some large
enterprises have used these tools effectively.
KR> This is another example of mixing proposed solutions in among the problem statement. I would argue for a clear separation, which may mean that this document needs to have a single-minded focus on "here are the problems and here's how enterprises currently address them."
BW> Also, enterprises increasingly allow BYOD programs for their employees, and such programs make it more difficult to ensure that adequate endpoint-based defenses are active. This is especially true when the area of risk in question is the above #5 "track misuse and abuse by employees". Note too that endpoint-based defenses can be less effective when the device is already compromised, in which case detection of the compromised device and effective remediation can be made more effective through the additional use of an on-path element.
KM> [made some subsequent edits to this section]
LGTM.
5.7. Further work
Although incident response work will continue, new methods to prevent
system compromise through security automation and continuous
monitoring [SACM] may provide alternate approaches where system
security is maintained as a preventative measure.
KR> Not clear how the unknowns relate to the purpose of this document. Being sarcastic for a minute, I'm interpreting this as "Any cleartext metadata just *might* be used in the future for some kind of enterprise security monitoring!"
KM> Hmm, it's meant to say endpoints (which you control) should be used and technology like what is expected out of SACM will help with automating this. We are open to text suggestions.
Yeah, I think I must have misread it the first time, because I get only your meaning now.
Kyle