Magnus- thanks for the clarification. Based on your feedback congestion control and relay placement, here is some proposed text for section 7.2: "TreeDN deployments MUST support the congestion control guidelines described in Section 4.1.4.2 of RFC7450 and Section 4.1 in RFC8085. This implies the AMT gateway SHOULD use the topologicaly closes AMT relay. Section 3.1 of RFC8777 describes a set of procedures for optimal relay selection." As for the unicast feedback channel, propose the following text in sect 7.1: "These "out-of-band" unicast streams SHOULD use the same congestion control and authentication mechanisms that are used today for mass audience unicast delivery." On Fri, 31 May 2024, Magnus Westerlund wrote: | | Hi, | | | | Please see inline. | | | | | | From: Leonard Giuliano <lenny@xxxxxxxxxxx> | Date: Thursday, 30 May 2024 at 21:18 | To: Magnus Westerlund <magnus.westerlund@xxxxxxxxxxxx> | Cc: tsv-art@xxxxxxxx <tsv-art@xxxxxxxx>, draft-ietf-mops-treedn.all@xxxxxxxx <draft-ietf-mops-treedn.all@xxxxxxxx>, last-call@xxxxxxxx <last-call@xxxxxxxx>, mops@xxxxxxxx <mops@xxxxxxxx>, | Ronald Bonica <rbonica@xxxxxxxxxxx>, Greg Shepherd <gjshep@xxxxxxxxx> | Subject: Re: [Mops] Tsvart last call review of draft-ietf-mops-treedn-04 | | [You don't often get email from lenny@xxxxxxxxxxx. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] | | Magnus- thanks for the detailed review. Please keep in mind that this doc | makes no attempt to change any protocols (eg, AMT, PIM-SSM), as that is | beyond the goal of this doc and indeed, outside the charter of MOPS. | Rather, the goal of the doc is to describe an architecture that combines | existing protocols and service models to deliver a service for producers | and consumers of existing content in a way that more efficiently utilizes | network resources than is done today. As such, we leverage the existing | mechanisms described in RFC7450 and RFC8085 for congestion control. We | can certainly call these references out more forcefully if this is not | clear enough, but otherwise please let us know where you think RFC8085 | falls short because we were under the impression that RFC8085 resolved the | concerns of congestion control with AMT. Copying several folks who I | believe were intimately involved with this issue previously, and they can | correct me if this understanding is inaccurate. | | | | MW: So RFC 7450 is very clear that it does not provide any solution for congestion control. It puts the requirement onto basically two parties: | | * The multicast application / multicast transport prootocl | * The deployer of AMT tunnel relays | | | | What RFC 7450 says: | | | | 4.1.4.2. Congestion Considerations | | | | AMT relies on UDP to provide best-effort delivery of multicast data | | to gateways. Neither AMT nor UDP provides the congestion control | | mechanisms required to regulate the flow of data messages passing | | through a network. While congestion remediation might be provided by | | multicast receiver applications via multicast group selection or | | upstream reporting mechanisms, there are no means by which to ensure | | that such mechanisms are employed. To limit the possible congestion | | across a network or wider Internet, AMT service providers are | | expected to deploy AMT relays near the provider's network border and | | its interface with edge routers. The provider must limit relay | | address advertisements to those edges to prevent distant gateways | | from being able to access a relay and potentially generate flows that | | consume or exceed the capacity of intervening links. | | | | RFC 8085 is also just putting requirements on applications and have some review of what has been defined by IETF. Note for example: | | | | Section 4.1 of RFC 8085 | | Congestion control is particularly important for | | any multicast session where all or part of the multicast distribution | | tree spans an access network (e.g., a home gateway). Two styles of | | congestion control have been defined in the RFC Series: | | | | I will note that what AMT relays result in is to make the issues with multicast traffic congestion control be present all the way between the AMT relay and the AMT client, for example over | general Internet. That as it removes the benefit of multicast of only having one copy of each packet flow on each link. And the above is written with the consideration that in backbone and | larger trunk links one multicast flow is a rather small load, compared to the outer part of access networks where it can be a much more significant. With AMT relays this multicast group’s | traffic is now amplified in the backbone link by a factor of N, where N is the number of AMT clients using this relay. | | | | And what RFC 8085 says is deploy congestion control. | | | | The reason I keep on banging on this is that most multicast applications are built to deploy in single domain networks with some capacity planning for this traffic. The whole idea of your | architecture is to remove this constraint which is great. However, you can’t take today’s application and deploy them on your architecture because that is likely to cause congestion | issues. At a minimal one have to update the multicast application with congestion control before one can use it over AMT relay links. I think your architecture need to be very explicit | about that this change is required. | | | | | | | | As for other transport (and higher) layer related issues like reliability, | authorization, encryption, accounting, etc, these questions did come up in | WG discussions. While the doc focuses primarily on network layer issues | pertaining to the efficient distribution of multidestination content, the | consensus of the WG was to identify those higher layer gaps and, rather | than specify a set of solutions, identify some existing solutions that | address this space. It is important to note that there are existing | TreeDN deployments on the Internet today, as well as existing UDP-based | streaming deployments. The solutions noted in section 7 are not intended | to be an exhaustive list, as many different approaches exist and this is | an area of continued innovation, so it is not possible to identify all of | these approaches. But we did want to provide a set of pointers to some of | these approaches for those interested in further understanding some of the | available options. | | | | MW: The reason I brought this up is that I at least suspect that are significant amount of network level assumptions baked into those solutions, that are then broken by TreeDN. | | | | Hope this helps to provide useful framing and background. Further | responses inline: | | On Fri, 3 May 2024, Magnus Westerlund via Datatracker wrote: | | | | | Reviewer: Magnus Westerlund | | Review result: Not Ready | | | | This document has been reviewed as part of the transport area review team's | | ongoing effort to review key IETF documents. These comments were written | | primarily for the transport area directors, but are copied to the document's | | authors and WG to allow them to address any issues raised and also to the IETF | | discussion list for information. | | | | When done at the time of IETF Last Call, the authors should consider this | | review as part of the last-call comments they receive. Please always CC | | tsv-art@xxxxxxxx if you reply to or forward this review. | | | | So I am glad the draft at least scratches a little on the surface when it comes | | to the transport challenges of multicast transport deployment. However, I think | | this architecture needs to actually discuss how it is going to handle some of | | these issues in more detail. This as they are both challenging and also can | | have significant impact on the underlying unicast network. To the level where | | they actually likely need to be part of the real architecture that enables | | multicast delivery. | | | | Congestion Control and bit-rate adaptation | | | | The AMT RFC 7450 contains section 4.1.4.2: | | | | "AMT relies on UDP to provide best-effort delivery of multicast data | | to gateways. Neither AMT nor UDP provides the congestion control | | mechanisms required to regulate the flow of data messages passing | | through a network. While congestion remediation might be provided by | | multicast receiver applications via multicast group selection or | | upstream reporting mechanisms, there are no means by which to ensure | | that such mechanisms are employed. To limit the possible congestion | | across a network or wider Internet, AMT service providers are | | expected to deploy AMT relays near the provider's network border and | | its interface with edge routers. The provider must limit relay | | address advertisements to those edges to prevent distant gateways | | from being able to access a relay and potentially generate flows that | | consume or exceed the capacity of intervening links." | | | | Which means that this requirement to have congestion control do apply on the | | one running the end-to-end service over the TreeDN RaaS. The draft is not | | particular clear on that it will fall on these deploying a multicast based | | service to have a solution for rate adapation in place. I rather think the text | | tries to gloss this over. For example noting that reliability can be addressed | | by FEC, which clearly is one of the tools in the tool box, hinting that packet | | loss rates up to 5% can be hidden. The other side of this is that if the | | multicast application running over AMT these flow will push basically all other | | congestion controlled traffic out of the way. Anyone deploying an application | | using multicast over this architecture will only be bitten themself when enough | | users actually receive multicast flows that are at higher bandwidth than the | | link capacity, or possibly when N users of poorly adopting services share the | | bottleneck and those N flows can fit. | | | | With SSM the congestion control and real-time media services, rather than file | | distribution broad cast discs one likely have to select the most approriate | | quality level and subscribe to that. Which means that the replication do gets | | fragmented. For large file deliverly and usage of broad cast discs congestion | | control could be more fine grained and use like WEBRC (RFC 3738). | | | | Using NORM have interesting implications as that will result in selecting a | | rate adapted to the lowest capable receivers within a multicast group. Also I | | want to come back to NORM when it comes to feedback implosion problems. But, I | | get the impression that this really is targeting large scale deployments, where | | some these problems will be elevated significantly. | | | | I think this document needs should be clearer about the non-negotiable | | requirement to have bit-rate adaptation. | | | | Feedback implosion | | | | The usage of SSM like structure also means that the service on top of this RaaS | | will have to think hard about Feedback implosion. And this goes further than | | many other multicast deployments as if this one is successful one have near | | global reach. So the number of participant a popular service needs to handle | | are clearly in the many millions, without a natural upper limit. So the | | application deployer need to deploy a structure for any gathered feedback that | | scale and have the tools to inform the receivers about where to send the | | traffic. | | | | This issue of feedback also come into the usage of AMT and its managability. | | Where there will be many users and thus receivers are not governed by the | | multicast topology, instead of where the users are. Thus, deploying feedback | | servers in the right topology places will be part of a managability challenge. | | | | I do want to note that how an application receiver directed to send some | | feedback depending on protocol actually need mechanism to prevent it from being | | an unwanted participant in a DDoS attack. | | Sorry, not following you here. This issue of feedback implosion seems no | different than what is handled with today's unicast-based solutions being | employed for these mass audience events. Sect 7.1 suggests using unicast | feedback channels to provide the same functionality as what is done today | with unicast delivery- the only difference is that multicast is used for | the actual data streams of interest (eg, video). Could certainly add that | these unicast streams should use the same types of congestion control | mechanisms that unicast-based CDNs use today if that is not clear. | | | | MW: So the issue I try to make clear is that the address any unicast feedback is directed is very sensitive information. If I as an attacker can change that for a legit service I can | create a hose of distributed feedback traffic to hit that address. If I am a malicious service that uses my users as involuntary participants in a DDoS attack even if the information is | source authenticated they still will be a bad actor. | | | | When it comes to comparing to unicast services, I would note that in those cases you already have automatic scaling of the feedback traffic as you normally are limited on outgoing traffic | and self regulating even, first on transport protocol level as less traffic can mean less feedback, as well as if the user doesn’t get the content correctly they are likely to discontinue | the use of the service thus stopping the load. While for unicast feedback for multicast traffic doesn’t have the same control, in this it needs to be done explicitly. | | | | | Managability of the AMT tunneling | | | | What is not discussed is what set of tools are needed to manage this RaaS. I | | see many challenges. First one needs to find the most appropriate AMT relay to | | ensure that the AMT traffic takes a way through the network that has | | appropriate resources. | | RFC8777 describes a mechanism (plus a set of alternatives in sect 3) that | addresses this. Can certainly add a reference to this if you think it is | appropriate. | | | | MW: Yes, I think some reference would be appropriate. | | | | | | | Another challenge is how to actually attempt to reach the desired goal that the | | users local access network actually support multicast. Here several challenges | | appear to exist. First determining that support and getting access to it from | | an endpoint within the local network. There also appear to exist a need for an | | happy eyeball handling of attempting to use local network to determine if it | | actually can reach the desired multicast group. | | Assuming the "users" you are referring to are "receiving end hosts", yes, | it would make sense for the receiver to attempt to join the multicast | group natively first, and if no traffic is received, fail over to AMT. | This is generally handled by the application layer today. Can mention | that explicitly if you think that would help. | | | | MW: I think this is a relevant aspect of how end hosts integrate with this architecture so please add it. | | | | | Will TreeDN move back multicast address to have global uniquness to ensure that | | one can ask for just the address and get the right service from a network | | potentially using AMT it self? There are requirement in the architecture on | | being able to find AMT relay's delivering the right group. Otherwise the | | complexitiy will not ensure the desired incentaive of deploying local multicast | | in an attempt to reduce the unnecessary duplication. | | | | The main point is that for TreeDN architecture to be successful it actually | | needs to include much more components and addressing a number of additional | | questions that actually makes it useful. Not simply provide replication of | | packets. How to actually manage the AMT tunnels, as well as ensure safe | | deployment over Internet also needs to be considred. Then there are some | | options for the transport protocol level depending on the actual application. | | That is clearly a seperate layer, but there are some requirements on this level | | that it would be good to make clear to avoid congesting some access networks or | | directing a feedback implosion onto some node. | | | | | | | | -- | | Mops mailing list -- mops@xxxxxxxx | | To unsubscribe send an email to mops-leave@xxxxxxxx | | | | | -- last-call mailing list -- last-call@xxxxxxxx To unsubscribe send an email to last-call-leave@xxxxxxxx
