Reviewer: Gyan Mishra Review result: Not Ready I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF documents being processed by the IESG for the IETF Chair. Please treat these comments just like any other last call comments. For more information, please see the FAQ at <https://trac.ietf.org/trac/gen/wiki/GenArtfaq>. Document: draft-ietf-bess-datacenter-gateway-?? Reviewer: Gyan Mishra Review Date: 2021-04-28 IETF LC End Date: 2021-04-29 IESG Telechat date: Not scheduled for a telechat Summary: This document defines a mechanism using the BGP Tunnel Encapsulation attribute to allow each gateway router to advertise the routes to the prefixes in the Segment Routing domains to which it provides access, and also to advertise on behalf of each other gateway to the same Segment Routing domain. This draft needs to provide some more clarity as far as the use case and where this would as well as how it would be used and implemented. From reading the specification it appears there are some technical gaps that exist. There are some major issues with this draft. I don’t think this draft is ready yet. Major issues: Abstract comments: It is mentioned that the use of Segment Routing within the Data Center. Is that a requirement for this specification to work as this is mentioned throughout the draft? Technically I would think the concept of the discovery of the gateways is feasible without the requirement of SR within the Data Center. The concept of load balancing is a bigger issue brought up in this draft as the problem statement and what this draft is trying to solve which I will address in the introduction comments. Introduction comments: In the introduction the use case is expanded much further to any functional edge AS verbiage below. OLD “SR may also be operated in other domains, such as access networks. Those domains also need to be connected across backbone networks through gateways. For illustrative purposes, consider the Ingress and Egress SR Domains shown in Figure 1 as separate ASes. The various ASes that provide connectivity between the Ingress and Egress Domains could each be constructed differently and use different technologies such as IP, MPLS with global table routing native BGP to the edge, MPLS IP VPN, SR-MPLS IP VPN, or SRv6 IP VPN” This paragraph expands the use case to any ingress or egress stub domain Data Center, Access or any. If that is the case should the draft name change to maybe a “stub edge domain services discovery”. As this draft can be used for any I would not preclude any use case and make the GW discovery open to be used for any service GW edge function and change the draft name to something more appropriate. This paragraph also states for illustrative purposes which is fine but then it expands the overlay/underlay use cases. I believe this use case can only be used for any technology that has an overlay/underlay which would preclude any use case with just an underlay global table routing such as what is mentioned “IP, MPLS with global table routing native BGP to the edge. The IP or global table routing would be an issue as this specification requires setting a RT and an export/import RT policy for the discover of routes advertised by the GWs. As I don’t think this solution from what I can tell would work technically for global table routing I will update the above paragraph to preclude global table routing. We can add back in we can figure that out but I don’t think any public or private operator would change from global table carrying all BGP prefixes in the underlay now drastic change to VPN overlay pushing all the any-any prefixes into the overlay as that would be a prerequisite to be able to use this draft. >From this point forward I am going to assume we are using VPN overlay technology such as SR or MPLS. NEW “SR may also be operated in other domains, such as access networks. Those domains also need to be connected across backbone networks through gateways. For illustrative purposes, consider the Ingress and Egress SR Domains shown in Figure 1 as separate ASes. The various ASs that provide connectivity between the Ingress and Egress Domains could be two as shown in Figure-1 or could be many more as exists with the public internet use case, and each may be constructed differently and use different technologies such as MPLS IP VPN, SR-MPLS IP VPN, or SRv6 IP VPN” with a “BGP Free” Core. This may work without “BGP Free” core but I think to simplify the design complexity I think constraining to “BGP Free” core transport layer. SR-TE path steering as well gets much more complicated if all P routers are running BGP as well. I think in this example we can even explicitly say this example shows the public internet as that would be one of the primary use cases. This paragraph is confusing to the reader As a precursor to this paragraph I think it maybe a good idea to state that we are talking global table IP only routing or VPN overlay technology with SR/MPLS underlay transport. That will make this section much easier to understand. Figure 1 drawing you should give a AS number to both the ingress domain and egress domain so the reader does not have to make assumptions if it iBGP or eBGP connected to the egress or ingress domain and state eBGP in the text below. Lets also call the intermediate ASNs in the middle as depicted in the diagram could be 2 as shown illustratively but could be many operator domains such as in the case of traversing the public internet. In the drawing I would replace ASBR for PE as per this solution as I am stating it has to be a VPN overlay paradigm and not global routing. Also in the VPN overlay scenario when you are doing any type of inter-as peering the inter-AS peering is almost always between PE’s and not a separate dedicated device serving a special “ASBR-ASBR” function as the PE is acting as the border node providing the “ASBR” type function. So in the re-write I am assuming the drawing has been updated changing ASBR to PE. Lets give each node a number so that we can be clear in the text exactly what node we are referring to. In the drawing please update that GW1 peers to PE1 and GW2 peers to PE2 and GW3 peers to PE3. GW3 also peers to GW4 and GW2 peers to GW5 which GW4 and GW5 are part of AS3. In the AS1-AS2 peering top peer would be PE6 peers to PE8 and bottom peer PE7 peers to PE9. So PE6 and PE7 are in AS1 and PE8 and PE9 are in AS2. I made the bottom to ASBRs in AS3 for the selective deterministic load balancing now calling them GW4 and GW5 used later in the problem statement. One major problem with this problem statement description is that it is incorrect as far as GW load balancing that it does not work today in the topology given in Figure-1. The function of edge GW load balancing is based on the iBGP path tie breaker lowest common denominator in the BGP path selection which is lowest IGP underlay metric and as long as the metric is equal and you have iBGP multipath enabled you now can load balance to egress PE1 and PE2 endpoints. So in this case flows coming from AS1 into AS2 hit a P intermediate router which has iBGP multipath enabled and has lets say equal cost for route to the next hop attribute assuming next-hop-self is set so the cost to loopback0 on PE1 and cost to loopback0 on PE2 is lets say 10, so now you have a BGP multipath. What is required though is the RD has to be unique in a “BGP Free” core RR environment where all PE’s route-reflector-clients peer to the RR and for all the paths that are advertised to the RR to be reflected to all the egress PE edges the RD must be unique for the RR to reflect all paths. BGP add-paths is only used if you have Primary and Backup routing setup where PE1-GW1 has a 0x prepend and PE2-GW2 has 1x prepend so now with BGP add-paths along with BGP PIC Edge you now have a edge pre-programmed backup path. So the add-paths is not necessarily something that helps for load balancing and is in fact orthogonal to load balancing as it for Primary / Backup routing and not Active/Active load balancing routing where load balancing with VPN overlay is simply achieved with unique RD per PE and iBGP multipath and equal cost paths to the underlay recursive IGP learned next-hop-attribute in this case the PE loopback 0 per the next hop rewrite via “next-hop-sellf” done on the PE-RR peering in a standard VPN overlay topology. As far as load balancing being accomplished in the underlay what I have stated is independent of SR-TE however with SR-TE candidate path the load balancing ECMP spray to egress PE egress GW AS can also happen as well with prefix-sid. OLD Suppose that there are two gateways, GW1 and GW2 as shown in Figure 1, for a given egress SR domain and that they each advertise a route to prefix X which is located within the egress SR domain with each setting itself as next hop. One might think that the GWs for X could be inferred from the routes' next hop fields, but typically it is not the case that both routes get distributed across the backbone: rather only the best route, as selected by BGP, is distributed. This precludes load balancing flows across both GWs. I am rewriting the text in the NEW as there is some discrepancy in the routes being distributed across the backbone and what gets distributed. So I am completely re-writing to make it more clear what we are trying to state here as the text appears technically to be incorrect. To help state the flow will use the BGP route flow to help depict the routing and try to get to the problem statement we are trying to portray. NEW Suppose that there are two gateways, GW1 and GW2 as shown in Figure 1, for a given egress SR domain and each gateway advertises via EBGP a VPN prefix X to AS2 core domain via EBGP with underlay next hop set to GW1 or GW2. In this case we are Active / Active load balancing with PE1 and PE2 receives the VPN prefix and advertised the VPN prefix X into the domain with next-hop-self set on the PE-RR peering to the PE’s loopback0. The P routers within the domain have ECMP path with IGP metric tie to the egress PE1 and egress PE2 for VPN Prefix X learned from GW1 and GW2. SR-TE path can now be stitched from GW3 to PE3 SR-TE Segment-1 to PE3 to PE6 and PE7 Segment-2 to PE8 and PE9 to Egress Domain via PE1 and PE2 to GW1 and GW2. In this case however we don’t want the traffic to be steered via SR-TE Load balanced via ingress GW3 and want to take GW3 out of rotation and load balance traffic to GW4 and GW5 instead. **Text above provides the updated selective deterministic gateway steering described below to achieve the goal. I think that may have been the intent of the authors and I am just making it more clear** As for problem statement as GW load balancing can occur in the underlay as stated easily that is not the problem. In my mind I am thinking the problem statement that we want to describe in both the Abstract and Introduction is not vanilla simple gateway load balancing but rather a predictable deterministic method of selecting gateways to be used that is each VPN prefix now has a descriptor attached - tunnel encapsulation attribute which contains multiple TLVs one or more for each “selected gateway” with each tunnel TLV contains an egress tunnel endpoint sub-tlv that identifies the gateway for the tunnel. Maybe we can have in the sub-tlv a priority field for pecking order preference of which GWs are pushed up into the GW hash selected for the SR-ERO path to be stitched end to end. So lets say you had 10 GWs and you break them up into 2 tiers or multi tiers and have maybe gateway 1-5 are primary and 6-10 are backup and that could be do to various reasons so you can basically pick and choose based on priority which GW that gets added to the GW hash. I have some feedback and comments on the solution and how best to write the verbiage to make it more clear to the reader. I think in the solution as far s the RT to attach for the GW auto discovery. So with this new RT we are essentially creating a new VPN RIB that has prefixes from all the selected gateways that are discovered from the tunnel encapsulation attribute TLV. In the text here what is really confusing is if the tunnel encapsulation attribute is being attached to the underlay recursive route to next hop attribute or the VPN overlay prefix. So the reason I am thinking it is being attached to the VPN overlay prefix and not the underlay next hop attribute is how would you now create another transport RIB and if you are creating a new transport RIB there is already a draft defined by Kaliraj Vairavakkalai or BGP-LU SAFI 4 labeled unicast that exits today to advertise next hops between domains for an end to end LSP load balanced path. https://tools.ietf.org/html/draft-kaliraj-idr-bgp-classful-transport-planes-07 IANA code point below 76 Classful-Transport SAFI [draft-kaliraj-idr-bgp-classful-transport-planes-00] Also in line with CT another option is BGP-LU SAFI 4 to import the loopbacks between domains which is the next hop attribute to be advertised into the core end to end LSP. So the BGP-LU SAFI RIB could be used for the next GW next hop advertisement between domains so that there is visibility of all the egress PE loopback0 between domains. So you can either stitch the LSP segmented LSP like inter-as option-b SR-TE stitched and use nex-hop self PE-RR next-hop rewrite on each of the PEs within the internet domain or you could import all the PE loopback from all ingress and egress domains into the internet domain similar to inter-as opt-c create end to end LSP instantiate an end to end SR-TE path. Maybe you could attach the RT tunnel encapsulation attribute tunnel tlv endpoint tlv to the VPN overlay prefix. Not sure how that would be beneficial the underlay steers the VPN overlay. So maybe you could couple the VPN overlay new GW RIB RT to the transport Underlay CT CLAS RIB or BGP-LU RIB coupling may have some benefit but that would have to be investigated but I think is out of scope of the goals of this draft. I think we first have to figure out the goal and purpose of this draft by the authors and how the GW discovery should work in light of the CT class CT RIB AFI/SAFI codepoint draft that exists today as well as the BGP-LU option for next hop advertisement within the internet domain. Section 3 comments “Each GW is configured with an identifier for the SR domain. That identifier is common across all GWs to the domain (i.e., the same identifier is used by all GWs to the same SR domain), and unique across all SR domains that are connected (i.e., across all GWs to all SR domains that are interconnected). **No issues with the above** A route target ([RFC4360]) is attached to each GW's auto-discovery route and has its value set to the SR domain identifier. **So here if the RT is attached to the GW auto-discovery route we need to state is that the underlay route and that the PE does a next-hop-self rewrite of the eBGP link to the BGP egress domain next hop to the loopback0 so the GW next hop that we are tracking of all the ingress and egress PE domains is the egress and ingress PE loopback0.** Each GW constructs an import filtering rule to import any route that carries a route target with the same SR domain identifier that the GW itself uses. This means that only these GWs will import those routes, and that all GWs to the same SR domain will import each other's routes and will learn (auto-discover) the current set of active GWs for the SR domain.” **So if this is the case and we are tracking the underlay RIB and attach a route target to all the ingress PE & P next hops which is loopback0 = this is literally identical to BGP-LU importing all the loopbacks between domains or using CT class** There is no need for this feature to use the tunnel encapsulation attribute. I am not following why you would not use BGP-LU or CT clas RIB.** “To avoid the side effect of applying the Tunnel Encapsulation attribute to any packet that is addressed to the GW itself, the GW SHOULD use a different loopback address for packets intended for it.” **I don’t understand this statement as the next hop is the ingress and egress PE loopback0 that is the next hop being tracked for the gateway load balancing. The GW device subnet between the GW and PE is not advertised into the internet domain as we do next-hop-self on the PE PE-RR iBGP peering and so the GW to PE subnet is not advertised.** Looking at it a second time I think we are thinking here BGP-LU inter-as opt c style import of loops between domains and so instead of importing the loop0 which carries all packets on the GW device use a different loopback GW1 so it does not carry the FEC of all BAU packets similar concept utilized in RSVP-TE to VPN mapping "per-vrf TE" concept. “As described in Section 1, each GW will include a Tunnel Encapsulation attribute with the GW encapsulation information for each of the SR domain's active GWs (including itself) in every route advertised externally to that SR domain. As the current set of active GWs changes (due to the addition of a new GW or the failure/ removal of an existing GW) each externally advertised route will be re-advertised with a new Tunnel Encapsulation attribute which reflects current set of active GWs.” **What is the route being advertised externally from the GW. So the routes advertised would be all the PE loopback would be advertised from both ingress and egress domains into the internet domain and all loopback from the internet domain into the ingress and egress domain which could be done via BGP-LU or CT RIB – no need do reinvent the wheel and create a new RIB. So BGP-LU or CT RIB track the current set of active next hop GWs loopbacks between domains**If you do SR-TE stitching then you can do the next-hop self on each PE PE-RR for the load balancing and that would work and the load balancing would be to the PE loopbacks or if its an end to end SR-TE path using BGP-LU or CT RIB via importing all the PE loopbacks between domains the current set of active GWs would be tracked via the BGP-LU or CT RIB. So if the active GWs change due to GW failures they would be withdrawn from the BGP-LU or CT underlay RIB. No need now for the tunnel encapsulation attribute at least for the GW auto discovery load balancing** I think it still maybe possible to retrofit this draft to utilize the CT RIB or BGP-LU for the GW load balancing so nothing new has to be designed as far as the underlay goes, however maybe the idea of providing some visibility into the VPN overlay route to the underlay – maybe their maybe some benefit of using the tunnel encapsulation attribute RT import policy to attach to the VPN overlay prefixes. As CT draft provides a complete solution of providing the VPN overlay per VPN or per prefix underpinning of the VPN overlay to underlay CT RIB the problem statement is completely solved with either the CT draft or BGP-LU. Minor issues: None Nits/editorial comments: Please add normative and informative references below. I would reference as normative and maybe even informative the CT Class draft which creates a new transport class and I think this draft can really work well in conjunction with use of the CT class to couple the GW RIB created to the CT class transport RIB and provide the end to end inter-AS stitching via the PCE CC controller. I am one of the co-authors of this draft and I think this draft could be coupled with this GW draft to provide the overall goals of selective GW load balancing. https://tools.ietf.org/html/draft-kaliraj-idr-bgp-classful-transport-planes-07 I would also reference this draft for CT class PCEP coloring extension. https://tools.ietf.org/html/draft-rajagopalan-pcep-rsvp-color-00 As this solution would utilize a centralized controller PCE CC for inter as path instantiation for the GW load balancing, I think it would be a good idea to reference the PCE CC, H-PCE and Inter-AS PCE and PCE SR extension as informative and maybe even normative reference. -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
