Hi Don, [FK4] inline. Thanks, Florian On 10.01.24 01:34, Don Fedyk wrote: > On 09.01.24 02:34, Don Fedyk wrote: >> On 23.12.23 00:09, Don Fedyk wrote: >>> On 19.12.23 22:19, Don Fedyk wrote: >>> >>>> + >>>> +4. tsn-app-flow: >>>> + -> In order to be able to reconstruct the full original L2 packet, we need >>>> + to retain the L2 header. So we need something like: >>>> + >>>> ++---------------+ +-----------------+ >>>> +| Payload | | Payload | >>>> ++---------------+ -> +-----------------+ >>>> +| L2 (Original) | | L2 (Original) | >>>> ++---------------+ +-----------------+ >>>> + | ??? | >>>> + +-----------------+ >>>> + | L2 (new) | >>>> + +-----------------+ >>>> >>>> [Don] This is specified in IEEE TSN. DetNet uses IP/MPLS as the service sub layer. >>> >>> [FK] ??? here is a placeholder for UDP/IP/MPLS headers, sorry that this was not clear. >>> In RFC 9024 Figure 3 the following two options are depicted (translated into the notation of my mail): >>> >>> +---------------+ +-----------------+ >>> | Payload | | Payload | >>> +---------------+ -> +-----------------+ >>> | L2 (Original) | | L2 (Original) | >>> +---------------+ +-----------------+ >>> | MPLS | >>> +-----------------+ >>> | L2 (new) | >>> +-----------------+ >>> [Don2] The L2 (new) is not in the YANG but you have a L2 new if you look on the wire. You are using MPLS as the service sub-layer and forwarding sub-layer. L2 is hop by hop. That’s on purpose we do not specify TSN Ethernet encapsulation as a service/forwarding sub-layer in DetNet YANG. There are TSN config parameters that are specified in the IEEE. A box that supported TSN configuration could specify those parameters. See RFC 9023. If that box supported that our YANG then you would have a box that did both. So as far as RFC 9024 figure 3 the L2 case and the TSN case are the same for us. >>> See management section of https://datatracker.ietf.org/doc/rfc9023/ Section 5. >> >> >> [FK2] Yes, I agree that "L2 (new)" is not (and should not) be covered by the DetNet YANG. My current approach for that (and I hope that is how it is supposed to be) is to link the DetNet YANG and the TSN YANG just via the forwarding sub-layer's outgoing-interface (see for example https://mailarchive.ietf.org/arch/msg/detnet/paairBoBOB2ETRP-yKadT5Lvqn0/ ). >> >> [Don3] On ingress L2 caries a PCP that can be mapped to a Diff serv code point. 1-1 or all to 1 there is no PCP translation table. On (final) egress the original PCP is visible. If you exit the box with an IP Forwarding sublayer there could be a L3 (Diffserv) to L2 (PCP) mapping of the local interface L2. (This is not in DetNet). > > > [FK3] Yes, DSCP and PCP are one option for DetNet flow and TSN stream identification, respectively. But in my understanding (according to RFC 8939 5.1 and IEEE 802.1CB/CBdb) using for example source IP, destination IP and destination port for flow/stream identification as I did in the example is equally valid and then I do not necessarily need to care about DSCP or am I wrong? > > [Don4] Normally I would expect the forwarding layer (outer header) to carry a specified traffic class. You have set it at the dot1cb layer. Your example is outside what we considered in the DetNet yang but could work. [FK4] I think that really depends on the application. If the PCP/DSCP is actually mapped to a reasonable globally valid traffic type (like Figure 6 in 60802 with Isochronous, Cyclic-Synchronous...), then it totally makes sense and its likely the most common approach. But I am more concerned with isolating traffic that has the same priority from the application point of view and in that case the PCP/DSCP can be one, but not necessarily the only way of identifying the flow. Anyway, I do not see any issues at the moment applying this to the current DetNet YANG, but we will see how it turns out when we have fully implemented it ;-) >>> >>> +---------------+ +-----------------+ >>> | Payload | | Payload | >>> +---------------+ -> +-----------------+ >>> | L2 (Original) | | L2 (Original) | >>> +---------------+ +-----------------+ >>> | MPLS | >>> +-----------------+ >>> | UDP | >>> +-----------------+ >>> | IP | >>> +-----------------+ >>> | L2 (new) | >>> +-----------------+ [Don2] This stack is >>> supported. (same comment about L2 though) >>> >>> (L2 can either be TSN or something else, therefore there are three >>> options in RFC 9024 Figure 3.) >>> >>> I would expect that at least these two cases could be differentiated via YANG somehow (either in the draft-ietf-detnet-yang or in another model). What do you think? >>> >>> [Don2] As I recall the service sub-layer and forwarding sub-layer were heavily discussed in the DetNet WG as being IP and/or MPLS. The DetNet architecture shows the layer clearly - we support IP and MPLS - IP tunneling was not in scope - there were no DetNet ID that specified tunneling at least when the effort was undertaken. DetNet works with TSN both as an application and as a TSN under lay application/service/forwarding layer. In DetNet we only specify the IP/and MPLS for service and forwarding sub-layers because that is DetNet scope. We detailed many combinations with the cases so that people could see the combinations and the WG wanted it captured in the ID. Our PowerPoint became the SVG and config examples in the appendix. >> >> >> [FK2] There is at least GRE mentioned as IP tunneling protocol in Figure 4 of RFC 8655, but I am fine with that if configuring GRE, VXLAN, L2TP3 or similar is not in scope of the YANG (or even DetNet in general) at the moment. >> [Don3] OK. >> >> >>> But there are certainly some more possibilities like: >>> >>> +---------------+ +-----------------+ >>> | Payload | | Payload | >>> +---------------+ -> +-----------------+ >>> | L2 (Original) | | L2 (Original) | >>> +---------------+ +-----------------+ >>> | UDP | >>> +-----------------+ >>> | IP | >>> +-----------------+ >>> | L2 (new) | >>> +-----------------+ >>> >>> But I have not seen that being specified anywhere yet, but it would be one option to realize the "TSN over IP over TSN without MPLS" case. >>> >>> [Don2] Again nothing in our model to prevent this but the if the L2 is TSN then same comments apply. >> >> >> [FK2] I am sorry, but that is what I still don't understand. If you >> say that all three stack options >> - L2 over MPLS >> - L2 over MPLS over UDP over IP >> - L2 over UDP over IP >> are supported by the model, how does the node know which one of these three options (or any potential further ones) it should apply? >> >> [Don 3] The app service layer on ingress is a match on interface, or L2 Src/Dest MAC address or , VLAN, or PCP, The application has a service-sub-layer-ref which is populated by the configuration of the app-flow-ref on the service sublayer. You have a forward and backward reference that links them together. The same goes for the >> Forwarding and service sub layer. (You have 1/2 of it in your email >> example you are missing the service-sub-layer-ref but that is auto >> populated by the service sublayer reference - We would not want >> conflicting references between layers, so we chose which ones to make >> config true (default) and which ones to make config false.) > > > [FK3] OK, I think I see my misunderstanding now. I somehow thought it should be possible to specify "MPLS over UDP over IP" with a single service-sub-layer, but if I now understand correctly, I would use two service sub-layers in succession, right? The first one would add the MPLS header, while the second one would add the UDP and IP headers. And I link these two layers with outgoing->service-sub-layer->aggregation-sub-layer (even though it is strictly speaking no aggregation of multiple flows in this particular case). And for the other two options I just use one of the both service sub-layers. Is that correct? > > [Don4] Normally this would happen with the ssl-1a, ssl-1b etc adding MPLS labels and the fsl doing aggregation of the multiple ssls. One stack. > The aggregation options in the ssl are used when merging ssls (or fsls) at a DetNet node doing service layer aggregation. [FK4] Oh, I see. So its is more like: - L2 over MPLS: Service sub-layer adds MPLS header (and optionally the forwarding sub-layer adds another label) - L2 over MPLS over UDP over IP: Service sub-layer adds MPLS header, and >forwarding sub-layer< adds the UDP and IP header - L2 over UDP over IP: Service sub-layer adds UDP and IP header, forwarding sub-layer adds nothing Is that correct now? >>>> +I know there are several different ways to realize this, including >>>> +just transmitting the L2 packet over MPLS (apparently not IETF >>>> +standardized?) or several others like L2TPv3 (RFC3931) or VXLAN >>>> +(RFC >>>> +7348) if no MPLS labels are to be added (like in case 1 above). >>>> >>>> + >>>> +I would expect the YANG model (either this or a linked one) to >>>> +somehow specify which of these options shall actually be applied. >>>> +Otherwise, two DetNet routers might not be able to communicate with >>>> +each other, because one might use L2TPv3 and the other one VXLAN. >>>> +And also for example in the case of VXLAN, we would need the option to specify a VXLAN Network Identifier. >>>> >>>> [Don] The scope of the YANG does what DetNet documents outline. There is no VXLAN or L2TP3 specified in the DetNet when this applied. >>>> The DetNet YANG works by specified the whole path and applying the appropriate encapsulation format at the appropriate places. Yes the encapsulation specified on each device must match just the way SDN controllers specify packet forwarding in several technologies. >>> >>> [FK] You have convinced me that it does not make sense to hold back the YANG model until someday someone might come along who finds a use case for VXLAN or L2TP3 in DetNet. And there is still the option to specify an extension to the YANG model in that particular spec if needed. >>> >>> >>>> uses data-flow-spec; >>>> choice application-type { >>>> description >>>> "This is the application type choices."; >>>> +--- >>>> +fk: This part is quite confusing to me, especially since it does not directly mirror the ingress since. >>>> +I guess one reason for that is there could be multiple interfaces that could be selected based on the packet headers? >>>> +But for the same reason you could also have multiple ingress interfaces. >>>> [Don] For Egress the options are different because we get here by decapsulating the service sub-layer. We have either some type of TSN frame (Ethernet) or IP/MPLS but we are basically only associating the flow with an interface at the end of a service sub-layer. >>> >>> [FK] So the TSN case is clear to me because in both cases you have zero or one interface (app_flow->ingress->interface and app_flow->egress->ethernet->interface). >>> [Don2] You can have many interfaces. (see below). >> >> >> [FK2] But only if you specify multiple app flows, not for a single app flow, right? >> >> [Don3] I'm not sure what you want. Are you saying you have a single application flow that arrives on two (ore more) distinct ingress interfaces? On ingress that is not an issue for the YANG we just call it app-1a on eth0 and app1-b on eth1. Those are simple DetNet names not necessarily separate application flows. But is it really the same application? Are the applications packets load-shared over the two interfaces? If so then I would direct each interface to a separate service sublayer then aggregate that at the forwarding-sublayer. Then on egress the separation at service sub-layer would allow you to separate the flows back to two (or whatever) interfaces even though they are a single application. All you burn is app-1a app-1b ssl-1, ssl-2 etc. names. But we don't have in the model a way to take a single aggregated app flow from a single interface on ingress and load-share it dynamically out multiple interfaces without an L2 or L3 discriminator that directs the separation. We could filter by static eg VLAN to app2-a and app2-b but not a dynamic load share. If your egress was a physical L2 LAG you would still direct to a single logical interface. > > > [FK3] See below for an aggregated comment. > > >>> >>> But in the case of IP or MPLS we have still zero or one interface for ingress (app_flow->ingress->interface), but since there can be multiple different next hops for A SINGLE app-flow, we can have multiple interfaces for egress (app_flow->egress->ip-mpls->next_hop*->outgoing_interface). >>> >>> That means the current model allows to split up A SINGLE app-flow like this: >>> >>> +---------------+ >>> ---> | Destination 1 | >>> +----------+ +---------+ +--------+ / +---------------+ >>> | Source 1 | -> | Ingress | -> | Egress |-- >>> +----------+ +---------+ +--------+ \ +---------------+ >>> ---> | Destination 2 | >>> +---------------+ >>> >>> But not to merge one like this: >>> >>> +----------+ >>> | Source 1 | -- >>> +----------+ \ +---------+ +--------+ +---------------+ >>> --> | Ingress | -> | Egress | -> | Destination 1 | >>> +----------+ / +---------+ +--------+ +---------------+ >>> | Source 2 | -- >>> +----------+ >>> >>> [Don2] You can aggregate incoming stream at all layers app/service/forwarding we have examples of that in the appendixes. >>> >>> So either we have exactly one source and one destination for an app flow. >>> [Don2] I may be missing your point several examples show multiple interfaces. Case A1 in the appendix shows aggregation on multiple interfaces. >> >> >> [FK2] Do you really meant A1? It has eth0 as ingress for both app-flows. >> [Don3] Right B-1 has two interfaces sorry. Binding of app1 to interface is just a name mapping. But the point is app-1 could be app-1a and app-1b (same TSN flow on two interfaces.) The caveat is as I said before if you aggregated a flow on ingress you are flooding on egress to multiple interfaces unless you did parallel service-sublayers. In the middle of the DetNet Network it a single flow on the wire. >> >> >>> Then why do we have multiple next hops for egress? >>> [Don2] You are talking IP? If the application is IP it is forwarded as IP natively, so we support IP forwarding if the layer is IP. >> >> >> [FK2] It is the same for both IP and MPLS in the YANG model. For both there is the option to have a next-hop-list with multiple entries: >> [Don3] Yes IP and MPLS do have ways to dynamically distribute to multiple interfaces through routing. >> >> | +--rw (application-type)? >> | +--:(ethernet) >> | | +--rw ethernet >> | | +--rw interface? if:interface-ref >> | +--:(ip-mpls) >> | +--rw ip-mpls >> | +--rw (next-hop-options)? >> | +--:(simple-next-hop) >> | | +--rw outgoing-interface? >> | | | if:interface-ref >> | | +--rw (flow-type)? >> | | +--:(ip) >> | | | +--rw next-hop-address? >> | | | inet:ip-address-no-zone >> | | +--:(mpls) >> | | +--rw mpls-label-stack >> | | +--rw entry* [id] >> | | +--rw id uint8 >> | | +--rw label? >> | | | rt-types:mpls-label >> | | +--rw ttl? uint8 >> | | +--rw traffic-class? uint8 >> | +--:(next-hop-list) >> | +--rw next-hop* [hop-index] >> | +--rw hop-index uint8 >> | +--rw outgoing-interface? >> | | if:interface-ref >> | +--rw (flow-type)? >> | +--:(ip) >> | | +--rw next-hop-address? >> | | inet:ip-address-no- >> | | zone >> | +--:(mpls) >> | +--rw mpls-label-stack >> | +--rw entry* [id] >> | +--rw id >> | | uint8 >> | +--rw label? >> | | rt-types:mpls- >> | | label >> | +--rw ttl? >> | | uint8 >> | +--rw traffic-class? >> | uint8 >> >> >> >>> Or we allow multiple sources and destinations per app flow, then we need to support multiple interfaces at ingress, too. Or do I overlook something? >>> [Don2] You can have App1 on interface eth0 and App 2 on interface eth1 etc - there are examples e.g. Figure 4. For IP you have wild card IP prefixes as aggregation or multiple Application definitions. There is an ingress config and egress config per flow direction. >> >> >> [FK2] Yes, there are several examples how to aggregate MULTIPLE app flows into one flow in the service or forwarding sub-layer. And by using wild card IP prefixes I can also have a single app flow for multiple source, but only if they come in via the same interface, not if they come in via different interfaces. In the latter case, I only have the fallback option to use multiple app flow definitions each with a different interface. That would be fine, but for egress it is sufficient to use a single app flow definition and still specify multiple interfaces via the next-hop-list. And that is what is confusing me. >> >> [Don3] For ingress IP you don’t specify the interface just the IP match criteria. You might need multiple app-1a app-1b etc to specify a complicated IP flow on ingress. >> If you need to specify a set of interfaces. - not all - then on ingress you currently would need a per interface spec app-1a app-1b. We support 1:1 and all to one for IP. On egress the next hop does allow multiple interfaces. But the name app1 is just a name the real IP application can be app1-a app1-b app1-c ... >> The YANG is for a unidirectional app flow and the ingress does not have be the exact opposite of the ingress in the other direction just support the DetNet flow headers. >> YANG names are strings, and you can use whatever mnemonic you wish. > > > [FK3] Ok, I agree that in most applications it would likely be more suitable to aggregate on a different sub-layer. And if you still want to have this, you could either use app-1a, app-1b... or just leave out the interface for a match-all interfaces? > I was somehow expecting leaving out the interface just means "this app-flow is not coming in through an interface but is internally generated" and not "try to match on all interfaces". I would guess it can get quite interesting (and potentially inefficient) if the app-flow also tries to match traffic from the DetNet side interface (i.e. the forwarding->sub_layer->incoming->forwarding_id->interface), but if we are careful with assigning the IP 6-tuple management it should still work. > > And this "match-all interfaces for app-flow" is the only reason to have the option "to dynamically distribute to multiple interfaces through routing" at egress as you write above? Or is there another reason? > > [Don4] The IP/MPLS case - routing in routing out were the ones that comes to mind. Yes. For Ethernet probably specific interfaces is more efficient. [FK4] Ok. I still think some applications could benefit from a change like the following making their modeling more compact. But, as you have correctly outlined, there are already ways in the current model to mimic that by replicating the app-flow entries, so it is definitely not a blocker. --- 1/draft-ietf-detnet-yang-19.txt 2024-01-10 11:26:06.019319474 +0100 +++ 2/draft-ietf-detnet-yang-19-fk.txt 2024-01-10 11:26:06.067319700 +0100 @@ -1454,25 +1454,29 @@ default none; config false; description "Status of ingress application flow. This is an operational status and defaults to none if incomplete."; reference "RFC 9016 Sections 4.1, 5.8"; } - leaf interface { - type if:interface-ref; + container interfaces { description "Interface is used for any service type when matching all flows to the interface."; + leaf interface { + type if:interface-ref; + description + "This is an ingress interface."; + } } uses data-flow-spec; } //End of app-ingress container egress { description "Egress DetNet application flows or a compound flow."; uses data-flow-spec; choice application-type { description "This is the application type choices."; > >> >>> >>> >>>> /detnet/service/sub-layer/incoming/app-flow: This links applications >>>> to services. >>>> @@ -2608,6 +2713,14 @@ >>>> | +--rw operation? operation >>>> | +--rw incoming >>>> | | +--rw (incoming)? >>>> +--- >>>> +fk: One thing I raised before, but it either my comment got lost or i missed the response: >>>> +Why is there no incoming forwarding sub-layer for the service sub-layer? >>>> +An app-flow has outgoing service-sub-layer and incoming >>>> +service-sub-layer, a forwarding sub-layer has incoming >>>> +service-sub-layer and outgoing service-sub-layer, a service >>>> +sub-layer has incoming app-flow and outgoing app-flow and it has outgoing forwarding sub-layer, but NO incoming forwarding sub-layer. >>>> [Don] Sorry I missed your comment before. >>>> The operation of Service-sub-layer to forwarding sub-layer is an association of an encapsulation operation. The reverse direction is simply a removal of the forwarding layer and a match in the now exposed service Sub-layer. Therefore, you do not need the same level of description. >>> >>> [FK] Yes, but what do I specify as service->sub-layer->incoming for an egress node? Just keep it empty? Especially if no MPLS is involved. >>> >>> [Don2] Please look at some of examples of aggregation/ disaggregation. Note at egress we call it outgoing service sub-layer service-disaggregation. Remember there are 2 configs for one each end for a unidirectional flow. Also depending on the example, you may have app->service->forwarding or just app-service or just service->forwarding or just forwarding etc. It depends on what operations you need. >>> >>> For a disaggregation example with 2 interfaces see Figure 23. There is an aggregated MPLS forwarding sub-layer afl-1 when the label is matched it pops the label and trys matching fsl-1 and fsl-2 -- if fsl-1 is the matched label swap and forward the next label out one interface. If fsl- 2 matches it swap and forwards out a different interface. And this logic is used for other disaggregation at other layers. This case is Forwarding sub-layer outgoing forwarding-disaggregation. >> >> >> [FK2] I meant an example for a final egress node (like the ones labeled Egress 1 or 2 in the pictures) and if I do not overlook something, there is none provided. But anyway, I would expect derived from asl-1 in example C-3 that the service sub-layer for Egress 1 in example A-1 should look like this: >> >> "service": { >> "sub-layer": [ >> { >> "name": "ssl-1", >> "service-rank": 10, >> "traffic-profile": "pf-2", >> "service-protection": { >> "protection": "none", >> "sequence-number-length": "long-sn" >> }, >> "operation": "termination", >> "incoming": { >> "service-id": { >> "mpls-label-stack": { >> "entry": [ >> { >> "id": 0, >> "label": "102" >> } >> ] >> } >> } >> }, >> "outgoing": { >> "app-flow": { >> "flow": [ >> "app-0", >> "app-1" >> ] >> } >> } >> } >> ] >> } >> >> So in a case where there is no MPLS S-label to be removed (like in the L2 over UDP over IP case mentioned above), it would just look like this: >> >> >> "service": { >> "sub-layer": [ >> { >> "name": "ssl-1", >> "service-rank": 10, >> "traffic-profile": "pf-2", >> "service-protection": { >> "protection": "none", >> "sequence-number-length": "long-sn" >> }, >> "operation": "termination", >> "incoming": { >> }, >> "outgoing": { >> "app-flow": { >> "flow": [ >> "app-0", >> "app-1" >> ] >> } >> } >> } >> ] >> } >> >> So the incoming container would be empty. That might be as it is supposed to be, but it just looks odd to have an empty incoming container, especially since - as mentioned before - all other pairs of sub-layers have the option to specify the link between the layers on both sides (incoming and outgoing), but not this specific case. >> >> [Don3] I would specify the Incoming as IP: ip-detnet-flow with an IP UDP filter. Otherwise, you may pull in non DetNet flows. >> }, >> "operation": "termination", >> "incoming": { >> src-ip-prefix: "10.0.2.2/16" >> protocol-next-header: 17, <-- UDP >> source-port : 1000 <--optional >> }, >> > > [FK3] Good point! That is exactly the problem, because it means I have to perform the flow identification AGAIN in the service sub-layer after I have already performed the same identification in the forwarding sub-layer. Yes, for disaggregation this is certainly necessary because I have to match on different things, but if you have just one forwarding sub-layer and one service sub-layer it would be much easier and more efficient if I could just specify something along the lines of the following, because it would allow me to only process traffic that is coming from fsl-1: > > "service": { > "sub-layer": [ > { > "name": "ssl-1", > "service-rank": 10, > "traffic-profile": "pf-2", > "service-protection": { > "protection": "none", > "sequence-number-length": "long-sn" > }, > "operation": "termination", > "incoming": { > "forwarding-sub-layer": { > "service-incoming": [ > { > "sub-layer": "fsl-1" > } > ] > } > }, > "outgoing": { > "app-flow": { > "flow": [ > "app-0", > "app-1" > ] > } > } > } > ] > } > > [Don4] If you configure one to one forwarding sub-layer (fsl) to service sub-layer (ssl) you are only looking up the flow ID at the fsl which is linked to a single instance SSL that is linked one or more APPs which could be linked to interfaces. Internally this would be the same as having the apps at the fsl layer outgoing. In your example you have a sequence number at the service layer so that needs to be checked and removed. [FK4] Yes, agreed. But my point is that it looks like the direct linking of fsl and ssl in this way without repeated flow identification is currently not possible without adding something like --- 1/draft-ietf-detnet-yang-19.txt 2024-01-10 13:10:50.984132397 +0100 +++ 2/draft-ietf-detnet-yang-19-fk.txt 2024-01-10 13:10:51.032133011 +0100 @@ -1582,20 +1586,28 @@ uses forwarding-sub-layer-group; } container service-id { description "This service sub-layer is related to the service or forwarding sub-layer of the lower layer and provide DetNet service relay or termination at the relay node or egress node."; uses detnet-flow-spec; } + container forwarding-sub-layer { + description + "This service sub-layer is related to the forwarding + sub-layer of the lower layer and provides + DetNet service relay or termination at the relay + node or egress node."; + uses forwarding-sub-layer-group; + } } } container outgoing { description "The DetNet service sub-layer outgoing configuration."; choice outgoing { description "The outgoing type may be a forwarding Sub-layer or a service sub-layer or aggregation type."; -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
