Transcription
Global vision.Local knowledge.Cisco Connect Dubrovnik27.-29.3.2019.
EVPN in Service Provider networkDejan JaksicSytstems Engineer Service Providers28-3-2019
Agenda Planet EVPN motivation EVPN Basics Network Fabric Architecture EVPN-VPNv4 interconnect EVPN and VPLS seamlessintegration EVPN positioning in SP network Conclusion
EVPN: Value PropositionDeploy with EaseCreate New RevenueStreams Network as a service through fabricdesigns E-LAN, E-LINE, E-TREE, L3, IRBServicesProtect Investments Unified Networks on single overlay Simplify protocols and operations Industry adoption and standardization Seamless Brownfield Integration Same principles and operationalexperience as IP VPNsEVPNIncrease Availability Workload Mobility Optimal forwarding All-Active Redundancy with FastConvergenceFast, Resilient, Flexible Unified Services
EVPN Unified Services AttributesSeamlessDeploymentOptimal EastWest trafficdeliveryAccess ServicesAll-Active MultiHomingEVPNWorkload MobilityFastConvergenceEVPN external Hub: https://e-vpn.io/Per-FlowRedundancyand loadbalancing
Evolution of ETFTRILL2008IEEE802.1ah(PBB)20112015IETFEVPN
Stolen Data Center requirements J§ Flexible service/workload placement§ Multi-tenancy with L2 and L3 VPN§ Optimal Forwarding, Workload mobility§ Fast Convergence§ Efficient bandwidth utilization EVPN with a choice of data plane encapsulation (MPLS/SR, VxLAN, SRv6) isthe designed technology to address these requirements.
What is EVPN?EVPNEVPN family introduces nextgeneration solutions for EthernetservicesP2P BGP control-plane for Ethernet Segmentand MAC distribution learning over MPLSand VXLAN data-planeMultipointEVPN-VPWS Same principles and operationalexperience as in IP VPNsNo use of PseudowiresRFC 7432RFC 8214Multi-vendor solutionsCisco leader in industrystandardization efforts (RFCs/Drafts)RFC forwarding
ConceptsEVPN Instance (EVI)Ethernet SegmentSHDEVIBDEVIBDPE EVI identifies a VPN in thenetwork Encompass one or morebridge-domains,depending on serviceinterface typePort-basedVLAN-based (shown above)VLAN-bundlingBGP Route AttributesRoute TypesExtended CommunitiesCE1ESI1MHDBGP RoutesPE1CE2ESI2PE2 Represents a ‘site’connected to one or morePEs Uniquely identified by a 10-byte global EthernetSegment Identifier (ESI) Could be a single deviceor an entire networkSingle-Homed Device (SHD)Multi-Homed Device (MHD)Single-Homed Network (SHN)Multi-Homed Network (MHN)[1] Ethernet Auto-Discovery (AD) RouteESI MPLS Label[2] MAC/IP Advertisement RouteES-Import[3] Inclusive Multicast RouteMAC Mobility[4] Ethernet Segment RouteDefault Gateway[5] IP Prefix Advertisement RouteEncapsulation New SAFI [70] New BGP extended Routes serve controlcommunities defined Expand informationcarried in BGP routes,including:plane purposes,including:MAC address reachabilityMAC mass withdrawalMAC address movesSplit-Horizon label adv.Redundancy modeAliasingMAC / IP bindings of a GWMulticast endpoint discoverySplit-horizon label encodingRedundancy group discoveryData plane EncapsulationDesignated forwarder electionIP address reachabilityL2/L3 Integration
Service Provider Network - Simplification JourneyUnified MPLSProvisioningEPN 5.0CompassMetro N ServicesInter-Domain CPFRR or TELDPBGPBGP-LULDPBGPBGP-LURSVPIntra-Domain CPhttps://xrdocs.io/design/LDPIGPBGPIGP with SRIGP with SR
Drastic Network Protocols ReductionBag of existing TPPWE3ISISOSPFRSVP-TELACPMC-LACPMP-BGPLDPLDP-TEIP OAMMPLS OAMEthernet OAMSTPG.8032RADIUSSNMPSyslogNetflowSSH CLI/XMLHSRP/VRRPNext Gen.ProtocolsSRv6SR (MPLS)PCEPISISBGP (TE, LS)IP OAMEthernet OAMEVPNNETCONF/YANGSSHKey enabler forReducing operations complexity§ Simpler automation§ Simpler to repair§ Simpler integration§ Foundation for service Orchestration
Why was EVPN needed?Solving VPLS challenges for per-flow Redundancy Existing VPLS solutions do not offer an All-Active per-flow redundancy – VPLStechnology lacks the capability ofpreventing L2 loopsM1M2CE1PE1PE3PE2PE4CE2Echo ! Looping of Traffic Flooded from PE (BUM) Duplicate Frames from Floods from theDuplicate !M1M2CE1Core MAC Flip-Flopping over Pseudowire E.g. Port-Channel Load-Balancing does notproduce a consistent hash-value for aframe with the same source MAC (e.g. nonMAC based lopPE3PE4CE2
Why was EVPN needed in 2016 ? Network Operators have emerging needs in their network:Data center interconnect operation (DCI) Cloud and Services virtualization (DC) Remove protocols and Network Simplification (ICCP, HSRP/VRRP) Integration of Layer 2 and Layer 3 VPN Services What about PBB-EVPN? ASR9k only and slowing down investmentsEVPN is all about BGP Control Plane and Services!!!
From MAC Bridging to MAC RoutingCommon BGP Control PlaneL3VPN – VPNv4/6, L2/3VPN – EVPN, EVPN-VPWSIP,MPLS,VXLANWAN/CoreAccessPE2Existing Solution:DCI2LeafA1overlapDCI1PE1UnderlayData Center NetworkVMLeafService Provider NetworkIP,MPLS,VXLANVMLeafIP, MPLSSpine SpineEvolution:OverlayVML2/L3VPN (BGP,T-LDP) - VPLS, EoMPLSVPLS, OTVTrill, Fabric-PathOverlayIP, IGP, MPLS (LDP), RSVP-TE, BGP-LUIP, MPLS, L2L2, STP, VLANUnderlay
MPLS Transport & BGP ServiceBGP L3VPNBGP SignalingBGP SignalingPE2BGP SignalingPE1PE2CE2PE3PE4MPLSCE1PE1Data PlaneIP PacketBGP SignalingPE4MPLSCE1BGP EVPNCE2PE3Data PlaneTransportMPLS LabelServiceBGP LabelIP PacketIP PacketL2 FrameTransportMPLS LabelServiceBGP LabelL2 FrameL2 Frame
EVPN - Ethernet VPN Concepts are same!!! Pick your side!Pick your side!SP1SP2PE2PE4PE1PE3CE1L1L2C1VML4L3C2VMVMVM
EVPN vs VPNv4/6 or BGP Control Plane? BGP integrates services with programmable SR transport Common across L2 / L3 services Services Control Plane is BGP with different AF / SAFI Single Service Control Plane is easy to manage and troubleshoot HUGE investment in existing VPNv4/6 EVPN doesn’t replace L3VPN VPNv4/6 - no technical reason to do it!
EVPN Flavors Multi-Homed All-Active Ethernet Access Standards-based Multi-chassis / Cluster Control Plane Replacement of: vPC, VSS, nVCluster, etc.Replacement of: HSRP, VRRP, etc.Carrier Ethernet Today Replacement of: mLACP, STP, T-LDP, BGP-AD, etc.E-LINE - 80% of SP’s L2VPN portfolio (PWs)E-LAN - Smaller # of L2 Multipoint VPN servicesThere isn’t other standard technology with Ethernet All-Active Multihoming
EVPN - Positioning EVPN should be door-opener for IOS XR in Next Generation CO(Network Fabric) EVPN L2/L3 multipoint brings optimal forwarding, MAC mobility, all-activeMH accessBGP Control-Plane Simplifies DCI/Border-Leaf configuration and service provisioning Provides common SLA signaling
EVPN - Ethernet VPN Leafs run Multi-Protocol BGP to advertise & learn MAC/IP addresses over theNetwork FabricMAC/IP addresses are advertised to rest of LeafsSPSPLLMAC/IP advertisement &learning via BGP EVPN NLRILLData Plane learningfrom the hostsCVMCVMVMVMAll Active multi-homingEthernet Segment
XR CLI:Step 1:Step 2:
EVPN - Ethernet-Segment for Multi-HomingThe bundle on the Leafsconnecting to a node shouldhave Identical ES identifier (ESI)L1Unique 10-byte global identifierper Ethernet SegmentSP1SP2L2L3C1VML4Ethernet Segment represents anode connected multiple LeafsC2VMVMVM
EVPN – Designated Forwarder (DF)Challenge:How to prevent duplicate copies of flooded traffic from being delivered to a multi-homedEthernet Segment (BUM traffic)?L1C1SP1SP2L2L3NDFDuplicateL4C2DF
EVPN – Split HorizonChallenge:How to prevent flooded traffic from echoing back to a multi-homed Ethernet Segment?BUM LabelTransportLabelSP2SP1SH LabelL1L2C1VMEcho !VM
EVPN – MAC Mass-WithdrawChallenge:How to inform other Leafs of a failure affecting many MAC addresses quickly while thecontrol-plane re-converges?MAC1 can bereached via ESI1L1MAC1 can NOT bereached via ESI1SP1SP2L2L3C1VMMAC1 à ESI1 à Leaf1 Leaf2L4C2VMESI1 MAC1VMVM
BUM Broadcast, Unknown unicast, MulticastEVPN – BUM Ingress ReplicationSP2SP1MBUL2MBUMBUL1L4L3MBUC1VMC2VMVMVM
R36, R37, R38, R39 - EVPN StartupR36 - Example1.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryService Carving: 100 modulo 2 0R36 is DF for EVI-100LACPR39H2R38R35RT-4 - DF ElectionLACPR37H1R34RD: 1.1.1.36:1ESI: 0036.3700.0000.0000.1100R36Ext-Com: 3637.0000.0000 (RT)
R36, R37, R38, R39 - EVPN StartupR36 - Example1.2.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)LACPR39H2RT-1 - Per ESI Ethernet ADR38R35RD: 1.1.1.36:1ESI: 0036.3700.0000.0000.1100LACPR37H1Ext-Com:R34 Flag:0x00 All-ActiveSplit-Horizon Label: 64005Ext-Com: 1:100 (RT)R36
R36, R37, R38, R39 - EVPN StartupR36 - Example1.2.3.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)RT3: Inclusive MulticastLACPR39H2RT-3 - Inclusive MulticastR38R35RD: 1.1.1.36:100Ext-Com: Type 6 Ingress-ReplicationMulticast(BUM) Label: 64120LACPR37H1R36R34Ext-Com:1:100 (RT)
BUM Forwarding1.2.3.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)RT3: Inclusive MulticastLACPR39H2LACPR38R35R37R34H1Transport Label R38-9R36BUM - TrafficIRBUM Label R38-9/EVI100BUM - Traffic
BUM Forwarding1.2.3.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)RT3: Inclusive MulticastLACPR39H2R38XLACPR37R34Transport Label R37H1BUM - TrafficR35BUM Label R37/EVI100R36SH Label R37/ESIxIRBUM - Traffic
R36, R37, R38, R39 - EVPN StartupR36 - Example1.2.3.4.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)RT3: Inclusive MulticastRT2: MAC AdvertisementLACPR39H2RT-2 - MAC AdvertisementR38R35RD: 1.1.1.36:100ESI: fbd7Label: 64004R36Ext-Com: 1:100 (RT)L2 Frame SMAC:0062.ec71.fbd7per EVPN Bridge Domain label
Unicast Forwarding (looks familiar?)1.2.3.4.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)RT3: Inclusive MulticastRT2: MAC AdvertisementL2 Frame Flow1DMAC: H1Transport Label R36LACPR39H2LACPL2 Frame Flow1DMAC: H1R38R35R37R34H1R36L2 Frame Flow1DMAC: H1RT-2 MAC Label/EVI
EVPN Routes – Cheat SheetBGP a PlaneL2 FrameTransportMPLS LabelServiceBGP LabelEVI1-BUMLL2 FrameBD1 EVI1VRF1 ARPIP-A MAC-A - BVI1IP-B MAC-B - BVI2VRF1-AGGLBVI2MAC-BIP-BBD1 MACMAC-A - BE1.1VRF1.2BE1Vlan2BE1 - ESI1BVI1MAC-AIP-AVlan1BE1.1L2 FrameBD2 MACMAC-B - BE1.2BD2 EVI2EVI2-LBE1-SHLEVI2-BUMLPE1 – Advertises:RT-4 Ethernet Segment Route I have ESI1 in case when someone needs this information forDesignated Forwarder(DF) ElectionRT-1 Per ESI Ethernet Auto-Discovery (AD) Route I have ESI1 ESI1 is All-Active AC with ESI1 is connected to EVI1 and EVI2 My Split Horizon Label for ESI1 isBE1-SHLRT-1 Per EVI Ethernet Auto-Discovery (AD) Route(s) EVI1 per-EVI (Aliasing) Label is EVI2 per-EVI (Aliasing) Label is EVI1-LRT-3 Inclusive Multicast Route(s) EVI1 Label for BUM traffic is EVI2 Label for BUM traffic isEVI2-LEVI1-BUMLEVI2-BUMLRT-2 MAC/IP Advertisement Route(s) MAC-A/IP-A in EVI1 and IP-A in VRF1 via label MAC-B/IP-B in EVI2 and IP-B in VRF1 via labelEVI1-LEVI2-LRT-5 Prefix Advertisement Route(s) IPv4/6 prefix of BVI1 in VRF1 via label VRF1-AGGL IPv4/6 prefix of BVI2 in VRF1 via label VRF1-AGGL
EVPN – Distributed Anycast GatewayPurpose:Optimal intra and inter-subnet connectivity with seamless workload mobilityIdentical Anycast Gateway Virtual IPand MAC address are configured onall the LeafsDistributed Anycast Gateway servesas the gateway for connected hostsAll the BVIs perform active forwardingin contrast to active/standby like Firsthop routing VMVMVMVM
EVPN – IRB in Network netForwardingSPGWGWGWGWLLLLCVMCVMCVMCVML3-L2Subnet 1Subnet 240
Centralized vs. Distributed RoutingDistributed Routing (IOS XR)Centralized RoutingBoarderLeafCentralized GWL3L2FabricFabricL3Subnet 1 LeafLeafL2VLAN 1Subnet 2Optimized forwarding of east-west trafficARP/MAC state localized to LeafsHelps with horizontal scaling of DC VLAN 2All east - west routed traffic traverses to centralized gatewaysCentralized gateways have full ARP/MAC state in the DCScale challengeWe do NOT support this design!
Integrated Routing and BridgingSymmetric IRBAsymmetric IRBBoarderLeafBoarderLeafFabricFabricLeaf Flexible workload placement – any subnetanywhereBridge- Route/Route- Bridge (symmetric VNIin both directions)ARP/MAC state localized to Leafs Helps with horizontal scaling of DC Cisco supports ONLY this modeLeaf Egress subnet must be localBridge- Route- Bridge (Different (Asymmetric) VNIdepending on directions)Ingress Leaf needs ARP/MAC state for every egressleaf Limits scaleSymmetric IRB and Asymmetric IRB are NOT interoperable!
LR36, R37, R38, R39 - EVPN StartupR36 – Example L2/L3 serviceAnycast IRB 192.168.2.1/241.2.3.4.RT4: DF Election & Multi-Homed EthernetSegment Auto-DiscoveryRT1: Per ESI Ethernet Auto-Discovery(Split-Horizon, Mass-Withdraw)RT3: Inclusive MulticastRT2: MAC/IP AdvertisementIRBR39LACPH2IRBR38IRBLACPRT-2 - MAC AdvertisementR35ESI: Label: 64004(BD) 64008(VRF)R36L2 Frame SMAC:0062.ec71.fbd7RD: 1.1.1.36:100IP: 192.168.1.10Ext-Com: 1:100 (RT) VRF RTIP Header SurceIP:192.168.1.10Anycast IRB 192.168.1.1/24
EVPN - load-balancing modesAll-Active(per flow)Single-Active(per VLAN)Port-Active(per port)PE1PE1PE1PE2V1V1CESingle LAG at the CEVLAN goes to both PETraffic hashed per flowBenefits: Bandwidth, ConvergenceCan replace: vPC, VSS, nV clusterPE2PE2V2V1CEMultiple LAGs at the CEVLAN active on single PETraffic hashed per VLANBenefits: QoS, Billing, PolicingCan replace: HSRP/VRRPV1, V2CEMultiple LAGs at the CEPort active on single PETraffic hashed per portBenefits: Protocol SimplificationCan replace: ICCP MC-LAG
EVPN - load-balancing modesSingle-Flow-Active(access L2 GW)PE1PE2H1CE1H2CE2CE3Single LAG at the CEVLAN goes to both PEAccess takes care of L2 loopBenefits: Legacy support for STP,REP, G.8032
EVPN-VPWS Benefits of EVPN applied to point-to-point servicesNo signaling of PWs. Instead signals MP2P LSPs instead(like L3VPN) All-active CE multi-homing (per-flow LB) - today Single-active CE multi-homing (per-service LB) - roadmap Relies on a sub-set of EVPN routes to advertiseEthernet Segment and AC reachabilityPE discovery & signaling via a single protocol – BGP Per-EVI Ethernet Auto-Discovery route (RT1 only)!!! PEPECECEPEPE
EVPN – Flexible Cross-Connect ServiceChallenge:How to bring multiple access services from different sources using a single EVPN erMUXDEMUXVLAN translationNormalized VLANover unique tunnelSingle MPLS label usedPE
Flexible Cross-Connect Service: Head-EndPurpose:Bring multiple ELINE services into Pseudo-wire Head-end ingle tunnel)VRFs
EVPN - L3 Multi-Homing using EVLAGNo ICCP!HSRP/VRRP/MC-LAG ReplacementCore / Metro CE4EVLAGEVLAGPE1L3VPNorEVPNMulti-homing is provided by EVPN (EVLAG)Determine DF/ NDF PESynchronization (ARP, IGMP, etc.)
EVPN - Anycast-PWAccessAGCEA1AccessCore / Metro ast-SIDAnycast-SIDA3CE
EVPN - access VPWS (H-EVPN)Multi-Homed EVPN-VPWS - RoadmapAccessA1AGEVPN-VPWSCEA2AccessCore / Metro FabricEVPN-VPWSEVPNAGA3AGAGCEA4
EVPN and VPNv4/6 InterconnectDCI/BL provides EVPN to VPNv4/6 stitchingDCI/BL participates in L3 Routing not in L2 BridgingDCI/BL is mandatory, because of P - EVPNBGP - L3VPN VPNv4/6DC/COSCE1LEAF
EVPN and VPNv4/6 InterconnectDCI/BL provides EVPN to VPNv4/6 stitchingDCI/BL participates in L3 Routing not in L2 BridgingDCI/BL is mandatory, because of summarization!!!InterconnectBGP - EVPNBGP - L3VPN VPNv4/6RT5 Prefix prefix-CE2/24RT5 Prefix prefix-CE2/24RT5 Prefix prefix-CE1/24RT5 Prefix prefix-CE1/24XACE2AAccessAPABRCorePSDCI/BLLEAFRT2 MAC/IP CE1/32COSCE1LEAF
EVPN and VPNv4/6 InterconnectBGP - EVPNBGP - L3VPN VPNv4/6RT5 Prefix prefix-CE2/24 RT: VRF ART5 Prefix prefix-CE2/24RT: VRF A StitchingRT5 Prefix prefix-CE1/24 RT: VRF ART5 Prefix prefix-CE1/24RT: VRF A StitchingRT2 MAC/IP CE1/32RT: VRF A StitchingXDCI/BLVRF ARD DCI:0RT import/export: VRF A StitchingRT import/export: VRF A
EVPN and VPNv4/6 InterconnectEVPN to VPNv4/6 Re-AdvertiseBGP - EVPNBGP - L3VPN VPNv4/6RT5 Prefix prefix-CE1/24 RT: VRF A1. Import: RT: VRF A Stitching3. Filter RT2 /32 Router2. Advertise to vpnv4: VRF ADCI/BL - BGP Configurationrouter bgp 1address-family l2vpn evpnimport stitching-rt re-originateadvertise vpnv4 unicast re-originated stitching-rt!address-family vpnv4 unicastimport re-originate stitching-rtroute-policy rt2-filter outadvertise vpnv4 unicast re-originated!XRT5 Prefix prefix-CE1/24RT: VRF A StitchingRT2 MAC/IP CE1/32RT: VRF A StitchingDCI/BLVRF ARD DCI:0RT import/export: VRF A StitchingRT import/export: VRF A
EVPN and VPNv4/6 InterconnectVPNv4/6 to EVPN Re-AdvertiseBGP - EVPNBGP - L3VPN VPNv4/6RT5 Prefix prefix-CE2/24 RT: VRF A2. Advertise to EVPN: RT: VRF A Stitching1. Import: VRF ART5 Prefix prefix-CE2/24DCI/BL - BGP Configurationrouter bgp 1address-family l2vpn evpnimport stitching-rt re-originateadvertise vpnv4 unicast re-originated stitching-rt!address-family vpnv4 unicastimport re-originate stitching-rtroute-policy rt2-filter outadvertise vpnv4 unicast re-originated!DCI/BLVRF ARD DCI:0RT import/export: VRF A StitchingRT import/export: VRF ART: VRF A Stitching
VPLS & EVPN Seamless Integration - MigrationPE2CE2PE1BD1CE1VFI1PW PE2 UPPW PE3 UPPW PE4 UPPE4CE4MPLSPE3Migrate VPLS Network to EVPN Network through Seamless IntegrationCE3VFI1 is by default in Split Horizon Group 1 SHG1 protects loops in MPLS Core Full Mesh of pseudowires (PW) is requiredfor Any-to-Any forwarding
VPLS & EVPN Seamless Integration - MigrationPE2CE2PE1BD1CE1SHG1XVFI1PW PE2 UPPW PE3 UPPW PE4 UPEVI1PE4CE4MPLSVFI1 is by default in Split Horizon Group 1 SHG1 protects loops in MPLS Core Full Mesh of pseudowires(PW) is requiredfor Any-to-Any forwardingEVI1 is also by default in Split Horizon Group 1 PE1 doesn’t forward data between VFI1 and EVI1PE3CE3
VPLS & EVPN Seamless Integration - MigrationPE2CE2PE1BD1CE1SHG1XVFI1PW PE2 UPPW PE3 DOWNPW PE4 UPEVI1PE4CE4MPLSBGP EVPNVFI1 is by default in Split Horizon Group 1 SHG1 protects loops in MPLS Core Full Mesh of pseudowires(PW) is requiredfor Any-to-Any forwardingEVI1 is also by default in Split Horizon Group 1 PE1 doesn’t forward data between VFI1 and EVI1PE3CE3PE1&PE3 run BGP EVPN PW PE3 goes DOWN Data Forwarding between PE1 and PE3 via EVI1
EVPN – MVPN in the Network Fabric (in progress)XR 6.6.1 - IGMP L2 EVPN state syncSourceVRF(x)BLBLMVPNL3--L2LSPSPLLState syncin EVPNLEVI-xIGMP Join / LeavemcastevpnCCReceiverReceiverEVI-yIRB vrf(x)
EVPN – Service LayeringMulticastFXCEVPN-HEE-TREEP2PL2 BridgingL3 EPPEBLSPSPSPSPSPSPSPSPAggregationAccessBL“Shared or single tenant”LLLCE
Service Provider AAAGPE/PPPE/PAGoooooDistributed Anycast Gateway EVPN-IRBAll-active Multi-HomingEVPN symmetric IRBEVPN L2 for east-west trafficEVPN L3 for north-south trafficüüüüüüSeamless mobilityOpti
MAC Flip-Flopping over Pseudowire E.g. Port-Channel Load-Balancing does not produce a consistent hash-value for a frame with the same source MAC (e.g. non MAC based Hash-Schemes) CE1 Echo ! M1 M2 PE1 PE2 PE3 PE4 CE2 CE1 M1 M2 PE1 PE2 PE3 PE4 CE2 Duplicate ! CE1 M
