In Part 3 we built the Metro Ethernet service. In this phase we add a separate data-center fabric on six vQFX switches, running EVPN-VXLAN. It covers an IP underlay, an EVPN overlay, two VLANs stretched across the fabric with anycast IRB gateways, a routed tenant VRF, multihoming for a bonded host, and a handoff back to the enterprise AS 65000.
In the previous part we used spanning tree. It keeps one Layer 2 domain working across two sites, but it blocks every backup path except one. A data center needs every leaf to forward traffic on every uplink at the same time, so we will use EVPN-VXLAN instead. EVPN-VXLAN has two things: VXLAN and EVPN. VXLAN puts each Ethernet frame inside an outer IP packet, so the fabric only routes IP packets between loopback addresses. EVPN is the control plane, a set of BGP route types. These route types tell every switch which VTEP holds which MAC address, which subnet, and which multihomed segment. Because IP routing block links the way spanning tree does, we get ECMP on every uplink.
Overview
For this part, the tasks are:
- Build an OSPF underlay on the fabric links and loopbacks only, with ECMP between every leaf pair.
- Run an EVPN iBGP overlay with both spines as route reflectors in one cluster.
- Turn the two server leaves into VTEPs and stretch VLAN 100 and VLAN 200 across the fabric.
- Add anycast IRB gateways so a host uses whichever leaf it is attached to.
- Route between the VLANs through a
TENANT1VRF using pure Type 5 over L3 VNI 9999. - Multihome a bonded host to both server leaves with an all-active ESI-LAG.
- Hand the tenant VRF off to both enterprise cores over four eBGP sessions, advertising only summaries.
Topology
Underlay
We are going to use OSPF in the underlay, bring the MTUs up to 9216 to make room for the extra bytes, and implement a load balancing policy.
We need the load balance policy because without it only one of the equal cost paths to the spines is installed in the forwarding table and other spine link stops being used.
On DC1-SPINE01:
set interfaces xe-0/0/0 mtu 9216set interfaces xe-0/0/0 unit 0 family inet address 10.3.1.0/31set interfaces xe-0/0/1 mtu 9216set interfaces xe-0/0/1 unit 0 family inet address 10.3.1.2/31set interfaces lo0 unit 0 family inet address 10.0.10.11/32set protocols ospf area 0.0.0.0 interface xe-0/0/0.0 interface-type p2pset protocols ospf area 0.0.0.0 interface xe-0/0/1.0 interface-type p2pset protocols ospf area 0.0.0.0 interface lo0.0 passiveset routing-options router-id 10.0.10.11set routing-options autonomous-system 65010set policy-options policy-statement LB term 1 then load-balance per-packetset routing-options forwarding-table export LB| Command | Purpose |
|---|---|
mtu 9216 | Leaves room for the extra 50 bytes VXLAN adds to each frame |
policy-statement LB ... load-balance per-packet | Enables per-flow hashing, so both ECMP next hops get installed |
forwarding-table export LB | Applies the policy. without this the forwarding table keeps only one of the two OSPF paths |
Every other device uses the same config toward its own uplinks.
On DC1-SLEAF01:
show ospf neighborAddress Interface State ID Pri Dead10.3.1.0 xe-0/0/0.0 Full 10.0.10.11 128 3710.3.2.0 xe-0/0/1.0 Full 10.0.10.12 128 36show route forwarding-table destination 10.0.10.31/32 table defaultRouting table: default.inetInternet:Destination Type RtRef Next hop Type Index NhRef Netif10.0.10.31/32 user 0 ulst 131070 7 10.3.1.0 ucst 1736 8 xe-0/0/0.0 10.3.2.0 ucst 1737 8 xe-0/0/1.0Both adjacencies are Full, and the forwarding table carries two next hops toward DC1-BLEAF01's loopback, one through each spine.
EVPN overlay
EVPN uses BGP as its control plane. It carries MAC addresses, MAC-and-IP pairs, VNI membership, and multihomed segment information as NLRI in normal BGP UPDATE messages. We need route reflectors because iBGP does not advertise a route learned from one iBGP peer to another iBGP peer. Without them, every leaf would need a direct BGP session with every other leaf. The route reflectors learn the EVPN routes from one leaf and advertise them to the other leaves, so each leaf only needs BGP sessions with the two spines.
We put both spines in reflector cluster ID 10.0.10.11 because one reflector would be a single point of failure. Every leaf peers with both spines directly, so each spine learns every route straight from the leaf.
On DC1-SPINE01:
set protocols bgp group OVERLAY type internalset protocols bgp group OVERLAY local-address 10.0.10.11set protocols bgp group OVERLAY family evpn signalingset protocols bgp group OVERLAY cluster 10.0.10.11set protocols bgp group OVERLAY neighbor 10.0.10.21set protocols bgp group OVERLAY neighbor 10.0.10.22set protocols bgp group OVERLAY neighbor 10.0.10.31set protocols bgp group OVERLAY neighbor 10.0.10.32set protocols bgp group SPINE-PEER type internalset protocols bgp group SPINE-PEER local-address 10.0.10.11set protocols bgp group SPINE-PEER family evpn signalingset protocols bgp group SPINE-PEER neighbor 10.0.10.12| Command | Purpose |
|---|---|
type internal / family evpn signaling | iBGP session that carries EVPN NLRI instead of IPv4 unicast |
cluster 10.0.10.11 | Marks a spine as a route reflector for that cluster |
show bgp summaryEvery device should show two Established sessions.
Stretching VLAN 100 and VLAN 200 across the server leaves
To stretch the VLANs between the server leaves, we map VLAN 100 to VNI 5100 and VLAN 200 to VNI 5200. The VLAN IDs stay local to each leaf, while the VNIs identify the segments across the VXLAN fabric. Each leaf uses a unique route distinguisher to keep its EVPN routes unique, and both leaves use the same route target so they import each other's routes. We use ingress replication to send broadcast, unknown unicast, and multicast traffic to every remote VTEP participating in the VNI.
On DC1-SLEAF01:
set protocols evpn encapsulation vxlanset protocols evpn extended-vni-list allset protocols evpn multicast-mode ingress-replicationset switch-options vtep-source-interface lo0.0set switch-options route-distinguisher 10.0.10.21:1set switch-options vrf-target target:65010:1set vlans SERVERS vlan-id 100set vlans SERVERS vxlan vni 5100set vlans STORAGE vlan-id 200set vlans STORAGE vxlan vni 5200set interfaces xe-0/0/2 unit 0 family ethernet-switching interface-mode accessset interfaces xe-0/0/2 unit 0 family ethernet-switching vlan members SERVERS| Command | Purpose |
|---|---|
protocols evpn encapsulation vxlan | Selects VXLAN as the EVPN data plane |
switch-options vtep-source-interface lo0.0 | Uses the loopback as the local VTEP address |
switch-options route-distinguisher 10.0.10.21:1 | Keeps this leaf's advertisements unique in the BGP table |
switch-options vrf-target target:65010:1 | Puts both server leaves in the same L2 EVPN instance |
vlans SERVERS vxlan vni 5100 | Maps local VLAN 100 to VNI 5100, stretching it across the fabric |
A leaf sends a Type 3 route as soon as a VNI is set up. The Type 3 routes a leaf collects for a VNI form its flood list for ingress replication. On DC1-SLEAF01:
show route table bgp.evpn.0 | match "^2:|^3:"2:10.0.10.21:1::5100::00:00:5e:00:01:01/304 MAC/IP2:10.0.10.21:1::5100::00:50:79:66:68:60/304 MAC/IP2:10.0.10.21:1::5100::50:00:00:62:00:01/304 MAC/IP2:10.0.10.21:1::5200::00:00:5e:00:01:01/304 MAC/IP2:10.0.10.22:1::5100::00:00:5e:00:01:01/304 MAC/IP2:10.0.10.22:1::5100::50:00:00:62:00:01/304 MAC/IP2:10.0.10.22:1::5200::00:00:5e:00:01:01/304 MAC/IP2:10.0.10.22:1::5200::00:50:79:66:68:61/304 MAC/IP2:10.0.10.21:1::5100::00:00:5e:00:01:01::10.100.100.1/304 MAC/IP2:10.0.10.21:1::5100::50:00:00:62:00:01::10.100.100.20/304 MAC/IP2:10.0.10.21:1::5200::00:00:5e:00:01:01::10.100.200.1/304 MAC/IP2:10.0.10.22:1::5100::00:00:5e:00:01:01::10.100.100.1/304 MAC/IP2:10.0.10.22:1::5100::50:00:00:62:00:01::10.100.100.20/304 MAC/IP2:10.0.10.22:1::5200::00:00:5e:00:01:01::10.100.200.1/304 MAC/IP3:10.0.10.21:1::5100::10.0.10.21/248 IM3:10.0.10.21:1::5200::10.0.10.21/248 IM3:10.0.10.22:1::5100::10.0.10.22/248 IM3:10.0.10.22:1::5200::10.0.10.22/248 IMTwo 3: routes per VNI, one local and one from DC1-SLEAF02, build the flood list before either server sends a frame. Once DC1-SRV-A and DC1-SRV-B talk, 2: routes appear for every learned MAC. Each MAC appears twice, once as a MAC-only route and once with the host's IP appended, plus the anycast gateway MAC 00:00:5e:00:01:01 that both leaves advertise for their local IRBs.
On the vQFX, VXLAN forwarding through the virtual PFE can differ from physical hardware, so check forwarding separately from the EVPN routes above.
Anycast IRB gateways
We use anycast gateways so both server leaves can route traffic for the same VLAN at the same time. Each leaf has its own IRB address, but they share the same virtual gateway address and virtual MAC. A host sends traffic to the shared gateway and the leaf it is connected to routes it locally. This removes the need for VRRP or HSRP to keep one gateway active and the other on standby, and the host's ARP entry stays valid if it moves between leaves.
On DC1-SLEAF01:
set interfaces irb unit 100 family inet address 10.100.100.2/24 virtual-gateway-address 10.100.100.1set interfaces irb unit 100 family inet address 10.100.100.2/24 preferredset interfaces irb unit 100 virtual-gateway-accept-dataset interfaces irb unit 100 virtual-gateway-v4-mac 00:00:5e:00:01:01set interfaces irb unit 200 family inet address 10.100.200.2/24 virtual-gateway-address 10.100.200.1set interfaces irb unit 200 family inet address 10.100.200.2/24 preferredset interfaces irb unit 200 virtual-gateway-accept-dataset interfaces irb unit 200 virtual-gateway-v4-mac 00:00:5e:00:01:01set vlans SERVERS l3-interface irb.100set vlans STORAGE l3-interface irb.200set protocols evpn default-gateway do-not-advertise| Command | Purpose |
|---|---|
virtual-gateway-address 10.100.100.1 | Shared anycast gateway, set next to the leaf's own real IRB address |
... preferred | Prefers the leaf's own, lower-numbered real address as its source address |
virtual-gateway-accept-data | Lets hosts ping the gateway address directly; without it the gateway stays silent |
virtual-gateway-v4-mac 00:00:5e:00:01:01 | Sets the same virtual MAC on every leaf |
protocols evpn default-gateway do-not-advertise | Keeps the gateway local, instead of sending it out as a Type 2 route |
On DC1-SLEAF01 and DC1-SLEAF02:
show interfaces irb.100 terseDC1-SLEAF01:
Interface Admin Link Proto Local Remoteirb.100 up up inet 10.100.100.1/24 10.100.100.2/24DC1-SLEAF02:
Interface Admin Link Proto Local Remoteirb.100 up up inet 10.100.100.1/24 10.100.100.3/24Both leaves list the shared virtual gateway 10.100.100.1 first, then their own real address, .2 on DC1-SLEAF01 and .3 on DC1-SLEAF02, as the preferred source. irb.200 looks the same, one line per leaf, for 10.100.200.0/24.
Tenant VRF, L3 VNI, and pure Type 5
EVPN routes between VNIs in two ways. In asymmetric routing, the ingress leaf routes the packet directly into the destination VNI, so the egress leaf must already have that VNI and IRB set up locally, to bridge the last step. In symmetric routing, the ingress leaf routes onto a shared L3 VNI. The egress leaf then routes a second time, off that VNI and into whichever local VNI holds the destination. The L3 VNI is the only thing the two leaves need in common, so the egress leaf needs the L3 VNI but not the destination VLAN. This is why the border leaves here route tenant traffic without a local VLAN.
We select this symmetric, or "pure", Type 5 model with advertise direct-nexthop, because it lets the border leaves route the tenant without holding either VLAN locally. The resulting IP Prefix route carries a router-MAC extended community instead of a next-hop IP address. A Type 5 route describes a whole subnet, not one host, so there is no ARP entry to take a destination MAC from, and the router-MAC community carries it instead. An export policy is required, because without one Type 5 advertises every eligible route in the VRF, including subnets that should stay inside the fabric.
On both server leaves:
set routing-instances TENANT1 instance-type vrfset routing-instances TENANT1 interface irb.100set routing-instances TENANT1 interface irb.200set routing-instances TENANT1 route-distinguisher 10.0.10.21:9999set routing-instances TENANT1 vrf-target target:65010:9999set routing-instances TENANT1 vrf-table-labelset routing-instances TENANT1 protocols evpn ip-prefix-routes advertise direct-nexthopset routing-instances TENANT1 protocols evpn ip-prefix-routes vni 9999set routing-instances TENANT1 protocols evpn ip-prefix-routes export EVPN-T5-SERVERset policy-options policy-statement EVPN-T5-SERVER term servers from protocol directset policy-options policy-statement EVPN-T5-SERVER term servers from route-filter 10.100.100.0/24 exactset policy-options policy-statement EVPN-T5-SERVER term servers then acceptset policy-options policy-statement EVPN-T5-SERVER term reject-all then reject| Command | Purpose |
|---|---|
ip-prefix-routes vni 9999 | Ties Type 5 route origination to the shared L3 VNI |
ip-prefix-routes advertise direct-nexthop | Selects pure Type 5 with a router-MAC community |
ip-prefix-routes export EVPN-T5-SERVER | Limits which VRF routes become Type 5 advertisements |
vrf-target target:65010:9999 | Shares one L3 EVPN instance across all four leaves |
A matching storage term covers 10.100.200.0/24, left out above to save space. The border leaves hold the same VRF and L3 VNI, but they don't have a local VLAN or an access port. They route the tenant only through Type 5, using export policy EVPN-T5-BORDER in place of EVPN-T5-SERVER. To verify on DC1-BLEAF01:
show route table bgp.evpn.0 | match "^5:"5:10.0.10.21:9999::0::10.100.100.0::24/2485:10.0.10.21:9999::0::10.100.200.0::24/2485:10.0.10.22:9999::0::10.100.100.0::24/2485:10.0.10.22:9999::0::10.100.200.0::24/2485:10.0.10.31:9999::0::10.0.0.0::16/2485:10.0.10.31:9999::0::10.100.0.0::19/2485:10.0.10.32:9999::0::10.0.0.0::16/2485:10.0.10.32:9999::0::10.100.0.0::19/248show route table TENANT1.inet.0TENANT1.inet.0: 10 destinations, 18 routes (10 active, 0 holddown, 0 hidden)+ = Active Route, - = Last Active, * = Both10.100.100.0/24 @[EVPN/170] 1w0d 15:12:42 to 10.3.1.4 via xe-0/0/0.0 > to 10.3.2.4 via xe-0/0/1.0 #[Multipath/255] 1w0d 15:12:42, metric2 2 to 10.3.1.4 via xe-0/0/0.0 > to 10.3.2.4 via xe-0/0/1.010.100.100.0/22 *[Aggregate/130] 1w0d 15:02:51 Reject10.100.200.0/24 @[EVPN/170] 1w0d 15:12:42 > to 10.3.1.4 via xe-0/0/0.0 to 10.3.2.4 via xe-0/0/1.010.100.200.0/22 *[Aggregate/130] 1w0d 15:02:51 Reject10.0.0.0/16 *[BGP/170] 1d 18:17:29, localpref 100 AS path: 65000 I, validation-state: unverified to 10.3.100.1 via xe-0/0/2.0 > to 10.3.100.9 via xe-0/0/3.0DC1-BLEAF01 sees Type 5 routes from both server leaves for 10.100.100.0/24 and 10.100.200.0/24, resolved as EVPN routes reachable over the fabric next hops rather than the underlay's own subnets. TENANT1.inet.0 shows those two prefixes active through the L3 VNI, the two /22 aggregates it originates toward AS 65000, and the 10.0.0.0/16 enterprise aggregate learned back from the cores.
Four eBGP sessions back into the enterprise
Each border leaf peers with both enterprise cores, over four eBGP sessions that carry the tenant VRF out to AS 65000. We put the two handoff interfaces on each border leaf inside TENANT1, so the fabric underlay stays invisible to the enterprise. The cores never see 10.3.x.x addressing or a fabric OSPF LSA.
On DC1-BLEAF01:
set interfaces xe-0/0/2 unit 0 family inet address 10.3.100.2/30set interfaces xe-0/0/3 unit 0 family inet address 10.3.100.10/30set routing-instances TENANT1 interface xe-0/0/2.0set routing-instances TENANT1 interface xe-0/0/3.0set routing-instances TENANT1 routing-options aggregate route 10.100.100.0/22set routing-instances TENANT1 routing-options aggregate route 10.100.200.0/22set policy-options policy-statement ADV-TENANT term servers from protocol aggregateset policy-options policy-statement ADV-TENANT term servers from route-filter 10.100.100.0/22 exactset policy-options policy-statement ADV-TENANT term servers then acceptset policy-options policy-statement ADV-TENANT term reject-all then rejectset policy-options policy-statement IMPORT-ENT term infra from route-filter 10.0.0.0/16 exactset policy-options policy-statement IMPORT-ENT term infra then acceptset policy-options policy-statement IMPORT-ENT term reject-all then rejectset routing-instances TENANT1 protocols bgp group TO-ENT type externalset routing-instances TENANT1 protocols bgp group TO-ENT peer-as 65000set routing-instances TENANT1 protocols bgp group TO-ENT export ADV-TENANTset routing-instances TENANT1 protocols bgp group TO-ENT import IMPORT-ENTset routing-instances TENANT1 protocols bgp group TO-ENT multipathset routing-instances TENANT1 protocols bgp group TO-ENT neighbor 10.3.100.1set routing-instances TENANT1 protocols bgp group TO-ENT neighbor 10.3.100.9| Command | Purpose |
|---|---|
routing-instances TENANT1 interface xe-0/0/2.0 | Puts the handoff interface inside the tenant VRF |
routing-options aggregate route 10.100.100.0/22 | Summarizes the tenant subnet before it leaves the VRF |
bgp group TO-ENT export ADV-TENANT | Advertises only the two tenant summaries |
bgp group TO-ENT import IMPORT-ENT | Accepts only the two approved enterprise aggregates |
bgp group TO-ENT multipath | Installs both enterprise next hops for each accepted prefix |
All-active ESI-LAG for the bonded host
DC1-SRV-MH connects to both server leaves over one LAG. We give the two leaves the same LACP system ID, so they look like one LACP partner to the host, and the same Ethernet Segment Identifier, so EVPN treats them as fronting the same segment. All-active means both leaves forward traffic for the segment at the same time. Type 4 routes let the two leaves find each other on the ESI and elect a designated forwarder, so the host does not get duplicate BUM traffic. Type 1 routes do two jobs. Per EVI, they enable aliasing: a remote leaf can send traffic to either leaf, even if it learned the host MAC from only one of them. Per ES, they let routes be withdrawn quickly if a leaf loses the segment.
Identical on both server leaves:
set chassis aggregated-devices ethernet device-count 1set interfaces xe-0/0/3 ether-options 802.3ad ae0set interfaces ae0 esi 00:11:22:33:44:55:66:77:88:99set interfaces ae0 esi all-activeset interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:11set interfaces ae0 unit 0 family ethernet-switching interface-mode accessset interfaces ae0 unit 0 family ethernet-switching vlan members SERVERS| Command | Purpose |
|---|---|
interfaces ae0 esi 00:11:22:... | Same Ethernet Segment Identifier on both leaves; this binds the segment |
interfaces ae0 esi all-active | Both leaves forward traffic for the segment at the same time |
aggregated-ether-options lacp system-id ... | Same LACP system ID, so the host sees one logical partner |
The host-side bond needs real 802.3ad mode. A static bundle does not generate LACP state, and the leaves' ESI logic depends on that state.:
modprobe bonding max_bonds=0ip link add bond0 type bond mode 802.3ad miimon 100 lacp_rate fast xmit_hash_policy layer3+4ip link set eth1 master bond0ip link set eth2 master bond0ip link set bond0 upip addr add 10.100.100.20/24 dev bond0Check the host first, with cat /proc/net/bonding/bond0. The leaves cannot show a working LACP partner until the host is running 802.3ad. Then, on DC1-SLEAF01:
show evpn instance extensiveNumber of ethernet segments: 3 ESI: 00:11:22:33:44:55:66:77:88:99 Status: Resolved by IFL ae0.0 Local interface: ae0.0, Status: Up/Forwarding Number of remote PEs connected: 1 Remote PE MAC label Aliasing label Mode 10.0.10.22 5100 0 all-active DF Election Algorithm: MOD based Designated forwarder: 10.0.10.21 Backup forwarder: 10.0.10.22 Last designated forwarder update: Aug 27 21:15:21show route table bgp.evpn.0 | match "^1:|^4:"1:10.0.10.21:0::112233445566778899::FFFF:FFFF/192 AD/ESI1:10.0.10.21:1::112233445566778899::0/192 AD/EVI1:10.0.10.22:0::112233445566778899::FFFF:FFFF/192 AD/ESI1:10.0.10.22:1::112233445566778899::0/192 AD/EVI4:10.0.10.21:0::112233445566778899:10.0.10.21/296 ES4:10.0.10.22:0::112233445566778899:10.0.10.22/296 ESDC1-SLEAF01 lists ESI 00:11:22:33:44:55:66:77:88:99 with DC1-SLEAF02 (10.0.10.22) as its one remote PE in all-active mode, itself elected Designated Forwarder and DC1-SLEAF02 as backup. A Type 4 (ES) route and a Type 1 (AD/ESI, AD/EVI) route exist from both leaves. With only two leaves on the segment, aliasing cannot be watched directly. The Type 1 AD/EVI routes above confirm it is enabled. Seeing the load-balance in practice needs a third leaf with its own local VLAN 100 attachment.
Key points:
- The underlay and the overlay are separate networks. OSPF only makes loopbacks reachable, while EVPN carries tenant reachability on top of that. Neither should run on
interface allonce IRB and VTEP interfaces exist. - The spines do not terminate any tunnel. They don't have any VLAN, VNI, or VTEP configuration. They only reflect EVPN routes.
- Type 3 and Type 2 route timing. Type 3 builds the flood list for a VNI, so it appears as soon as the VNI is configured, before any host sends traffic. Type 2 carries a learned MAC address, so it only appears after a host sends its first frame.
- Anycast IRB is all-active. Every leaf routes locally, using the same virtual address and MAC. A host's gateway stays the same if the host moves to a different leaf.
- Symmetric routing at the border leaf. Pure Type 5 routes twice, through a shared L3 VNI that has to match between leaves. That's why a border leaf doesn't need a local VLAN. It needs an explicit export policy, or it advertises every route in the VRF.
- Where the L3 VNI gets used. The border leaves route tenant traffic through L3 VNI 9999. Two hosts routed and bridged between server leaves that both hold both IRBs never touch the L3 VNI.
- The ESI and the LACP system ID bind a segment. A static host-side bundle does not work here, because it never builds LACP or partner state.
- VXLAN forwarding needs its own check on the vQFX. Correct EVPN routes in
bgp.evpn.0say nothing about VXLAN forwarding through the virtual PFE, which needs a separate check.
