In the first EVPN lab we built a two-spine, two-leaf fabric, made the leaf loopbacks reachable with OSPF, ran iBGP EVPN over the top, and stretched VLAN 100 across it as VNI 5100. Two hosts in the same subnet reached each other over a VXLAN tunnel.
However routing between two subnets needs a gateway. We can do it in two different ways. Centrally-routed bridging puts every IRB on one device or one pair. The leaves stay Layer 2 gateways and hand anything that needs routing up to the middle. That makes the path longer than it needs to be. Two hosts on the same leaf in different subnets send their traffic up to the gateway and back down again. The gateway holds every subnet in the tenant, so it grows as the tenant grows. Edge-routed bridging puts the gateway on the leaves. Each leaf routes for the subnets it has hosts in, so the first switch a packet reaches is the one that routes it. The trade is that every leaf needs gateway configuration and the table space for it. In this lab we will try this approach.
Also the L2 fabric has a second weakness. Host1 has one link to one leaf, so an upgrade on that leaf takes Host1 down with it. The fix is two links to two leaves, bundled, and here too there are two ways to do it. The traditional way is to make the two switches into a single logical switch: MC-LAG, a stack, or a virtual chassis. The host sees one LAG partner because there is one device. The second way is an ESI-LAG. The leaves stay independent switches. Each one advertises that it fronts the same Ethernet segment, and the rest of the fabric learns from BGP that there are two ways in. We will try this approach in this lab.
Overview
We will perform the following tasks:
- Give the leaves anycast IRB gateways that share one virtual IP and one virtual MAC.
- Put the IRBs in a tenant VRF with an L3 VNI so the fabric routes symmetrically with Type 5 routes.
- Route between subnet A and subnet B across the fabric.
- Multihome a host to both leaves with an ESI-LAG and verify the designated forwarder and the Type 1 and Type 4 routes.
Topology
Starting point
We start from the committed end state of the L2 lab.
On Host2:
ifconfig eth1 10.200.0.10 netmask 255.255.255.0 uproute add -net 10.100.0.0 netmask 255.255.255.0 gw 10.200.0.1 dev eth1On Host1:
route add -net 10.200.0.0 netmask 255.255.255.0 gw 10.100.0.1 dev eth1I had to add specific routes instead of a default route because I am using Docker images and they use eth0 for management, and a default route through the tenant gateway would mess it up.
H-mh needs two links, one to each leaf. Its eth1 goes to Leaf1 xe-0/0/3 and its eth2 to Leaf2 xe-0/0/3.
Anycast IRB gateways
An IRB is a logical interface with an IP address that sits inside a VLAN and routes for it. When a host sends to an address outside its own subnet, it addresses the frame to the IRB's MAC, and the leaf routes the inner packet instead of bridging it.
The question is which leaf gets that address. If Leaf1 owns the gateway for subnet A, a host on Leaf2 crosses the fabric to reach its own default gateway, and a host that moves to another leaf needs a new one. So every leaf serving a subnet gets the same virtual IP and the same virtual MAC for it. The host sees one gateway wherever it is, and the leaf it is attached to answers.
This is like VRRP but not like VRRP at the same time. In VRRP one router is active and the others wait. Here every leaf is active, each routing for its own local hosts.
Each leaf still needs its own real address on the IRB, which is what the fabric uses to reach that specific leaf. The virtual address is used by hosts.
Leaf1
Leaf1 serves subnet A only.
set protocols evpn default-gateway no-gateway-communityset interfaces irb unit 100 family inet address 10.100.0.2/24 virtual-gateway-address 10.100.0.1set interfaces irb unit 100 virtual-gateway-v4-mac 00:00:5e:00:01:01set interfaces irb unit 100 virtual-gateway-accept-dataset interfaces irb unit 100 no-auto-virtual-gateway-esiset vlans v100 l3-interface irb.100Leaf2
Leaf2 serves subnet A for the multihomed host and subnet B for Host2, so it gets two IRBs.
set protocols evpn default-gateway no-gateway-communityset interfaces irb unit 100 family inet address 10.100.0.3/24 virtual-gateway-address 10.100.0.1set interfaces irb unit 100 virtual-gateway-v4-mac 00:00:5e:00:01:01set interfaces irb unit 100 virtual-gateway-accept-dataset interfaces irb unit 100 no-auto-virtual-gateway-esiset vlans v100 l3-interface irb.100set vlans v200 vlan-id 200set vlans v200 vxlan vni 5200set interfaces irb unit 200 family inet address 10.200.0.3/24 virtual-gateway-address 10.200.0.1set interfaces irb unit 200 virtual-gateway-v4-mac 00:00:5e:00:01:02set interfaces irb unit 200 virtual-gateway-accept-dataset interfaces irb unit 200 no-auto-virtual-gateway-esiset vlans v200 l3-interface irb.200Host2's access port moves to the new VLAN:
delete interfaces xe-0/0/2 unit 0 family ethernet-switching vlan members v100set interfaces xe-0/0/2 unit 0 family ethernet-switching vlan members v200virtual-gateway-accept-data lets the virtual address answer a ping, which it does not do by default. no-auto-virtual-gateway-esi turns off an automatically generated ESI, so the only ESI in the fabric is the one we configure.
Verification
From Host1:
ping -c 3 10.100.0.1The gateway should answer. Do the same from Host2 to 10.200.0.1.
Symmetric routing with a tenant VRF
The leaves can route now, but we need to get a packet from a subnet on one leaf to a subnet on another. There are two ways and it depends on how the fabric scales which one you choose.
In the asymmetric model every leaf holds an IRB and an L2 VNI for every subnet in the tenant. The ingress leaf routes the packet into the destination subnet's VNI and bridges it across, so only the ingress leaf routes. Leaf1 would need VLAN 200 and VNI 5200 even though no host in subnet B is anywhere near it. With a few subnets that is fine but when you have 100 subnets that's going to become a nuisance quick.
In the symmetric model each leaf holds only the VNIs of its own local subnets, plus one shared transit VNI called the L3 VNI. The ingress leaf routes the packet into the L3 VNI, tunnels it to the egress leaf, and the egress leaf routes it out into the destination subnet. It's called symmetric because both leaves route at the same time.
The subnets travel between leaves as Type 5 routes. In the previous lab we only carried MAC addresses Type 2 for a host MAC, Type 3 for VTEP membership. A Type 5 route carries an IP prefix with no MAC, and the destination does not have to be a host the fabric has heard from.
Type 5 routes live in a tenant VRF rather than the switch's main table, which gives the tenant its own routing table on top of the per-tenant bridging it already has.
Leaf1
set routing-instances TENANT1 instance-type vrfset routing-instances TENANT1 interface irb.100set routing-instances TENANT1 route-distinguisher 10.0.0.11:9999set routing-instances TENANT1 vrf-target target:65000:9999set routing-instances TENANT1 protocols evpn ip-prefix-routes advertise direct-nexthopset routing-instances TENANT1 protocols evpn ip-prefix-routes encapsulation vxlanset routing-instances TENANT1 protocols evpn ip-prefix-routes vni 9999Leaf2
set routing-instances TENANT1 instance-type vrfset routing-instances TENANT1 interface irb.100set routing-instances TENANT1 interface irb.200set routing-instances TENANT1 route-distinguisher 10.0.0.12:9999set routing-instances TENANT1 vrf-target target:65000:9999set routing-instances TENANT1 protocols evpn ip-prefix-routes advertise direct-nexthopset routing-instances TENANT1 protocols evpn ip-prefix-routes encapsulation vxlanset routing-instances TENANT1 protocols evpn ip-prefix-routes vni 9999Verification
On Leaf1:
show route table TENANT1.inet.0There should be a route to 10.200.0.0/24 learned from Leaf2, next to Leaf1's own 10.100.0.0/24. It resolves over the L3 VNI and Leaf1 has no VLAN 200 and no VNI 5200 anywhere in its configuration. Leaf1 can route into a subnet it holds no bridge domain for.
The /32 entries in the same table are the hosts themselves, learned from the Type 2 routes that carry an IP alongside the MAC.
Routing between the subnets
From Host1:
ping -c 5 10.200.0.10Host1 sends the packet to its local gateway on Leaf1 because Host2 is off its subnet. Leaf1 routes it in the tenant VRF, finds the Type 5 route from Leaf2, and encapsulates it into VNI 9999 with Leaf2's VTEP as the destination. Leaf2 decapsulates, sees VNI 9999, routes the inner packet in its own copy of TENANT1, and sends it out into VLAN 200 to Host2.
Multihoming with an ESI-LAG
Host1 has one link to one leaf. If Leaf1 reboots, Host1 goes offline. The fix is to attach a host to two leaves at once, which means the two leaves have to convince the host they are one device, and convince each other they are looking at the same host.
The two leaves share a LACP system ID, so both links look like they terminate on one partner and the host can bundle them.
On the fabric side each leaf learns the host's MAC on its own port, and without anything else every other leaf would see the same MAC advertised from two places and treat it as a MAC move. So both leaves put the same Ethernet Segment Identifier on the bundle. An ESI names a physical segment, and two leaves advertising the same ESI are telling the fabric they are two doors into the same place.
We get two more route types appear with ESI.
A Type 4 route is the Ethernet Segment route. Each leaf advertises the ESI it holds, which is how the leaves sharing a segment find each other, and they use it to elect a designated forwarder. Only the designated forwarder sends BUM traffic down onto the segment. Without that election a broadcast would reach the host twice, once from each leaf.
A Type 1 route is Ethernet auto-discovery and it has two jobs. Per EVI it enables aliasing: a remote leaf that only learned the host's MAC from one of the two leaves still knows both can reach the segment, so it can load-balance to both. Per ES it enables mass withdraw: a leaf that loses the segment withdraws one route instead of one per MAC.
A split-horizon rule sits underneath. A leaf that receives BUM traffic from the fabric does not send it back out onto the segment it came from, which would return the frame to the host that sent it.
Leaf1 and Leaf2
Same configuration on both leaves. The ESI, the LACP system ID and the admin key have to match.
set chassis aggregated-devices ethernet device-count 1set interfaces xe-0/0/3 ether-options 802.3ad ae0set interfaces ae0 esi 00:11:22:33:44:55:66:77:88:99set interfaces ae0 esi all-activeset interfaces ae0 aggregated-ether-options lacp activeset interfaces ae0 aggregated-ether-options lacp system-id 44:55:66:77:88:99set interfaces ae0 aggregated-ether-options lacp admin-key 1set interfaces ae0 unit 0 family ethernet-switching interface-mode accessset interfaces ae0 unit 0 family ethernet-switching vlan members v100The host side
H-mh runs an 802.3ad bond over its two interfaces:
modprobe bonding max_bonds=0ip link add bond0 type bond mode 802.3ad miimon 100ip link set eth1 downip link set eth2 downip link set eth1 master bond0ip link set eth2 master bond0ip link set eth1 upip link set eth2 upip link set bond0 upip addr add 10.100.0.30/24 dev bond0ip route add 10.200.0.0/24 via 10.100.0.1 dev bond0Verification
On Leaf1:
show evpn instance default-switch esi 00:11:22:33:44:55:66:77:88:99 designated-forwarderOne of the two leaves is the designated forwarder for the segment.
The multihoming route types
On Leaf1:
show route table bgp.evpn.0 | match "^1:|^4:"There should be Type 4 routes from both leaves for the ESI, which drove the election above, and Type 1 routes for aliasing and mass withdraw.
Reaching the multihomed host
From Host2:
ping -c 5 10.100.0.30Host2 is in subnet B on Leaf2, H-mh is in subnet A behind a segment both leaves front, so the packet is routed in the tenant VRF and delivered through the ESI-LAG.
Key points:
- An IRB is the Layer 3 gateway. It decapsulates, routes the inner packet, and re-encapsulates into the destination VNI.
- The anycast gateway is one virtual IP and one virtual MAC on every leaf serving a subnet. Every leaf is active at once, unlike VRRP, so a host keeps its gateway wherever it attaches.
- Asymmetric routing needs every VNI on every leaf. Symmetric routing doesn't. A tenant VRF, a shared L3 VNI and Type 5 routes let a leaf hold only its local bridge domains and still route to everything else.
- A Type 5 route carries an IP prefix, and lives in the tenant VRF. It is how subnets move between leaves, and how external or summary routes get into the fabric.
- An ESI names a physical segment. Two leaves advertising the same ESI, with the same LACP system ID and all-active, are two doors into the same place.
- Type 4 elects the designated forwarder so BUM traffic reaches the segment once. Type 1 enables aliasing and mass withdraw, so remote leaves load-balance to both and a failure withdraws one route instead of thousands.
- Unique RD, matching RT, at both layers. It applies to the L3 VRF exactly as it applies to the bridge domain.
