>_nexthopreachable

JNCIP-ENT: EVPN-VXLAN Fabric and Layer 2 Gateway

Ali Bahadır Coşkun14 min read

A host must be able to keep its address and its broadcast domain regardless of which switch it is attached to, and to move between switches without renumbering. There are two ways to meet that requirement. The first is to extend the VLAN over trunks and let spanning tree keep the topology loop-free, which blocks links and limits how far the domain can grow. The second is to route between the switches and carry the VLAN across the routed network inside tunnels.

VXLAN achieves this by providing the tunnels and EVPN provides the BGP control plane that tells each switch which host is behind which tunnel endpoint.

The topology in this lab is four vQFX switches, two spines and two leaves, with one host on each leaf. The hosts are in the same subnet and reach each other across the routed core.

The fabric consists of an underlay and an overlay. The underlay is a routed IP network that makes every loopback reachable. The overlay is a BGP control plane running on top of the underlay that carries the location of each host MAC address.

Overview

DeviceUnderlay roleOverlay role
Spine1, Spine2OSPF transit between the leavesiBGP EVPN route reflectors with one cluster
Leaf1, Leaf2OSPF, loopback is the VTEP addressEVPN VTEP and Layer 2 gateway for VNI 5100
Host1, Host2noneendpoints in VLAN 100, subnet 10.100.0.0/24

We will perform the following tasks:

  • Configure the underlay with OSPF on the fabric links and loopbacks, and confirm two equal-cost paths between the leaf loopbacks.
  • Configure the overlay as iBGP with family evpn signaling, with both spines as redundant route reflectors.
  • Configure the leaves as VTEPs and Layer 2 gateways, and verify the Type 3 routes before any host traffic exists.
  • Bring up the hosts and verify the Type 2 routes that carry their MAC addresses.

Topology

Underlay

Every encapsulated frame leaves a leaf with an outer IP header whose source and destination are leaf loopbacks. The underlay's only job is to deliver those packets. It holds no tenant state, so its size depends on the number of switches, it doesn't deal with the hosts.

A loopback stays up when a physical interface goes down. A fabric link failure reroutes the underlay and the tunnel is unaffected. An address on a physical interface would take the tunnel down with the link. That's why we use the loopbacks.

We use p2p links because without interface-type p2p, OSPF treats an Ethernet link as a broadcast segment, elects a DR and a BDR, and generates a network LSA for a link with two routers on it. The election delays the adjacency for no reason. The /31 addresses doesn't waste address space on a network or broadcast address.

We will increase MTU because VXLAN adds 50 bytes of outer Ethernet, IP, UDP, and VXLAN header. A 1514-byte host frame becomes 1564 bytes in the fabric.

Spine1

junos
set interfaces xe-0/0/0 mtu 9216set interfaces xe-0/0/0 unit 0 family inet address 10.1.11.1/31set interfaces xe-0/0/1 mtu 9216set interfaces xe-0/0/1 unit 0 family inet address 10.1.21.1/31set interfaces lo0 unit 0 family inet address 10.0.0.1/32set protocols ospf area 0.0.0.0 interface xe-0/0/0.0 interface-type p2pset protocols ospf area 0.0.0.0 interface xe-0/0/1.0 interface-type p2pset protocols ospf area 0.0.0.0 interface lo0.0 passiveset routing-options router-id 10.0.0.1

Spine2

junos
set interfaces xe-0/0/0 mtu 9216set interfaces xe-0/0/0 unit 0 family inet address 10.1.12.1/31set interfaces xe-0/0/1 mtu 9216set interfaces xe-0/0/1 unit 0 family inet address 10.1.22.1/31set interfaces lo0 unit 0 family inet address 10.0.0.2/32set protocols ospf area 0.0.0.0 interface xe-0/0/0.0 interface-type p2pset protocols ospf area 0.0.0.0 interface xe-0/0/1.0 interface-type p2pset protocols ospf area 0.0.0.0 interface lo0.0 passiveset routing-options router-id 10.0.0.2

Leaf1

junos
set interfaces xe-0/0/0 mtu 9216set interfaces xe-0/0/0 unit 0 family inet address 10.1.11.0/31set interfaces xe-0/0/1 mtu 9216set interfaces xe-0/0/1 unit 0 family inet address 10.1.12.0/31set interfaces lo0 unit 0 family inet address 10.0.0.11/32set protocols ospf area 0.0.0.0 interface xe-0/0/0.0 interface-type p2pset protocols ospf area 0.0.0.0 interface xe-0/0/1.0 interface-type p2pset protocols ospf area 0.0.0.0 interface lo0.0 passiveset routing-options router-id 10.0.0.11

Leaf2

junos
set interfaces xe-0/0/0 mtu 9216set interfaces xe-0/0/0 unit 0 family inet address 10.1.21.0/31set interfaces xe-0/0/1 mtu 9216set interfaces xe-0/0/1 unit 0 family inet address 10.1.22.0/31set interfaces lo0 unit 0 family inet address 10.0.0.12/32set protocols ospf area 0.0.0.0 interface xe-0/0/0.0 interface-type p2pset protocols ospf area 0.0.0.0 interface xe-0/0/1.0 interface-type p2pset protocols ospf area 0.0.0.0 interface lo0.0 passiveset routing-options router-id 10.0.0.12

Equal-cost forwarding

OSPF computes two equal-cost paths to the remote loopback and puts both of them in the routing table. The forwarding table only takes one. If we leave it like that, one spine carries all the traffic between the leaves and the other one sits idle.

We add a forwarding-table export policy because that is the only way to get both next hops into the forwarding table. We apply it on all four devices.

junos
set policy-options policy-statement LB term 1 then load-balance per-packetset routing-options forwarding-table export LB

The keyword says per-packet but Junos hashes per flow on this platform. This was one of the weirdest things about Junos for a long time until they changed it in the later versions, but my version still uses this command. Each flow picks one path and stays on it, so the packets in a flow don't arrive out of order.

Verification

On Leaf1:

junos
show route forwarding-table destination 10.0.0.12/32 table default

The forwarding table should show two next hops for Leaf2's loopback, one through each spine.

Repeat the check on Leaf2 for 10.0.0.11/32.

Overlay

The overlay is one iBGP autonomous system whose sessions carry EVPN routes instead of IPv4 unicast. We will use the family evpn signaling for that. The sessions are loopback-to-loopback and depend on the underlay.

iBGP does not re-advertise a route learned from one iBGP peer to another, which would require a full mesh between the leaves. Instead, both spines act as route reflectors for a single client cluster. Each leaf peers with both spines, each spine receives every leaf's routes directly, and each spine reflects them to the other leaf.

The spines reflect EVPN routes without being VTEPs. They do not encapsulate or decapsulate VXLAN packets, and they don't carry VLAN, VNI, or VTEP configuration.

Spine1

junos
set routing-options autonomous-system 65000set protocols bgp group LEAVES type internalset protocols bgp group LEAVES local-address 10.0.0.1set protocols bgp group LEAVES family evpn signalingset protocols bgp group LEAVES cluster 10.0.0.100set protocols bgp group LEAVES neighbor 10.0.0.11set protocols bgp group LEAVES neighbor 10.0.0.12

Spine2

junos
set routing-options autonomous-system 65000set protocols bgp group LEAVES type internalset protocols bgp group LEAVES local-address 10.0.0.2set protocols bgp group LEAVES family evpn signalingset protocols bgp group LEAVES cluster 10.0.0.100set protocols bgp group LEAVES neighbor 10.0.0.11set protocols bgp group LEAVES neighbor 10.0.0.12

Leaf1 and Leaf2

Both leaves are clients of both spines. The configuration differs only in the local address.

On Leaf1:

junos
set routing-options autonomous-system 65000set protocols bgp group OVERLAY type internalset protocols bgp group OVERLAY local-address 10.0.0.11set protocols bgp group OVERLAY family evpn signalingset protocols bgp group OVERLAY neighbor 10.0.0.1set protocols bgp group OVERLAY neighbor 10.0.0.2

On Leaf2:

junos
set routing-options autonomous-system 65000set protocols bgp group OVERLAY type internalset protocols bgp group OVERLAY local-address 10.0.0.12set protocols bgp group OVERLAY family evpn signalingset protocols bgp group OVERLAY neighbor 10.0.0.1set protocols bgp group OVERLAY neighbor 10.0.0.2

Verification

On Leaf1:

junos
show bgp summary

Both sessions should be Established.

The table is empty at this point. No VNI is configured yet, so the leaves don't advertise anything to each other.

Layer 2 gateway on the leaves

Only the leaves are VTEPs so we will only do this configuration on the leaves.

protocols evpn connects the EVPN control plane to the VXLAN data plane. It sets the encapsulation to VXLAN, selects which VNIs are extended, and sets ingress replication for broadcast, unknown-unicast, and multicast traffic.

switch-options selects the loopback as the source address for VXLAN encapsulation, assigns a route distinguisher to the local EVPN instance, and assigns the route target used to import and export the routes.

The vlans stanza maps VLAN 100 to VNI 5100, which extends the local VLAN across the fabric.

Route distinguisher and route target

The route distinguisher makes an EVPN route globally unique, so that identical MAC advertisements from different instances or different VTEPs do not overwrite each other in the BGP table.

The route target is an extended community used by import and export policy. Matching route targets allow the two leaves' EVPN instances to import each other's routes. A route target does not create a tenant and does not identify a VNI. It controls route membership only.

Leaf1

junos
set protocols evpn encapsulation vxlanset protocols evpn extended-vni-list allset protocols evpn multicast-mode ingress-replicationset switch-options vtep-source-interface lo0.0set switch-options route-distinguisher 10.0.0.11:1set switch-options vrf-target target:65000:1set vlans v100 vlan-id 100set vlans v100 vxlan vni 5100set interfaces xe-0/0/2 unit 0 family ethernet-switching interface-mode accessset interfaces xe-0/0/2 unit 0 family ethernet-switching vlan members v100

Leaf2

Identical except for the route distinguisher.

junos
set protocols evpn encapsulation vxlanset protocols evpn extended-vni-list allset protocols evpn multicast-mode ingress-replicationset switch-options vtep-source-interface lo0.0set switch-options route-distinguisher 10.0.0.12:1set switch-options vrf-target target:65000:1set vlans v100 vlan-id 100set vlans v100 vxlan vni 5100set interfaces xe-0/0/2 unit 0 family ethernet-switching interface-mode accessset interfaces xe-0/0/2 unit 0 family ethernet-switching vlan members v100

Type 3 routes

A leaf originates a Type 3 route as soon as it has a VNI configured. The route is called IMET, and it carries the leaf's VTEP address and the tunnel type.

It says I have VNI 5100 and I want the BUM traffic for it. Every leaf collects the Type 3 routes it imports for a VNI, and that makes the flood list. When Leaf1 has to flood a frame, it sends one copy to each VTEP on that list.

On Leaf1:

junos
show route table bgp.evpn.0

An EVPN route key begins with its route type, so the Type 3 routes are the entries starting with 3:. There should be two of them for VNI 5100, one originated locally and one received from Leaf2.

The received one only arrives if the underlay carries the BGP session, the spines reflect the route, and the route target matches.

There are no 2: routes yet because no host MAC has been learned.

Hosts and MAC learning

Both hosts are in 10.100.0.0/24, so no gateway is needed.

On Host1:

text
ifconfig eth1 10.100.0.10 netmask 255.255.255.0 up

On Host2:

text
ifconfig eth1 10.100.0.20 netmask 255.255.255.0 up

ARP and flooding

From Host1:

text
ping -c 5 10.100.0.20

Host1 has no ARP entry for Host2, so the first packet is a broadcast ARP request. Leaf1 has to flood it. There is no multicast in the underlay, so it floods by ingress replication instead. It makes one copy for every VTEP on the flood list, wraps each copy in VXLAN, and sends them as ordinary unicast packets.

Leaf2 gets that packet on UDP port 4789. It reads VNI 5100, strips the outer headers, and sends the original ARP request out of its access port to Host2.

Leaf1 saw Host1's source MAC on xe-0/0/2.0, so it advertises it in a Type 2 route. Leaf2 sees Host2's MAC in the ARP reply and does the same. The spines reflect both routes to the other leaf.

After that the flooding stops. Each leaf knows which VTEP the remote MAC is behind, so it encapsulates the frame and sends it to that VTEP.

The first echo request can be lost while all of this happens.

MAC table

On Leaf1:

junos
show ethernet-switching table

There should be two MACs in VLAN v100. Host1's is learned locally on xe-0/0/2.0. Host2's is learned through EVPN and sits behind the VTEP interface instead of a physical port, because Leaf1 reaches it through the tunnel and not through a port of its own.

Type 2 routes

On Leaf1:

junos
show route table bgp.evpn.0

The table now holds 2: MAC advertisement routes as well as the 3: routes from before. Each one carries the route distinguisher of the leaf that originated it, so the local ones start with 10.0.0.11:1 and the ones from Leaf2 start with 10.0.0.12:1.

There are four of them, not two, because each host is advertised twice. One route carries the MAC on its own, ending in 50:00:00:2e:00:01/304. The other carries the same MAC with the host's IP appended, ending in 50:00:00:2e:00:01::10.100.0.10/304. The leaf learned the MAC from the frame and the IP from the ARP, and it advertises what it knows. The MAC-only route is what does the work in this lab. The MAC-plus-IP route is what later designs use for ARP suppression and for Layer 3 forwarding once IRB gateways exist.

Tunnel resolution

On Leaf1:

junos
show route table :vxlan.inet.0

The table should hold a route to Leaf2's VTEP address.

Key points:

  • The underlay and the overlay are separate networks. The underlay makes the VTEP loopbacks reachable with ECMP. The overlay carries tenant MAC reachability. Loopbacks are used as VTEP addresses.
  • VXLAN is the data plane. It encapsulates a Layer 2 frame in UDP and IP, with a 24-bit VNI identifying the broadcast domain, and sends it loopback to loopback. The core forwards it as an ordinary IP packet, and spanning tree is not used.
  • EVPN is the control plane. A Type 3 route advertises a VTEP's membership in a VNI and builds the flood list. A Type 2 route advertises a MAC, optionally with its IP. With ingress replication, the remote Type 3 must be imported before the first flooded ARP can cross the fabric.
  • The route distinguisher is unique per EVPN instance and the route target matches across instances. The RD keeps advertisements unique in the BGP table. The RT decides which instance imports them.
  • BUM traffic uses ingress replication. The ingress leaf makes one unicast copy for each remote VTEP that advertised a Type 3 route for the VNI. No multicast is required in the underlay.
  • In a bridged overlay the leaves are the VTEPs and the spines are IP transit and route reflectors. The spines carry no VLAN, VNI, or VTEP configuration.
  • Run the underlay IGP on named interfaces, not interface all. Once IRB interfaces exist, admitting them into the underlay allows a tunnel to be resolved through the interface that depends on it.