Control Plane learning with BGP and EVPN is one of the newer enhancements to VxLAN. Gone are the days where you need to rely on flooding.

There are also extra features that this brings, at least on the Nexus platform. This adds Integrated Routing and Bridging (IRB) which lets the switches route locally, rather than needing an external router. Anycast gateway allows hosts to connect to any switch and still use the same default gateway.

ARP suppression allows a switch to respond to ARP requests locally, further reducing flooding. And finally, head-end replication (AKA ingress replication) adds the option of simpler configuration for BUM traffic handling.

Before proceeding with the labs in this article, I recommend that you look at the Bridging Configuration article. This covers the core VxLAN concepts, which I’m going to gloss over here.

As with the simpler flood and learn lab, we’re going to use a simple two-switch topology. Yes, spine/leaf may be better in the real world, but this is simpler for learning VxLAN.

The routed link between the switches is the underlay. OSPF is used as the underlay routing protocol.

We’re using IPv4, but IPv6 would follow the same principles.

This time, we have two tenants. One tenant has a LAN segment (VNI) that spans the two switches. The other has two VNI’s, which requires routing between. Some of the address space overlaps, so we will see how BGP can keep the tenants separate.

We use MP-BGP in the overlay. Both switches are in AS 65535. Integrated Routing and Bridging (IRB) is used to route between the VNI’s. This also means that we’ll use anycast gateway on both switches.

BUM traffic will be handled by Head End Replication. This is also known as Ingress Replication. This is not as efficient as multicast, but is very simple to configure. Also, we saw multicast used in the flood and learn lab, so we’re trying something different here.

Now that we’re not using multicast, we only need a single loopback interface. This is only used to give the NVE interface an IP, and the BGP source.

The underlay configuration is nearly the same as in the previous lab, so I won’t get into too much detail.

The big difference is that there is only a single loopback interface.

As a bonus tip, if you want to use vPC, add a secondary IP to the loopback interface. This IP needs to be the same on the pair of vPC switches.

We won’t be using vPC in this lab, but this is a starting point if you need it.

We’ll start by enabling features. The two new features here are interface-vlan and nv overlay evpn.

Interface vlan is used to create a virtual interface based on a VLAN. We use this to create the anycast gateway IP for the VNI, and to tie VNI’s to a tenant’s VRF. Don’t worry, it’ll make more sense soon.

Overlay evpn adds the EVPN address family.

There is a vMAC address on each switch for the Anycast Gateway. This is the same on each switch. Making this the same on each means that any switch can respond as the default gateway.

As shown below, some versions of NXOS need a routing template to be set. This is used to change the way memory is partitioned for routes. This needs to be set in 7.0(3)I5(1) and earlier. Newer versions take care of this transparently.

The NVE interface is the VTEP. There is only one of these per switch. It uses the loopback interface to get its IP address.

We set BGP as the host-reachability protocol, which enables BGP control plane learning. If this isn’t enabled, then we’re using flood and learn.

The BGP instance is configured mostly as normal. Remember that this is iBGP, so in the real world you need to make sure that you have a full mesh or route reflectors.

We enable the L2VPN EVPN address family, which lets MP-BGP carry MAC addresses.

Extended communities are enabled. This is to support carrying route-target information.

It’s time to add the first tenant. The configuration here is the same on both switches.

Firstly, we need to create a VRF. This is associated with VNI 900001, making this an L3VNI. The L3VNI defines the tenant within the fabric, and contains L3 routes for the tenant.

A Route Distinguisher is used to keep the VRF unique in the MP-BGP database. Setting the value automatically is the simplest option.

Route Targets are also assigned automatically. Automatic assignment is recommended unless you have a mixed-vendor environment.

The L3VNI needs an SVI to be created on each switch. SVI’s are based on VLAN, so we need to associate VNI 900001 with VLAN 101 first.

The SVI then needs to be created and associated with the tenant’s VRF. The SVI and VRF represent the tenant’s routing boundary.

The ip forward command is added to the SVI. This command enables routing. Technically, it enables the switch to take the decapsulated VxLAN packet, and forward it to the CPU or Supervisor for handling.

In BGP, we need to add the Tenant’s VRF. Inside this, we use the advertise l2vpn evpn command. This enables advertising EVPN routes (MAC addresses) within the tenant.

Now it’s time for the L2VNI’s. These are the more traditional LAN segments.

We start with binding VNI 5000 to VLAN 1000. VNI 5000 then needs to be added to the VTEP. This is also where we enable ARP Suppression, and configure ingress replication. Finally, associate the VNI with the tenant’s L3VNI/VRF to enable IRB.

Each VNI that needs to be routable needs to have an SVI in the tenant’s VRF. This SVI is given the same IP address on each switch, which enables Anycast Gateway.

Now we need to enable sharing EVPN routes (MAC addresses). This looks a lot like normal BGP configuration.

In EVPN configuration, each L2VNI needs to have an RD and RT’s assigned. This is because they use a MAC-VRF. In the MP-BGP database, L3 routes and L2 MAC addresses are in separate VRF’s. These values are still set to auto in our case, but are different to the L3VNI’s RD’s and RT’s in MP-BGP.

I know that this might sound a bit confusing. Just remember that you need to advertise L3 information into BGP, as well as separate L2 information. Even though all this is part of the same tenant, L2 and L3 addresses are different, and are treated like they’re different address families.

The host ports are configured as normal access ports. This example is only on Switch-1.

Now let’s add another VNI for this tenant. This is basically the same process that we just went through.

And finally the host port on Switch-2.

Now to repeat the process to add the second tenant. This follows the same basic process:

1. Create VRF
2. Create L3VNI
3. Create L2VNI and anycast gateway
4. Add L2VNI to VTEP
5. Add L2VNI to EVPN

To verify, start by checking that BGP neighbours are forming. This is almost identical to regular BGP.

Remember to check the State/PfxRcd field.

Now, lets look at the NVE peers. This is where we can see the remote VTEPs that have been discovered.

Notice that the LearnType field shows CP, for control plane learning.

[rtbs name=”vxlan_evpn-nve”]

Now, see the VNI’s that are associated with the local VTEP.

Look at the Type column. This tells us if this is a layer-2 or layer-3 VNI. If it’s an L2VNI, the associated VLAN comes next. If it’s an L3VNI, the VRF comes next.

L2VNI’s may have an SA flag. This indicates that ARP suppression is enabled.

The Multicast-group column would normally contain the mcast group used for BUM traffic. This is set to UnicastBGP in our case, as we’re using ingress replication.

[rtbs name=”vxlan_evpn-vni”]

show vxlan is a simple command to show VLAN to VxLAN (VN-Segment) bindings.

Now, have a look at layer-2 routing information. As MAC addresses are learned, they are added here.

The first entry in the example below is learned through BGP. The second has been learned on the local switch, on interface eth 1/1.

[rtbs name=”vxlan_evpn-routes”]

This next command shows the IP to MAC relationship.

This final command is the big one. This is the BGP database.

There are several entries of note here. Routes are grouped by route distinguisher. This represents the tenant’s VRF.

Each of these entries is type-2 or type-3, meaning L2VNI or L3VNI. The important parts though are the MAC and IP addresses, and the next-hop (remote VTEP).