Dynamic Routing and Fabric Extenders

Wednesday August 30, 2017 The Problem

A few weeks ago I was working on a customer’s network when I found an OSPF problem. For some reason, an ASA wouldn’t peer with a Nexus switch. To make it a bit weirder, the problem only happened on the default VRF, and only with OSPFv3. On the Nexus side, I could see the ASA neighbour, but it was stuck in INIT. On the ASA side, I couldn’t see the neighbour at all.

After hours of troubleshooting, I got the TAC involved. But before I tell you what they found, let me show you the topology I was working with.

There is a pair of ASA’s, running Threat Defence. They are an active-standby failover pair.

The ASAs connect to a pair of Nexus Fabric Extenders (FEX). The FEX’s connect to Nexus 9000 series parent switches in straight through mode.

The ASA’s to the switches with vPC. The layer3 peer-router command makes dynamic routing over vPC possible. Neighbours use SVI’s for peering.

Take a look at how vPC’s work

[maxbutton id=”4″ text=”vPC’s” url=”https://networkdirection.net/Virtual+Port+Channels”][maxbutton id=”4″ text=”Routing over vPC” url=”https://networkdirection.net/vPC+and+Routing+Protocols”]

The Cause

I had fallen victim to missing a footnote in one of Cisco’s guides. Have a look at the article Cisco Nexus 2000 Series NX-OS Fabric Extender Configuration Guide for Cisco Nexus 9000 Series Switches. Under the Guidelines and Limitations section, the article points out that when a FEX is connected to a 9000 series parent, “the queuing capability on the FEX host interface is limited.”

So what does this mean? It means that if we connect a router to a layer-2 FEX port and use SVI’s, we cannot reliably run a routing protocol. The port will not prioritise control plane traffic over other types of traffic. This results in dynamic routing traffic getting dropped.

This is the only time I’ve seen a problem in a topology. Most of the time it works fine. But, the bottom line is that it’s an unsupported design.

The Solution

There are three solutions that come to mind. The first is to run static routes. This issue does not affect static routing, as it does not use any control plane messages.

The second option is to bypass the FEX and connect straight to the parent switch. Depending on your port requirements, this may need extra SFPs.

The third possibility is to use routed ports on the FEX. This won’t be possible in all cases, but is generally preferred to peering over SVI’s anyway. Be aware though, that not all FEX models support routed ports.

Have you had this problem? Any other ideas to solve it? Share your thoughts in the comments below.