A few weeks ago I was working on a customer’s network when I found an OSPF problem. For some reason, an ASA wouldn’t peer with a Nexus switch. To make it a bit weirder, the problem only happened on the default VRF, and only with OSPFv3. On the Nexus side, I could see the ASA neighbour, but it was stuck in INIT. On the ASA side, I couldn’t see the neighbour at all.
After hours of troubleshooting, I got the TAC involved. But before I tell you what they found, let me show you the topology I was working with.
There is a pair of ASA’s, running Threat Defence. They are an active-standby failover pair.
The ASAs connect to a pair of Nexus Fabric Extenders (FEX). The FEX’s connect to Nexus 9000 series parent switches in straight through mode.
The ASA’s to the switches with vPC. The layer3 peer-router command makes dynamic routing over vPC possible. Neighbours use SVI’s for peering.
Take a look at how vPC’s work
So what does this mean? It means that if we connect a router to a layer-2 FEX port and use SVI’s, we cannot reliably run a routing protocol. The port will not prioritise control plane traffic over other types of traffic. This results in dynamic routing traffic getting dropped.
This is the only time I’ve seen a problem in a topology. Most of the time it works fine. But, the bottom line is that it’s an unsupported design.
The second option is to bypass the FEX and connect straight to the parent switch. Depending on your port requirements, this may need extra SFPs.
The third possibility is to use routed ports on the FEX. This won’t be possible in all cases, but is generally preferred to peering over SVI’s anyway. Be aware though, that not all FEX models support routed ports.
Have you had this problem? Any other ideas to solve it? Share your thoughts in the comments below.