Spanning-tree has played a vital role in layer-2 networks for many years. It allows redundant loop-free topologies, and quick recovery when links fail. Unfortunately, it has a critical downside. While allowing many uplinks, loop the prevention algorithm will disable some of them. This limits the bandwidth and scalability of the network. Additionally, spanning-tree implements a single Root Bridge. This often leads to sub-optimal paths.

In the example below, one of the links is in a blocking state. The switches at the bottom need to send traffic toward the root bridge to communicate. This results in a sub-optimal path.

One way to address this problem is with etherchannels. This allows many active links between two devices. The problem is that it won’t allow uplinks to more than one device at a time.

An extension to EtherChannels is virtual port-channels. vPC’s allow uplinks to two different switches to be active at the same time. This is great for a host, such as a server. It’s not great for many switches, as it only allows connections to two upstream switches per vPC. It is not scalable for this purpose.

Cisco’s Fabricpath is scalable and allows for many active uplinks. It does this by creating a layer-2 fabric or mesh of active links between switches. It is able to make all the links active by taking layer-3 concepts, and applying them at layer-2. FabricPath uses the IS-IS routing protocol. It supports multi-area design, and is not dependent on IP.

Fabricpath is a Cisco proprietary protocol, found on some of the Nexus switch range, such as the N5K and N7K. Fabricpath is not supported on the N3K or N9K series. These switches support VxLAN based fabric. This is an alternate way to put non-blocking uplinks in place.

The classic three-tier network topology uses core, aggregation, and access layers. These use a mixture of routing (layer-3) and spanning-tree or vPC (layer-2). This segments the network into blocks. This can be an advantage, especially in networks where north-south traffic is common. In the data centre, this may not be desirable.

In the data centre, east-west traffic is more common. This is a lot of traffic is between servers residing at the same layer. Breaking the network into blocks could mean that traffic may first travel north into the core. Then it travels south into another block. These blocks may be in racks sitting side-by-side. You can see how this is inefficient.

Fabricpath uses the spine/leaf topology. Each switch is classified as a spine or leaf. As shown below, leaves connect to hosts and other devices. Spines only direct traffic within the fabric and do not have hosts connected. There is a full mesh of links between spines and leaves. Every leaf has a connection to every spine, and every connection is non-blocking.

This topology favours east-west traffic patterns. All the uplinks in the fabric are layer-2, but they behave like layer-3 ECMP paths.

Enable FabricPath by installing the Fabricpath feature-set, then enabling the feature-set.

In Fabricpath, VLANs can be in either Classical Ethernet (CE) mode or Fabricpath mode. The VLAN has the mode applied in VLAN config, as shown below.

VLAN mode is locally significant to the switch or VDC. This means that there is nothing passed from switch to switch to identify a VLAN as CE or FabricPath.

Fabricpath defines two types of ports; Edge and Core.

Edge ports connect to regular Classical Ethernet devices. These ports behave as normal CE ports. They use traditional MAC learning, and don’t encapsulate traffic with Fabricpath headers.

Edge ports may use either CE or Fabricpath VLANs. If they need to carry traffic from CE devices to the Fabricpath domain, the VLAN must be in Fabricpath mode. The port can be access or trunk.

Core ports are Fabricpath aware ports that connect to other core ports. These connections form Fabricpath adjancencies. All traffic passed between Fabricpath adjacancies are encapsulated with a Fabricpath header.

Core ports only carry VLANs in Fabricpath mode

To configure a Fabricpath port:

A Fabricpath switch can be an ingress switch, a core switch or an egress switch. Ingress and egress switches are also called Layer-2 Gateways.

An ingress switch contains an edge port and receives a Classical Ethernet frame. This is a leaf switch, as shown earlier. The switch does a lookup to determine which Fabricpath interface to forward the frame to.

A core switch contains a core port. This is a spine switch. When a core port receives a Fabricpath frame, it determines which interface to send the frame out of. To do this, it looks at the Fabricpath header on the frame. The egress port will also be a core port.

An egress switch receives a Fabricpath frame on a core port. It then determines which interface to send the frame from. The egress port is an edge port, so it strips the Fabricpath header from the frame. The frame is then sent as a CE frame. Traditional MAC learning only happens on an egress edge port. Core ports do not need MAC addresses, as they use the SID in the Fabricpath header.

The Fabricpath Control Plane disseminates reachability information. This is like a routing protocol, that builds a view of subnets in the routing table.

Each switch in the Fabricpath domain has a 12-bit switch identifier. An administrator may assign this identifier. Or, Dynamic Resource Allocation Protocol (DRAP) could assign it. The Switch ID is used frequently in the Fabricpath header.

Each switch will also build adjacencies, which are based on IS-IS adjacencies. These form when an interface connects to another Fabricpath switch.

When the adjacencies have formed, they exchange layer-2 routing information. Routes aren’t really used at layer-2, but applying IS-IS concepts here make it possible. The Unicast Layer-2 Routing Information Base (U2RIB) is then populated with the routing information.

An SPF algorithm is then run (just like IS-IS) to get the path costs. The lowest path cost is the best path. FabricPath can use as many as 16 ECMP’s (Equal Cost Multi-Paths). This is what makes Fabricpath so scalable.

Fabricpath can also handle BUM (Broadcast, Unknown unicast, Multicast) traffic. This traffic requires special handling. Without FabricPath this type of traffic would use Classical Ethernet techniques such as ARP. These techniques just aren’t used in pure Fabricpath. This is done with multi-destination trees, which is a bit like a multicast tree. A single switch is elected as the root of the tree. The switch priority, system ID and switch ID influence the election.

The CE frame contains a MAC address section in it’s header. This includes the Source Address, Destination Address, and the Type. When a frame enters a FabricPath domain, a new Fabricpath header is added to the front of the frame. The CE MAC address section is still there, but it’s now called the iMAC, or Inside MAC. The Fabricpath header added to the front is called the OMAC, or Outside MAC.

The OMAC is made upof three fields. The Outer Destination Address (ODA), the Outer Source Address (OSA), and the FTag. The Ftag contains the ethertype and TTL fields.

The ODA and OSA are the Fabricpath destination and source addresses. These are the switch ID’s within the Fabricpath domain.

The data plane encapsulates and decapsulates CE frames, and forwards of Fabricpath frames. Fabricpath uses a MAC-in-MAC style encapsulation. It prepends a 16-byte Fabricpath header to each frame. It removes the original CRC field and generates a new one.

The Fabricpath header includes a TTL field. Just like the TTL in the IP header, this prevents loops and storms. This is part of what makes it possible to have many active links. This is in contrast to Classical Ethernet, which has no built-in TTL. CE relies on external mechanisms such as spanning-tree for loop prevention.

A FabricPath frame is larger than a CE frame, due to the extra header. FabricPath switches should use jumbo frames, or have the MTU increased.