Etherchannels and LAGs

Last Updated: [last-modified] (UTC)

Introduction

An EtherChannel (Cisco) or Link Aggregation Group (LAG, other vendors) is used to bundle several physical links into a single logical link. This is commonly used between switches, and from large physical hosts to a switch.

This article approaches this topic from the Cisco perspective unless otherwise noted.

Trunk Link

Sometimes an EtherChannel or LAG is incorrectly referred to as a ‘trunk’. A trunk is a method of carrying multiple VLANs between devices (often switches), which can make use of an EtherChannel or LAG. However, keep in mind that although they may be complementary, they are not the same thing.

There are two main reasons for bundling links like this; The first of which is increased bandwidth. If you were to bundle four 1Gbps links into a single EtherChannel, you will now have a single virtual link that has up to 4Gbps of bandwidth available.

Increased bandwidth is especially useful between switches. Think of a case where you have two switches with 1Gbps ports. That means that you will have 1Gbps of bandwidth down to your hosts, which means that you may require more than 1Gbps of bandwidth between the switches. An EtherChannel work very well in this scenario.

The second reason for etherchannels is for resiliency. If you have four links in an EtherChannel and one is lost, the other three will keep on working (although at a reduced capacity). This can be especially useful in cases of long cable runs that can be dug up or damaged in some way.

So why bundle several links together into a virtual interface, and not just plug in a few extra links between switches? If you connect two or more links without bundling them together, one of two things will happen.

The first scenario is that spanning-tree will block all but one link. The other links can still be used for resiliency, but there will be no bandwidth gains.

The second scenario is that spanning-tree is not running, and the additional links create a loop in the network. This could result in a broadcast storm, which would be catastrophic to the network.

Etherchannels can be configured as layer-2 links (which are the most common) or layer-3 links. The difference between the two is that the Layer-3 link has an IP address, while the layer-2 link does not. Depending on the model of switch, additional licensing may be required to use an L3 EtherChannel.

Etherchannels are used with physical switches, virtual switches, and other network devices. Examples of etherchannels with virtual switching is connecting a hypervisor host to the network or connecting a NetScaler.

Layer-3 capable devices such as routers and ASA firewalls can also be configured with an EtherChannel. Even with a layer-3 device, L2 etherchannels are still quite common.

One very important factor to remember is that etherchannels are meant to be connected to a single switch. An EtherChannel by default cannot be spread across multiple switch chassis.

There are two exceptions to this rule. The first is when using a switch stack. This is allowed, as the multiple switch chassis behave as a single switch.

The second is when using a multi-chassis EtherChannel technology, such as vPC (virtual port channel) on the Nexus platform (other vendors have equivalents, such as Force 10’s VLT), or other TRILL-based technology. Both of these exceptions are outside the scope of this article.

EtherChannel properties need to match between the two endpoints. This can be configured in two basic ways; Statically, where the EtherChannel is just ‘turned on’, or dynamically, where the two endpoints negotiate settings between themselves.

Dynamic negotiation is the preferred method in most cases. There are two protocols that can be used for dynamic negotiation. The first is Cisco’s proprietary protocol, called PAgP (Port Aggregation Protocol). PAgP is not as common these days, mainly due to the second protocol, LACP (Link Aggregation Control Protocol), which is an IEEE industry standard and allows negotiation between devices of different vendors.

LACP vs PAgP

LACP is more common, as it can be used across vendors. PAgP is fairly rare in modern networks.
For this reason, this article will only discuss LACP in regard to dynamic etherchannel negotiation.

Load Distribution

Traffic across an EtherChannel is distributed across its physical links. This happens in both static and dynamic etherchannels.

It is often assumed that the traffic is spread evenly, but this is often not the case, as the switch (or other devices, but Cisco switches are considered here) needs to select a suitable link for each traffic flow. Using the wrong method may result in a sub-optimal spread of traffic.

TCP Packets

A major advantage of ‘pinning’ a traffic flow to a single link is that it limits the possibility of TCP packets being received out of order, and needing to be reassembled.

MPLS

MPLS tags can also be considered on high-end switches (such as the Catalyst 6500), but is beyond the scope of this article.

The switch breaks down the traffic passing over the etherchannel into flows. Flows can be based on several criteria, and are assigned a hash value. A simple flow hashing method would consider source and/or destination MAC, and create a logical ‘flow’ for each combination. More advanced criteria could include MAC at layer-2, IP address at layer-3, and port number at layer-4. Generally combinations of these can be used, as well as considering a combination of source and/or destination. The more values that go into the hashing algorithm, the more logical flows can be created.

It is important to remember that the model of switch plays an important part in this. Entry level switches will likely only support MAC address bases hashing, mid-tier switches will be able to consider MAC address and IP addresses, and high-level switches can consider MAC, IP address, and port number.

A common default hashing algorithm is called src-dst-ip. If this is set, the switch will XOR the source IP and the destination IP to get a hash value.

CatOS Switches

CatOS is a little different to IOS, and is outside the scope of this article
See http://www.cisco.com/c/en/us/support/docs/lan-switching/etherchannel/12023-4.html#cat6k for more information

There is an important reason for selecting the correct load-balancing type. Think of an example where an etherchannel is primarily used by a server on one VLAN which communicates to a server on another VLAN via a router. In this case, the MAC of the router will be seen, rather than the server itself. If the load balancing method were set to src-mac or dst-mac, nearly all traffic will be classified into a single flow. This in turn will be allocated to a single link in the etherchannel, making one link heavily used, and the rest under-utilized. Basing the load-balancing on IP address also won’t be efficient, as the same two IP’s (source and destination) are used for all traffic. Perhaps basing the load-balancing on IP and port number would be better in this case.

The typical Cisco etherchannel algorithm will generate a hash value of 0 – 7 for each flow. These values then map to to an interface in the etherchannel. These values are based around the maximum number of active links in an etherchannel.

Nexus Etherchannels

From NXOS 5.1, the Nexus switches will allow 16 active links, instead of 8 active and 8 passive. The method of distributing traffic across the links remains very similar to the methods discussed in this article.
For more information, see: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_cli/if_portchannel.html#56785

In the case of an etherchannel with 8 active links, each link will have one of these hash values mapped to it. In other cases where there are less than eight links in the etherchannel, a single link may have more than one hash value assigned to it. For example, in an etherchannel with four links, each link will have two hash values assigned.

The way hashes are distributed across links (or ports) is shown in the Hash Distribution table below.

Hash Distribution

Number of Links Used	Port-1	Port-2	Port-3	Port-4	Port-5	Port-6	Port-7	Port-8
1	0-7
2	0-3	4-7
3	0, 1, 2	3, 4, 5	6, 7
4	0, 1	2, 3	4, 5	6, 7
5	0, 1	2, 3	4, 5	6	7
6	0, 1	2, 3	4	5	6	7
7	0, 1	2	3	4	7	6	7
8	0	1	2	3	8	5	6	7

The result of distributing hashes like this means that some links can be more heavily utilised than others. Consider an etherchannel with six links assigned. This will result in two of the links having two hashes assigned, while the other four links have one hash assigned. The ideal is to have the same number of hashes assigned to each link in the etherchannel.

The Traffic Distribution table below shows how the traffic is spread across the links, depending on how many links are in the etherchannel. Keep in mind that this assumes that traffic flows are assigned to different hashes evenly; This is where it is important to select the correct hashing algorith,.

Traffic Distribution

Number of Links Used	Port-1	Port-2	Port-3	Port-4	Port-5	Port-6	Port-7	Port-8
1	100%
2	50%	50%
3	37.5%	37.5%	25%
4	25%	25%	25%	25%
5	25%	25%	25%	12.5%	12.5%
6	25%	25%	12.5%	12.5%	12.5%	12.5%
7	25%	12.5%	12.5%	12.5%	12.5%	12.5%	12.5%
8	12.5%	12.5%	12.5%	12.5%	12.5%	12.5%	12.5%	12.5%

Additional articles in the Etherchannel series

[maxbutton id=”4″ text=”Configuration” url=”https://networkdirection.net/Etherchannel+Configuration”][maxbutton id=”4″ text=”LACP” url=”https://networkdirection.net/Etherchannels+with+LACP”][maxbutton id=”4″ text=”Design” url=”https://networkdirection.net/Etherchannel+Design”]

Best Practices

When configuring an etherchannel, it is best to make sure that all interfaces have the same settings before adding them to the port-channel. This includes using the same link-speeds, duplex settings, allowed VLANs (if it is layer-2), native VLAN, and so on. If there is a mismatch, the etherchannel may not form at all, or it may form and have unexpected issues later. In addition to this, none of the ports in the channel should be manually shut down.

In terms of features, it is recommended not to use a member of an etherchannel as a SPAN or ERSPAN port. If 802.1X is going to be used, check if there are any IOS incompatibilities on the particular switch platform thats going to be used.

It is important to select the best hashing mode. Firstly, see if there’s any recommendations from the vendor that the switch is connecting to; They may require a particular hashing method. Additionally, try to understand the traffic flow. As discussed previously, there are factors that can make certain methods sub-optimal.

Consider the number of links in the channel. It is recommended that the number of links has a base of 2. For example, 2¹ (2 links), 2² (4 links), 2³ (8 links), and so on. As seen in the table in the load distribution section, This will cause the hash values to be better distributed across the links in the channel.