HTTPS and Fragmentation

Broken Glass

HTTPS, Fragmentation, and MTU Size

I faced a situation where a server had been migrated to a public cloud provider, and suddenly certain services were no longer working. Looks like we’ve got an MTU problem!

 

In particular, we found that accessing a particular third-party finance service over SSL was failing. We also found that we could not access their website.

On deeper inspection, we also found that we were having trouble getting to various other websites, but only the ones that used HTTPS.

This error only happened when we used our WAN connection to the provider. If we used the provider’s native internet link, everything was fine.

 

 


What Was The Issue?

I’ll get straight to the point on this one, and keep it brief. We found that the don’t fragment bit was set on SSL traffic.

It seems that the websites that were failing had a lot of data to transfer, pushing the packet size over the MTU on our WAN.

Packets need to fragment if they are over the MTU sizeThis meant that the packet needed to be fragmented, but the DF bit prevented it. So, the result was that the packet was dropped.

I haven’t been able to find anything to say that SSL/HTTPS traffic should have the DF bit set, although others report having the same issue. It seems that this is specific to webservers or browsers.

 

How is This Resolved?

The problem was clearly on the WAN. Part of the solution was an encrypted GRE tunnel over a private link.

Looking at the tunnel, it was easy to see that the MTU values were incorrect. Specifically, in this case, the MPLS MTU size.

The resolution included finding the correct size using ping and then altering the configuration to match.

 

After this, everything started working correctly. Show’s how important it is to understand how MTU affects the network.

 

ASA VPN Troubleshooting

The environment has an ASA which is the local VPN endpoint, as well as another ASA on the edge of the network

ASA VPN Troubleshooting

Yesterday, I assisted with troubleshooting ASA VPN issues. A local ASA needed to build a site-to-site (aka L2L) IPSec VPN tunnel to a non-ASA third-party. The tunnel was not coming up.

The config all appeared to be there, and the third-party said their config was in place too.

It’s time to troubleshoot. Here are the steps I followed.

 

Check if SA’s are Forming

This is always my first step when troubleshooting. There should be phase-1 SA’s and phase-2 SA’s for the ASA VPN to work.

You can find phase-1 SA’s with:

show crypto isakmp sa

And phase-2 SA’s with:

show crypto ipsec sa

 

In my case, there were no phase-1 SA’s, so there was no point looking for phase-2 SA’s.

Perhaps the ASA hasn’t seen any interesting traffic yet and hasn’t tried to bring the tunnel up. We can try to do this with packet tracer:

packet-tracer input Inside tcp 10.0.0.1 http 172.16.0.1 http

 

This just simulates some http traffic from 10.0.0.1 to 172.16.0.1. These are hosts that should be able to communicate over the tunnel.

This doesn’t work, and no SA’s formed. So, phase-1 looks like a good place to focus on.

 

 


Check IKE Proposals

The first step in troubleshooting phase-1 (IKEv2 in my case) is to confirm that there are matching proposals on both sides. The proposals include acceptable combinations of cyphers, hashes, and other crypto information.

This is easy if you control both ends of the ASA VPN tunnel. Just look at what’s configured. In my case, it’s a little harder, as a third-party manages the remote end of the tunnel.

Instead, I can find this with a debug command:

debug crypto ikev2 protocol 64

 

This will show us any errors with IKEv2 (you can substitute IKEv1 if you need to).

The ’64’ is the debugging level. This can be from 1 to 256. The higher the number, the more detail you get. Don’t go too high too quickly, as there may be too much information to search through.

 

The debug gave me this:

IKEv2-PROTO-1: (16): Received Policies:
Proposal 1: AES-CBC-256 SHA1 SHA96 DH_GROUP_1024_MODP/Group 2
Proposal 2: AES-CBC-256 SHA256 SHA256 DH_GROUP_1024_MODP/Group 2
Proposal 3: AES-CBC-128 SHA1 SHA96 DH_GROUP_1024_MODP/Group 2
Proposal 4: AES-CBC-128 SHA256 SHA256 DH_GROUP_1024_MODP/Group 2
Proposal 5: 3DES SHA1 SHA96 DH_GROUP_1024_MODP/Group 2
Proposal 6: 3DES SHA256 SHA256 DH_GROUP_1024_MODP/Group 2
IKEv2-PROTO-1: (16): Failed to find a matching policy
IKEv2-PROTO-1: (16): Expected Policies:
Proposal 1: AES-CBC-256 SHA384 SHA384 DH_GROUP_2048_MODP_256_PRIME/Group 24

 

We can see here that the remote endpoint is offering six proposals, and we are offering one.

Our proposal does not match any of theirs, so IKE will fail. There are two ways to address this:

  1. We change our proposal to match theirs
  2. They change their proposal to match ours

 

For testing, it’s simpler just to change our side. We can figure out better security algorithms later.

Once the proposal has been changed, these debug errors no longer show up. But we’re still not getting phase-1 SA’s.

Also, it looks like the ASA is not even trying to bring up the tunnel. The proposal errors were coming from the remote end.

Time to check the network path…

 

 


Check the Path

There is another ASA on the path, so it sounds like a good idea to check that it’s not interfering.

The environment has an ASA which is the local VPN endpoint, as well as another ASA on the edge of the network
The environment has an ASA which is the local VPN endpoint, as well as another ASA on the edge of the network

 

A quick check of the ACLs showed that there were ‘IP Any’ rules for the local and remote endpoints. Unfortunately, that’s not enough.

We need to add ESP and AH in there (AH is not used in all environments though). These do not use IP, they are completely separate protocols.

 

Running packet-tracer on the edge, I noticed that the wrong interface was being used for egress traffic. I won’t get into too much detail, but there was a problem with routing.

One more issue fixed, but still no phase-1 SA’s.

 

 


Further Debugs

While on the edge ASA, A packet capture revealed two things:

  1. The third-party was periodically trying to connect to us
  2. The local VPN ASA was not trying to connect to the ASA at all

 

Time for another debug on the VPN ASA:

debug crypto ikev2 platform 64

 

This time we’re looking for platform related issues. As soon as I run another packet-tracer, I get some output, revealing something interesting:

IKEv2-PLAT-5: INVALID PSH HANDLE
IKEv2-PLAT-1: Failed to request a Other VPN license. Maximum tunnel count reached and/or license limit exceeded to initiate tunnel
IKEv2-PLAT-1: asa connect start L2L failed

 

We don’t normally need to put much thought into licensing site-to-site tunnels on the ASA, so what’s happening here?

A quick Google search revealed that this can happen when running ASA Contexts. Yep, that’s what we’re doing.

 

Fortunately, it’s not a difficult fix. We just need to create (or edit) a resource class in the system context and add it to this context.

class vpn
  limit-resource VPN Other 5%

context OurContext
  member vpn

 

This allocates 5% of our licenses to the context that the ASA VPN is in. After switching back to the context the VPN is on, I can see that the SA’s are finally forming, and the VPN is up.

 

 


Conclusion

So many things went wrong with this ASA VPN connection, and any one of them alone could have broken the tunnel. That’s what made this so interesting, and worth documenting here.

We’ve covered a few troubleshooting techniques. Obviously this is not a comprehensive list, but hopefully, the methodology will help you in the future.

 

If there’s anything generic I can recommend, it is:

  1. Understand how IPSec works
  2. Use the two show SA commands
  3. Use the two debug commands

 

Wednesday, September 26th, 2018

AWS to ASA VPN Issues

ASA to AWS VPN Drops Traffic

Scenario

I’ve been working with a company that integrates with several partners. One of these partners requires an AWS to ASA VPN to access their services.

That shouldn’t be a problem at all of course. The company in question has ASA’s running Firepower Threat Defence, which supports site-to-site VPN’s in a very similar manner to the traditional ASA.

So, I configured an ‘always on’ policy-based VPN (No VTI support in FTD yet), which seems to work fine. Well, for a while anyway.

Continue reading “AWS to ASA VPN Issues”

High CPU in Firepower

High CPU Usage in Firepower

Friday June 22, 2018

The Symptoms

I use Firepower Management Center quite a bit. Recently, I started getting health monitoring alerts. It looked something like this:

Health Monitor Alert from 10.10.10.10Severity: Critical Module: CPU Usage

Description: Using CPU05 95.34%

These alerts were spamming me every 5 minutes for a few hours.

One of our ASA’s running Firepower Services was having a bad time.

Continue reading “High CPU in Firepower”

BGP With a Service Provider

BGP With a Service Provider

Tuesday December 19, 2017

 

So you want to peer with a service provider. Never done it before? Overwhelmed? Don’t know where to start? If this sounds familiar, then this article is for you! We’re going to have a look at the process of peering with an ISP. We’re not going to look too deeply into the technical details. Rather, we’ll focus more on the process.

Continue reading “BGP With a Service Provider”

vPC and LAG Convergence

vPC and LAG Convergence

Thursday November 16, 2017

 

Recently Cisco released NXOS 7.0(3)I7(1) for the Nexus 9000 series switches. This brings two new features, called vPC Fast Convergence and LACP Convergence. These are also available on the 7000 series switches.

There wasn’t a lot of information readily available, so I’m going to share what I’ve learned here. I’d like to take a moment to thank Amith Ronad from Cisco for helping me to understand these features.

Continue reading “vPC and LAG Convergence”

Hitless vPC Role Change

Hitless vPC Role Change

Thursday October 19, 2017

 

“Always two there are; no more, no less. A vPC primary and a vPC secondary.”Yoda (paraphrased)

 

Like Yoda says, there has always been a primary and secondary in a vPC relationship. But, they’ve always been non-preemptive. That means that a secondary will not automatically become primary unless there’s a failure of some sort.

Continue reading “Hitless vPC Role Change”

Dynamic Routing and FEX

Dynamic Routing and Fabric Extenders

Wednesday August 30, 2017

 

The Problem

A few weeks ago I was working on a customer’s network when I found an OSPF problem. For some reason, an ASA wouldn’t peer with a Nexus switch. To make it a bit weirder, the problem only happened on the default VRF, and only with OSPFv3. On the Nexus side, I could see the ASA neighbour, but it was stuck in INIT. On the ASA side, I couldn’t see the neighbour at all.

Continue reading “Dynamic Routing and FEX”