I faced a situation where a server had been migrated to a public cloud provider, and suddenly certain services were no longer working. Looks like we’ve got an MTU problem!
In particular, we found that accessing a particular third-party finance service over SSL was failing. We also found that we could not access their website.
On deeper inspection, we also found that we were having trouble getting to various other websites, but only the ones that used HTTPS.
This error only happened when we used our WAN connection to the provider. If we used the provider’s native internet link, everything was fine.
What Was The Issue?
I’ll get straight to the point on this one, and keep it brief. We found that the don’t fragment bit was set on SSL traffic.
It seems that the websites that were failing had a lot of data to transfer, pushing the packet size over the MTU on our WAN.
This meant that the packet needed to be fragmented, but the DF bit prevented it. So, the result was that the packet was dropped.
I haven’t been able to find anything to say that SSL/HTTPS traffic should have the DF bit set, although others report having the same issue. It seems that this is specific to webservers or browsers.
How is This Resolved?
The problem was clearly on the WAN. Part of the solution was an encrypted GRE tunnel over a private link.
Looking at the tunnel, it was easy to see that the MTU values were incorrect. Specifically, in this case, the MPLS MTU size.
Yesterday, I assisted with troubleshooting ASA VPN issues. A local ASA needed to build a site-to-site (aka L2L) IPSec VPN tunnel to a non-ASA third-party. The tunnel was not coming up.
The config all appeared to be there, and the third-party said their config was in place too.
It’s time to troubleshoot. Here are the steps I followed.
Check if SA’s are Forming
This is always my first step when troubleshooting. There should be phase-1 SA’s and phase-2 SA’s for the ASA VPN to work.
You can find phase-1 SA’s with:
show crypto isakmp sa
And phase-2 SA’s with:
show crypto ipsec sa
In my case, there were no phase-1 SA’s, so there was no point looking for phase-2 SA’s.
Perhaps the ASA hasn’t seen any interesting traffic yet and hasn’t tried to bring the tunnel up. We can try to do this with packet tracer:
This just simulates some http traffic from 10.0.0.1 to 172.16.0.1. These are hosts that should be able to communicate over the tunnel.
This doesn’t work, and no SA’s formed. So, phase-1 looks like a good place to focus on.
Check IKE Proposals
The first step in troubleshooting phase-1 (IKEv2 in my case) is to confirm that there are matching proposals on both sides. The proposals include acceptable combinations of cyphers, hashes, and other crypto information.
This is easy if you control both ends of the ASA VPN tunnel. Just look at what’s configured. In my case, it’s a little harder, as a third-party manages the remote end of the tunnel.
Instead, I can find this with a debug command:
debug crypto ikev2 protocol 64
This will show us any errors with IKEv2 (you can substitute IKEv1 if you need to).
The ’64’ is the debugging level. This can be from 1 to 256. The higher the number, the more detail you get. Don’t go too high too quickly, as there may be too much information to search through.
We can see here that the remote endpoint is offering six proposals, and we are offering one.
Our proposal does not match any of theirs, so IKE will fail. There are two ways to address this:
We change our proposal to match theirs
They change their proposal to match ours
For testing, it’s simpler just to change our side. We can figure out better security algorithms later.
Once the proposal has been changed, these debug errors no longer show up. But we’re still not getting phase-1 SA’s.
Also, it looks like the ASA is not even trying to bring up the tunnel. The proposal errors were coming from the remote end.
Time to check the network path…
Check the Path
There is another ASA on the path, so it sounds like a good idea to check that it’s not interfering.
The environment has an ASA which is the local VPN endpoint, as well as another ASA on the edge of the network
A quick check of the ACLs showed that there were ‘IP Any’ rules for the local and remote endpoints. Unfortunately, that’s not enough.
We need to add ESP and AH in there (AH is not used in all environments though). These do not use IP, they are completely separate protocols.
Running packet-tracer on the edge, I noticed that the wrong interface was being used for egress traffic. I won’t get into too much detail, but there was a problem with routing.
One more issue fixed, but still no phase-1 SA’s.
Further Debugs
While on the edge ASA, A packet capture revealed two things:
The third-party was periodically trying to connect to us
The local VPN ASA was not trying to connect to the ASA at all
Time for another debug on the VPN ASA:
debug crypto ikev2 platform 64
This time we’re looking for platform related issues. As soon as I run another packet-tracer, I get some output, revealing something interesting:
IKEv2-PLAT-5: INVALID PSH HANDLE
IKEv2-PLAT-1: Failed to request a Other VPN license. Maximum tunnel count reached and/or license limit exceeded to initiate tunnel
IKEv2-PLAT-1: asa connect start L2L failed
We don’t normally need to put much thought into licensing site-to-site tunnels on the ASA, so what’s happening here?
Fortunately, it’s not a difficult fix. We just need to create (or edit) a resource class in the system context and add it to this context.
class vpn
limit-resource VPN Other 5%
context OurContext
member vpn
This allocates 5% of our licenses to the context that the ASA VPN is in. After switching back to the context the VPN is on, I can see that the SA’s are finally forming, and the VPN is up.
Conclusion
So many things went wrong with this ASA VPN connection, and any one of them alone could have broken the tunnel. That’s what made this so interesting, and worth documenting here.
We’ve covered a few troubleshooting techniques. Obviously this is not a comprehensive list, but hopefully, the methodology will help you in the future.
If there’s anything generic I can recommend, it is:
I’ve been working with a company that integrates with several partners. One of these partners requires an AWS to ASA VPN to access their services.
That shouldn’t be a problem at all of course. The company in question has ASA’s running Firepower Threat Defence, which supports site-to-site VPN’s in a very similar manner to the traditional ASA.
So, I configured an ‘always on’ policy-based VPN (No VTI support in FTD yet), which seems to work fine. Well, for a while anyway.
So you want to peer with a service provider. Never done it before? Overwhelmed? Don’t know where to start? If this sounds familiar, then this article is for you! We’re going to have a look at the process of peering with an ISP. We’re not going to look too deeply into the technical details. Rather, we’ll focus more on the process.
Recently Cisco released NXOS 7.0(3)I7(1) for the Nexus 9000 series switches. This brings two new features, called vPC Fast Convergence and LACP Convergence. These are also available on the 7000 series switches.
There wasn’t a lot of information readily available, so I’m going to share what I’ve learned here. I’d like to take a moment to thank Amith Ronad from Cisco for helping me to understand these features.
“Always two there are; no more, no less. A vPC primary and a vPC secondary.”Yoda (paraphrased)
Like Yoda says, there has always been a primary and secondary in a vPC relationship. But, they’ve always been non-preemptive. That means that a secondary will not automatically become primary unless there’s a failure of some sort.
If you’ve never worked in a third-party data centre before, the first time can be a bit of a shock. There are a lot of rules and procedures to follow, and each data centre is a bit different from the last one.
A few weeks ago I was working on a customer’s network when I found an OSPF problem. For some reason, an ASA wouldn’t peer with a Nexus switch. To make it a bit weirder, the problem only happened on the default VRF, and only with OSPFv3. On the Nexus side, I could see the ASA neighbour, but it was stuck in INIT. On the ASA side, I couldn’t see the neighbour at all.
