ASA VPN Troubleshooting

Yesterday, I assisted with troubleshooting ASA VPN issues. A local ASA needed to build a site-to-site (aka L2L) IPSec VPN tunnel to a non-ASA third-party. The tunnel was not coming up.

The config all appeared to be there, and the third-party said their config was in place too.

It’s time to troubleshoot. Here are the steps I followed.

Check if SA’s are Forming

This is always my first step when troubleshooting. There should be phase-1 SA’s and phase-2 SA’s for the ASA VPN to work.

You can find phase-1 SA’s with:

show crypto isakmp sa

And phase-2 SA’s with:

show crypto ipsec sa

In my case, there were no phase-1 SA’s, so there was no point looking for phase-2 SA’s.

Perhaps the ASA hasn’t seen any interesting traffic yet and hasn’t tried to bring the tunnel up. We can try to do this with packet tracer:

packet-tracer input Inside tcp 10.0.0.1 http 172.16.0.1 http

This just simulates some http traffic from 10.0.0.1 to 172.16.0.1. These are hosts that should be able to communicate over the tunnel.

This doesn’t work, and no SA’s formed. So, phase-1 looks like a good place to focus on.

Check IKE Proposals

The first step in troubleshooting phase-1 (IKEv2 in my case) is to confirm that there are matching proposals on both sides. The proposals include acceptable combinations of cyphers, hashes, and other crypto information.

This is easy if you control both ends of the ASA VPN tunnel. Just look at what’s configured. In my case, it’s a little harder, as a third-party manages the remote end of the tunnel.

Instead, I can find this with a debug command:

debug crypto ikev2 protocol 64

This will show us any errors with IKEv2 (you can substitute IKEv1 if you need to).

The ’64’ is the debugging level. This can be from 1 to 256. The higher the number, the more detail you get. Don’t go too high too quickly, as there may be too much information to search through.

The debug gave me this:

IKEv2-PROTO-1: (16): Received Policies:
Proposal 1: AES-CBC-256 SHA1 SHA96 DH_GROUP_1024_MODP/Group 2
Proposal 2: AES-CBC-256 SHA256 SHA256 DH_GROUP_1024_MODP/Group 2
Proposal 3: AES-CBC-128 SHA1 SHA96 DH_GROUP_1024_MODP/Group 2
Proposal 4: AES-CBC-128 SHA256 SHA256 DH_GROUP_1024_MODP/Group 2
Proposal 5: 3DES SHA1 SHA96 DH_GROUP_1024_MODP/Group 2
Proposal 6: 3DES SHA256 SHA256 DH_GROUP_1024_MODP/Group 2
IKEv2-PROTO-1: (16): Failed to find a matching policy
IKEv2-PROTO-1: (16): Expected Policies:
Proposal 1: AES-CBC-256 SHA384 SHA384 DH_GROUP_2048_MODP_256_PRIME/Group 24

We can see here that the remote endpoint is offering six proposals, and we are offering one.

Our proposal does not match any of theirs, so IKE will fail. There are two ways to address this:

  1. We change our proposal to match theirs
  2. They change their proposal to match ours

For testing, it’s simpler just to change our side. We can figure out better security algorithms later.

Once the proposal has been changed, these debug errors no longer show up. But we’re still not getting phase-1 SA’s.

Also, it looks like the ASA is not even trying to bring up the tunnel. The proposal errors were coming from the remote end.

Time to check the network path…

Check the Path

There is another ASA on the path, so it sounds like a good idea to check that it’s not interfering.

The environment has an ASA which is the local VPN endpoint, as well as another ASA on the edge of the network
The environment has an ASA which is the local VPN endpoint, as well as another ASA on the edge of the network

A quick check of the ACLs showed that there were ‘IP Any’ rules for the local and remote endpoints. Unfortunately, that’s not enough.

We need to add ESP and AH in there (AH is not used in all environments though). These do not use IP, they are completely separate protocols.

Running packet-tracer on the edge, I noticed that the wrong interface was being used for egress traffic. I won’t get into too much detail, but there was a problem with routing.

One more issue fixed, but still no phase-1 SA’s.

Further Debugs

While on the edge ASA, A packet capture revealed two things:

  1. The third-party was periodically trying to connect to us
  2. The local VPN ASA was not trying to connect to the ASA at all

Time for another debug on the VPN ASA:

debug crypto ikev2 platform 64

This time we’re looking for platform related issues. As soon as I run another packet-tracer, I get some output, revealing something interesting:

IKEv2-PLAT-5: INVALID PSH HANDLE
IKEv2-PLAT-1: Failed to request a Other VPN license. Maximum tunnel count reached and/or license limit exceeded to initiate tunnel
IKEv2-PLAT-1: asa connect start L2L failed

We don’t normally need to put much thought into licensing site-to-site tunnels on the ASA, so what’s happening here?

A quick Google search revealed that this can happen when running ASA Contexts. Yep, that’s what we’re doing.

Fortunately, it’s not a difficult fix. We just need to create (or edit) a resource class in the system context and add it to this context.

class vpn
  limit-resource VPN Other 5%

context OurContext
  member vpn

This allocates 5% of our licenses to the context that the ASA VPN is in. After switching back to the context the VPN is on, I can see that the SA’s are finally forming, and the VPN is up.

Conclusion

So many things went wrong with this ASA VPN connection, and any one of them alone could have broken the tunnel. That’s what made this so interesting, and worth documenting here.

We’ve covered a few troubleshooting techniques. Obviously this is not a comprehensive list, but hopefully, the methodology will help you in the future.

If there’s anything generic I can recommend, it is:

  1. Understand how IPSec works
  2. Use the two show SA commands
  3. Use the two debug commands

Wednesday, September 26th, 2018