IPSec Basics

 IPSec Basics

Last Updated: [last-modified] (UTC)

Introduction to IPSec

Internet Protocol Security (RFC 4301) is vendor-agnostic. It is a collection of protocols and methods that encrypt traffic.

By design, IPSec supports IPv4 or IPv6. Any upper layer protocols that run on IP may be encrypted with IPSec.

IPSec provides data confidentiality, data integrity, origin authentication, and anti-replay services. This makes is a popular choice to use across an insecure network, such as the internet.

IPSec is independent of specific encryption algorithms. It can support several cryptographic methods, such as AES, DES, and so on.

The purpose of this article is to provide a very basic understanding of IPSec. For this reason, IKEv1 in main mode will be primarily discussed. The alternative to main mode is aggressive mode.


The Authentication Header (AH) provides data integrity and origin authentication. That is, AH provides security for the packet headers.

This verifies that the data is not altered in transit, and that the data is coming from the correct source.

The Encapsulating Security Payload (ESP) protocol provides data confidentiality. It prevents tampering with the packet payload.

This is the encryption of the main body of data (the payload) that is in transit.

Security Association (SA) is a collection of procedures and parameters. Security devices agree on these parameters when creating a connection. The SA provides the framework for AH and ESP to work.

As part of the SA establishment, the devices are authenticated and keys are exchanged. Internet Security Association and Key Management Protocol (ISAKMP) does this. ISAKMP is only a framework., IKE does the actual key exchange.

Internet Key Exchange (IKE) is key management. This is a method of securely sharing keys between two security devices. There are currently two versions of IKE; IKEv1 and IKEv2.



IPSec Process

An IPSec connection using IKEv1 has two main phases.

Phase 1 – IKE. Endpoints identify themselves, and mutually authenticate. They agree on security parameters, to create SA’s.

Phase 2 – Quick Mode. Using the Phase 1 tunnel, phase 2 creates the tunnel for data. Additional SA’s are created


Phase 1

In phase 1, IKEv1 uses a Diffe-Hellman (DH) key exchange to begin creating the SA between the devices. This creates a tunnel for the negotiation of further security parameters.

Diffie Hellman

Diffie Hellman is a mathematical algorithm used to establish shared secrets between two endpoints. This works even though they are using an insecure medium, such as the internet.

Phase 1 includes a policy containing:

  • The authentication method; Such as certificates or preshared keys
  • The encryption method; Such as DES or AES
  • The hashing method, MD5 or SHA
  • The DH Group; Usually 1, 2, or 5
  • The lifetime in seconds


There are three exchanges between the two endpoints, containing six messages:

  1. The initiator sends proposed policies to the responder
  2. If the responder has a corresponding policy, it will accept one and reply
  3. The initiator generates DH values and sends them
  4. The responder generates DH values.
    • The responder sends te DH values
    • Both ends now used the DH values to generate a DH shared secret
  5. The initiator sends its ID and a hash of its preshared key for authentication
  6. The responder sends its ID and a hash of its preshared key
    • Both ends check that their hashes match


Three successful exchanges result in the creation of the initial tunnel.


Phase 2

In this phase, IPSec establishes SA’s between endpoints using the Phase 1 tunnel.


Perfect Forward Secrecy

At this point, IPSec negotiation will fail if one side is using PFS, but the other isn’t.

A new set of DH keys need to be generated if PFS is used.

Three messages are used in this phase:

  1. The initiator sends some additional parameters to the responder
    • The encapsulation method; ESP or AH
    • The hashing method; MD5-HMAC or SHA-HMAC
    • The DH group; 1, 2, or 5
    • The SPI; The label for the end of the tunnel
  2. The responder accepts the proposal if it can, and sends its own SPI and matching parameters
    • Both ends generate new DH keys for encryption
  3. The initiator acknowledges that the message was received
    • The SA has now been established


Any traffic that passes over the tunnel is now encrypted, and will continue to be until the tunnel is terminated.



Changes in IKEv2

There are a few improvements in IKEv2. Notable changes are:

  • Lower bandwidth usage
  • Addition of EAP support
  • Built-in NAT traversal
  • Supports a tunnel ‘liveness’ check
  • Adds support for the MOBIKE protocol (mobility and multihoming)



Network Sorcery – IPSec, Internet Protocol Security protocol suite

Wikipedia – IPSec

Wikipedia – Internet Security Association and Key Management Protocol

PeteNetLive – Cisco ASA Site to Site VPN’sSite to Site ISAKMP VPN (Main Mode)

Cisco Support Forum – How does ikev1, ikev2 relate to older ipsec/isakmp?

Difference Between – Difference Between IKEv1 and IKEv2

Cisco Meraki – Networking Fundamentals: IPSec and IKE

CCIE or Null – IKE main mode, aggressive mode, & phase 2

Marwan Al-shawi and Andre Laurent – Designing for Cisco Network Service Architectures (ARCH) Foundation Learning Guide: CCDP ARCH 300-320 (ISBN 158714462X)

Leave a Reply