Theory - The Official IPsec Howto For Linux

What is IPsec?

IPsec is an extension to the IP protocol which provides security to the IP and the upper-layer protocols. It was first developed for the new IPv6 standard and then “backported” to IPv4. The IPsec architecture is described in the RFC2401. The following few paragraphs will give you a short introduction into IPsec.

IPsec uses two different protocols - AH and ESP - to ensure the authentication, integrity and confidentiality of the communication. It can protect either the entire IP datagram or only the upper-layer protocols. The appropiate modes are called tunnel mode and transport mode. In tunnel mode the IP datagram is fully encapsulated by a new IP datagram using the IPsec protocol. In transport mode only the payload of the IP datagram is handled by the IPsec protocol inserting the IPsec header between the IP header and the upper-layer protocol header (see Figure 1).

Figure 1. IPsec tunnel and transport mode

To protect the integrity of the IP datagrams the IPsec protocols use hash message authentication codes (HMAC). To derive this HMAC the IPsec protocols use hash algorithms like MD5 and SHA to calculate a hash based on a secret key and the contents of the IP datagram. This HMAC is then included in the IPsec protocol header and the receiver of the packet can check the HMAC if it has access to the secret key.

To protect the confidentiality of the IP datagrams the IPsec protocols use standard symmetric encryption algorithms. The IPsec standard requires the implementation of NULL and DES. Today usually stronger algorithms are used like 3DES, AES and Blowfish.

To protect against denial of service attacks the IPsec protocols use a sliding window. Each packet gets assigned a sequence number and is only accepted if the packet's number is within the window or newer. Older packets are immediately discarded. This protects against replay attacks where the attacker records the original packets and replays them later.

For the peers to be able to encapsulate and decapsulate the IPsec packets they need a way to store the secret keys, algorithms and IP addresses involved in the communication. All these parameters needed for the protection of the IP datagrams are stored in a security association (SA). The security associations are in turn stored in a security association database (SAD).

Each security association defines the following parameters:

• Source and destination IP address of the resulting IPsec header. These are the IP addresses of the IPsec peers protecting the packets.

• IPsec protocol (AH or ESP), sometimes compression (IPCOMP) is supported, too.

• The algorithm and secret key used by the IPsec protocol.

• Security Parameter Index (SPI). This is a 32 bit number which identifies the security association.

Some implementations of the security association database allow further parameters to be stored:

• IPsec mode (tunnel or transport)

• Size of the sliding window to protect against replay attacks.

• Lifetime of the security association.

Since the security association defines the source and destination IP addresses, it can only protect one direction of the traffic in a full duplex IPsec communication. To protect both directions IPsec requires two unidirectional security associations.

The security assocations only specify how IPsec is supposed to protect the traffic. Additional information is needed to define which traffic to protect when. This information is stored in the security policy (SP) which in turn is stored in the security policy database (SPD).

A security policy usually specifies the following parameters:

• Source and destination address of the packets to be protected. In transport mode these are the same addresses as in the SA. In tunnel mode theymay differ!

• The protocol (and port) to protect. Some IPsec implementations do not allow the definition of specific protocols to protect. In this case all traffic between the mentioned IP addresses is protected.

• The security association to use for the protection of the packets.

The manual setup of the security association is quite error prone and not very secure. The secret keys and encryption algorithms must be shared between all peers in the virtual private network. Especially the exchange of the keys poses critical problems for the system administrator: How to exchange secret symmetric keys when no encryption is yet in place?

To solve this problem the internet key exchange protocol (IKE) was developed. This protocol authenticates the peers in the first phase. In the second phase the security associations are negotiated and the secret symmetric keys are chosen using a Diffie Hellmann key exchange. The IKE protocol then even takes care of periodically rekeying the secret keys to ensure their confidentiality.