How IPSec embeds encryption in the ESP

IPSec handles the encryption at the packet level. The protocol it uses is called ESP. ESP also offers some Authentication as in the AH but the ESP authentication doesn’t authenticate the new IP header in front of the packet – in contrast to AH. The ESP (see below) follows the standard IP header in a IP datagram. It contains all the data and the higher level protocols relying on IP for routing. The figure below does not show the IP header.

The ESP contains six parts. The first two parts are not encrypted but are authenticated.

• Security Parameter Index (SPI) is an arbitrary 32bit number that specifies to the device receiving the packet what security parameters the sender is using for communication. The SPI points to the Security Association (SA) which defines the IPSec parameters for a session (which algorithms, which keys and how long those keys are valid
• The Sequence Number is a counter that increases each time a packet is sent to the same destination using the same IPSec parameters (SPI). It is an indicator which packet is which and how many packets have been sent with the same group of parameters (SPI). The sequence number provides protection against replay attacks, in which an attacker copies a packet and send it out of order, to confuse communication nodes
• The remaining parts of the ESP packet are all encrypted during transmission (with the exception of the authentication data).
• The Payload data is the actual data being carried by the packet
• The Padding allows for the fact that certain algorithms require the data to be a multiple of certain numbers of bytes
• The Pad length defines how much of the data is padding as opposed to the data
• The next header field is like a normal IP next header field - it specifies the type of data carried and the protocol above

For the authentication field, see "Authentication within ESP"

The ESP header is added after a standard IP header. The IP header specifies in the next header field with a certain number that an ESP header will follow (instead of a TCP header). Every standard IP router can forward this packet because it uses a standard IP header.

Encryption algorithms supported by ESP

ESP can support any number of encryption algorithms. It is up to the user which algorithm to use. You can even use different protocols for each party with whom you are communicating.

Tunneling with the ESP

Tunneling takes the original IP packet (with header) and encapsulates it in the payload of the ESP. It then adds to the packet a new IP header with a destination address of a VPN gateway.

Tunneling allows you to pass illegal IP addresses trough a public network (the Internet for example).

Note: Keep in mind that the tunneling mode generates more overhead as opposed to the transport mode because transport mode doesn’t need a new IP header. Transport mode, therefore, cannot be used with private IP addresses over a public network.

Another advantage of tunneling is, that the original source and destination addresses are hidden from users on the public network. This defeats or at least reduces the power of traffic analyses attacks. An attacker doing traffic analyses doesn’t know what is being said, but does know how much is being said.

Authentication within the ESP

The authentication field in the ESP contains something called an ICV (Integrity Check Value). In other words, this is a digital signature computed over the ESP minus the authentication field itself. It may also be omitted entirely if the authentication field isn’t checked for use or if you want to use the Authentication Header (AH).

The ICV is calculated on the ESP once encryption is complete. The source encrypts a hash of the data payload and attaches this value as the authentication data. The recipient confirms that there has been no tampering and that the payload did come from the expected person.

AH - Authentication Header

AH provides authentication without confidentiality. The IPSec suite’s second protocol, the AH, provides authentication services but does not address confidentiality. The AH may be applied alone, in concert with the ESP or in a nested fashion when using tunnel mode.
Authentication provided by the AH differs from that provided by the ESP. ESP’s authentication service does not protect the IP header that precedes the ESP. The AH protects the external IP header, along with the entire content of the ESP packet.

Note: The AH does not protect all of the fields in the external IP header because some of them change while in transit – AH is designed to work around these fields.
In the packet, the AH goes after the IP header but before the ESP (if present) or higher level protocol (TCP or UDP, in the absence of ESP).

The AH parts are as follows:

• The Next header field indicates what the higher level protocol following the AH is (ESP or TCP, for example)
• The Payload length field is a 8 bit field specifying the size of the AH
• The Reserved field – says it all
• The SPI, as in the ESP, specifies the set of security parameters (algorithms, keys and how long to use that keys) for that connection. In other words, the SPI is a pointer to a SA
• The Sequence number, as in the ESP, increases with every packet that is sent to the same destination with a given SPI. It is for the purpose of keeping track of the order the packets go in, to make sure that the same set of parameters is not used for too many packets, and finally, to defeat against replay attacks
• The Authentication data is the actual ICV (Integrity Check Value), or digital signature for the packet. It is much the same as the ICV used in the ESP. This field may include padding to bring the length of the header to an integral multiple of 32 bits in IPv4 or 64 bits in IPv6

Like the ESP, the AH can be used to implement tunneling mode. Also, as in the ESP, IPSec requires specific algorithms to be available for implementing in the AH. All IPSec implementations must support at least HMAC-MD5 and HMAC-SHA-1 for a minimum interoperability. HMAC is a symmetric authentication scheme supported by MD5 and SHA-1 hashes.

AH and ESP – two parts in the puzzle

You can name the AH and ESP to be as the building blocks of the IPSec suite. These two protocols provide such important services as authentication, confidentiality and integrity.All the services mentioned above rely on a secure key exchange. The encryption services provided by the ESP and AH represent a powerful technology to keep your data secret, for verifying its origin, and for protecting it from undetected tampering. But they mean little without a wisely designed infrastructure to support them, to distribute keys, and to negotiate security parameters between communicating parties.

So, the next part of this document discusses the key distribution service in IPSec, provided by the IKE protocol.

How many keys do you need?

Imagine twenty people need to exchange data on a secure way. They want to talk to each other on a secure and safely way trough the public network.
To get this job done, they use a symmetric encryption scheme - but unfortunately, they need 190 keys among them, and every user need to keep track of 19 keys. So, how to exchange this host of keys? For, example, they could distribute their keys on meetings specially initiated for that reason - but imagine how many meetings you will need – the management won’t be excited about so many meetings. Last but not least, to keep the encrypted data protected from tampering and brute force attacks, you need to change your symmetric encryption keys at least once a month.

Obviously, this is not practical. The first thing we can do in that situation is to change from symmetric to asymmetric encryption. With asymmetric encryption, each person issues a public key to all the others. Every user has his own private/public key pair – so every user needs to remember twenty keys (nineteen public, plus the own private key). Key exchanging in this example is much more easily compared to the first example – still complicated, but getting easier.

The above example illustrates two important things:

• Key exchange is fundamentally a complicated process
• Key exchange gets more complicated as the numbers of communicating players expands

So just because a system says it does encryption, that alone doesn’t mean that it is going to be appropriate for your needs. Any proposed VPN solution is only as good as it’s method of key distribution.