Encryption basics

This document gives a general view about encryption, encryption schemes and how they work.


Public Key Distribution System

Public key systems, where each corresponded has a pair of keys, can solve both these problems. A key pair is composed of two mathematically related keys: a public key known to everyone and a private key only known to its owner.

The Diffie-Hellman public key distribution system can be used to share a secret key without communicating any secret information, so a secure channel is not required for the key exchange. Also, under this system, only one key pair needs to be managed for each corresponded. Once the correspondents have computed the share secret, they can use it to encrypt communication between them.


Diffie-Hellman Scheme

Under the Diffie-Hellman Scheme, a secret key is agreed on and exchanged as follows, using the private and public parts of Werder’s and Bremens Diffie-Hellman keys.

  1. Werder gets Bremens public key and performs a calculation involving his own private key and Bremens public key
  2. Bremen gets Werder’s public key and performs a calculation involving his own private key and Werder’s public key


The results of both calculation are the same, and this result serves as the secret key. In this way, a secret key can be agreed on without any secret information being communicated. There is no opportunity for an eavesdropper to determine the secret key.

diffie-hellman scheme


RSA Public Keys

In contrast to a Diffie-Hellman public-private key pair, an RSA public-private key pair is used for encrypting and decrypting messages. A message encrypted with the public key can only be decrypted with the private key and vice versa.


Digital Signatures

Generating a digital signature: A digital signature acts as proof of the sender’s identity and the message’s integrity. Digital signatures can be created using a public key encryption system, such as RSA, as follows:

  1. Werder wants to send Bremen a message. First, he computes a hash of the message
    A hash is a mathematical computation performed on the text of a message. Since changing even one bit in the message results in a completely different hash. It is computationally unfeasible to compute a message from a given hash, the hash uniquely identifies the message.
  2. Werder encrypts the hash with its own private key
  3. Next, Werder encrypts the message with Bremens public key
  4. Werder then sends both the encrypted message and the encrypted hash to Bremen


Verifying a Digital Signature: Upon receiving the message, Bremen can confirm that the message was indeed sent by Werder and that it was not tampered with, as follows:

  1. First, Bremen decrypts the message with its own private key >> Bremen can do this, because only he knows his private key
  2. Next, he decrypts the hash with Werder’s public key
  3. Bremen calculates the hash of the unencrypted message (this is the same calculation that Werder performed) and compares it to the hash received from Werder


If they are the same, then Bremen can be sure that:

  • The message can only have been sent by Werder, because Werder is the only who knows Werder’s private key and thus the only person who could have encrypted the hash
  • The message was not tampered, because the hash Werder calculated is the same as the hash Bremen calculated



If Werder and Bremen obtain each other’s public key over an insecure channel, such as the Internet, they must be certain that the keys are genuine. For this reason, there must be a trusted third party, known as a Certificate Authority (CA), from which Werder can reliably obtain Bremens public key, even over an insecure channel (and vice versa).

The CA sends Werder a certificate, which consists of:

  • Bremens unique identifier (for example, his LDAP Distinguished Name)
  • Bremens public key
  • The CA’s digital signature, encrypted with the CA’s private key


Werder can then verify the certificate (see above). Werder does not have to obtain Bremens certificate directly from the CA. Bremen can obtain the certificate and send it to anyone who requests it, because a certificate cannot be tampered with.

The certificate is “proof” that the message, in this case, Bremens public key is genuine. Werder and Bremen can use each other’s public keys with confidence only if they are certified by certificates from a trusted CA.


Encryption Schemes

An encryption scheme consists of the following elements:

  1. An encryption algorithm for encrypting messages
  2. An authentication algorithm for ensuring integrity, that is, that the message have not been tampered with
  3. A key management protocol for generating and exchanging keys


Encryption Schemes supported by various vendors

  1. Manual IPSec - manual IPSec is an encryption and authentication scheme that uses fixed keys
  2. SKIP (Simple Key Management for Internet Protocol) - SKIP is developed by Sun Microsystems and adds two features to Manual IPSec
    Improved keys - Manual IPSec uses fixed keys. In contrast, SKIP uses a hierarchy of constantly changing keys
    Key management - Manual IPSec does not provide a mechanism by which users can exchange keys. SKIP implements a key management protocol for Manual IPSec)
  3. ISAKMP/OAKLEY - like SKIP, ISAKMP/OAKLEY adds key management features to Manual IPSec


Manual IPSec Encryption Scheme

Manual IPSec is an encryption and authentication scheme. A security association (SA) is associated with each packet, consisting of:

  • Functionality – indicates whether the packet is encrypted, authenticated, or both
  • Algorithms – specifies the encryption algorithm (for example AES) and the authentication algorithm (for example SHA-1) used in the packet
  • Keys used in the above algorithm
  • Additional data, for example initialization vector (IV)


A 32-bit number, known a the security parameter index (SPI), identifies a specific SA. An SPI is simply an identifier, assigned by correspondents themselves in a particular context, and has no meaning outside the context.



IP packets are encrypted in accordance with the Encapsulation Security Payload (ESP) standard (RFC 1825, 1827 and 1829). As its name implies, the ESP standard specifies that the original packet is encrypted and then encapsulated into a new, longer packet. There are two modes of performing this encapsulation:

  • Tunnel Mode
  • Transport Mode


Tunnel Mode:

In tunnel mode, the entire packet (including the IP header) is encrypted in accordance with the SA previously decided upon the correspondents. An ESP header containing the SPI and other data is added to the start of the packet, and a new IP header is constructed.

The new IP packet consists of:

  • The new IP header
  • The ESP header
  • The encrypted original packe


The new packet is then send on its way. An advantage of this mode is that the destination specified in the new IP header may be different from the one in the original IP header. The picture below shows a paket in IPv4 tunnel mode.

ipv4 tunnel mode


Transport Mode:

In the transport mode, the IP header is not encrypted. An ESP header is inserted between the IP header and the transport layer header. The transport layer header and everything following is encrypted.

This mode does not increase the length of a packet as much as the tunnel mode does. The encrypted packet must be sent to its original destination.



If authentication is specified by the SA, then an Authentication header (AH) is added to the packet, in addition to the ESP header (and to the second IP header in Tunnel Mode). The picture below shows a paket in IPv4 transport mode.

ipv4 transport mode

Manual IPSec has two shortcomings:

  • The keys are fixed for the connection’s duration
  • There is no mechanism for exchanging keys


Both SKIP and ISAKMP/OAKLEY addresses these shortcomings.


SKIP Encryption Scheme

SKIP provides a hierarchy of keys that change over time, which are used to encrypt the connection as well as to implement a key management protocol. SKIP also includes ESP and AH, and adds his own header to the packet.

The Encryption Key and the Authentication Key are derived from the Session Key, which changes at fixed intervals or when the amount of data encrypted exceeds a given threshold. The newly changed Session Key is communicated by encrypting it with the Kijn key, which changes once every hour. The Kijn key is derived from the Diffie-Hellman shared secret Kij key, using a cryptographic hash function. Each correspondent obtains the public part of the others correspondent’s Diffie-Hellman key from a Certificate Authority, which signs the transmission with its own RSA key.

SKIP includes a protocol (Certificate Discovery Protocol) for exchange of public keys. However, this protocol is not supported by FireWall-1, which uses instead a proprietary protocol (RDP = Reliable Data Protocol) for key exchange.


ISAKPMP/OAKLEY Encryption Scheme

ISAKPMP/OAKLEY is a standard for negotiating Security Associations (SA) between two hosts that will be using IPSec, and is the key management scheme that was chosen for IP Version 6. In IP Version 4, ISAKMP/OAKLEY is optional. ISAKPMP/OAKLEY offers improved authentication and Perfect Forward Secrecy (PFS).

ISAKPMP/OAKLEY key exchange is divided into two phases:

  • ISAKPMP Phase One (Main/Aggressive Mode)


In this phase, the peers negotiate an ISAKPMP Security Association that will be used for encrypting and authentication Phase Two exchanges. Phase One involves long and CPU-intensive computations, and so is executed infrequentl. The negotiated SA includes the encryption method, authentication method and keys. This SA is used in Phase two negotiation.

  • ISAKPMP Phase Two (Quick Mode)


Using the SA negotiate in Phase One, the peers negotiating an SA for encrypting the IPSec traffic. Keys can be modified as often as required during a connection’s lifetime by performing Phase Two.


Comparison of Encryption Schemes

Manual IPSec
Portability Standard Standard supported by Sun and other vendors Industry standard
Key Management No Yes Yes
Session Keys Fixed Keys change over time or as amount of data encrypted exceeds a given threshold Keys change at configurable time during a connection’s lifetime
Number of keys required is proportional to the .. Square of the number of correspondents Number of correspondents Square of the number of correspondents
Packet size Increased Increased Increased
Gateway can encrypt/decrypt on behalf of other hosts Yes (in tunnel mode) Yes (in tunnel mode) Yes (in tunnel mode)




Encrypting a message modifies (encrypts) its text (plaintext) or „cleartext“ so that the encrypted text (ciphertext) can only be read (decrypted) with the aid of some additional information (e.g. a key) known only to the sender and the intended recipient.



Authenticating a message verifies the authenticity of both the sender and the message, checking for possible tampering and interference with the communication.


Encrypting a message

The simplest way to encrypt a message is by using a secret key, one that both encrypts and decrypts the message. Ensuring this key is very critical, since anyone who knows the key can decrypt and read the message.


Sharing a Secret Key

Secret key encryption is simple and fast, but there are two problems associated with its use:

  • A secure channel is required by which the correspondents can agree on a key
  • The number of keys can quickly become unmanageable, since there must be a different key for each pair of possible correspondents



AH = Authentication Header
Cert = Computer Emergency Response Team
Chap = Challenge Handshake Authentication Protocol
DES = Data Encryption Standard
DMZ = De-militarized Zone
ESP = Encapsulated Security Payload
Idea = International Data Encryption Algorithm
IETF = Internet Engineering Task Force
IPSec = IP Security
IPV4 = Internet Protocol Version 4
IPv6 = Internet Protocol Version 6
L2F = Layer 2 Forwarding
L2TF = Layer 2 Tunneling Protocol
MD5 = Message Digest 5
NAT = Network Address Translation
NCP = Network Control Protocol
PIN = Personal Identification Number
PPP = Point-to Point Protocol
PoP = Point of Presence
RADIUS = Remote Authentication Dial-in User Service
RAS = Remote Access Server
RFC = Request for Comment
Satan = Security Administrator Tool for Analyzing Networks
SHA = Secure Hash Algorithm
SLIP = Serial Line Protocol
TCP = Transmission Control Protocol
TTL = Time to Live
UDP = User Datagram Protocol
VPN = Virtual Private Network
VTP = Virtual Tunneling Protocol