ESP Transport and Tunnel Mode
Figure 9-12 compares ESP Transport mode versus ESP
Tunnel mode. In ESP Transport mode, the IP payload is
encrypted, while the original IP header is left intact. An ESP header is
inserted after the IP header and before the upper-layer protocol header
(original Layer 3 data), while an ESP trailer is appended after the original
data. The inserted ESP header contains a Security parameter index (SPI) value to
identify the VPN security association, a sequence number field, and
authentication data to verify packet authenticity.
In Transport mode, only the original data and ESP trailer fields
are encrypted. The IP header isn’t encrypted because some fields will be
required by a Layer 3 device encountered in transit. The ESP header field isn’t
encrypted because it’s needed by the destination peer to decipher the
payload.
If ESP authentication is used, the upper-layer protocols (original
data payload) are hashed with the ESP header and trailer, and then appended to
the packet as the ESP Authentication field. Cisco IOS software and the PIX
Firewall refer to this authentication service as ESP HMAC. ESP Transport mode
doesn’t authenticate any portion of the IP header itself.
In ESP Tunnel mode, as in AH, a new IP header reflecting the end
points of the VPN tunnel is added. Both encryption and authentication
incorporate the entire original IP header.
When both authentication and encryption are selected,
encryption is performed before authentication. This order facilitates rapid
detection and rejection of replayed or bogus packets by the receiving peer,
potentially reducing the impact of denial-of-service (DoS) attacks.
Choosing AH versus ESP
Both AH and ESP support both MD5 and SHA-1 hashing
algorithms for authentication. The main difference between ESP and AH
authentication is this: ESP doesn’t protect any IP header fields in Transport
mode. Both ESP and AH authenticate all IP header fields in Tunnel
mode.