IPSec (Internet Protocol Security) provides a framework for securing communications over IP networks by using cryptographic security services. It operates primarily at the Network Layer (Layer 3) of the OSI model, ensuring the security of IP packets. IPSec supports two primary encapsulation modes: Transport Mode and Tunnel Mode. Each mode serves specific purposes and operates differently to secure data transmissions.

Lets us discuss both the models in detail

IPSec Transport Mode

Transport Mode is designed to protect the payload of the IP packet while leaving the original IP header unchanged. This mode is primarily used for securing end-to-end communications between two hosts on the same network.

IPSec Transport Mode
IPSec Transport Mode

How Transport Mode Works:

  1. Original IP Packet:
    • The original IP packet consists of an IP header and a payload (which includes the transport layer segment, such as TCP or UDP).
  2. Encapsulation:
    • IPSec encapsulates the payload with the appropriate security protocol headers (either Authentication Header (AH) or Encapsulating Security Payload (ESP)).
    • Authentication Header (AH): Provides integrity and authentication for the entire IP packet.
    • Encapsulating Security Payload (ESP): Provides confidentiality, integrity, and authentication for the payload.
  3. Resulting IP Packet:
    • The resulting IP packet retains the original IP header with added IPSec headers, ensuring that the packet can be routed to its destination without modification.

Characteristics of Tunnel Mode:

  • Focus on Payload: In Transport Mode, IPSec secures only the payload of the IP packet, leaving the original IP header unchanged. This ensures that the data being transmitted between two hosts is protected without modifying the IP routing information.
  • Minimal Overhead: Since the IP header is not encapsulated, Transport Mode adds less overhead compared to Tunnel Mode, resulting in smaller packet sizes and potentially better performance.

IPSec Tunnel Mode

Tunnel Mode is designed to encapsulate the entire original IP packet, including its IP header, within a new IP packet. In this mode there are two IP headers i.e. inner IP header and outer IP header. The inner IP Header is the original IP header. The outer IP Header is added by this mode to provide IPSec. This mode is commonly used for network-to-network or gateway-to-gateway communications, such as in Virtual Private Networks (VPNs).

IPSec Tunnel Mode
IPSec Tunnel Mode

How Tunnel Mode Works:

  1. Original IP Packet:
    • The original IP packet includes the IP header and the payload.
  2. Encapsulation:
    • The entire original IP packet, including the IP header and payload, is encapsulated within a new IP packet.
    • IPSec security protocol headers (AH or ESP) are added for protection. In picture, we have taken example of ESP.
  3. New IP Header:
    • A new IP header is added by the network device providing the IPSec service. This new header includes the source and destination IP addresses of the IPSec tunnel endpoints.

Characteristics of Tunnel Mode:

  • Double IP Headers: The encapsulated packet has two IP headers: the original IP header (inner header) and the new IP header (outer header) added by the IPSec gateway.
  • Full Packet Protection: The entire original IP packet is protected by IPSec security protocols, ensuring confidentiality, integrity, and authentication.

Comparision

FeatureTransport ModeTunnel Mode
Primary Use CaseEnd-to-end communication between two hostsNetwork-to-network or gateway-to-gateway communication
EncapsulationOnly the payload (data) of the IP packet is encrypted and authenticatedEntire IP packet (including the original IP header and payload) is encapsulated within a new IP packet
Original IP HeaderRemains intact, visible to routers and intermediariesEncapsulated within the new IP packet, not visible to intermediaries
New IP HeaderNot added, uses the original IP headerAdded by the IPSec device, specifying tunnel endpoints
OverheadLower overhead, smaller packet sizeHigher overhead, larger packet size due to additional headers
Security ProtocolsAH or ESPAH or ESP
ConfidentialityPayload confidentiality with ESPFull packet confidentiality with ESP
Integrity and AuthenticationProvided for payload with AH or ESPProvided for the entire original packet with AH or ESP
Typical Use CasesSecure internal host-to-host communication, application-level securitySite-to-site VPNs, remote access VPNs, gateway-to-gateway communications
RoutingOriginal IP addresses are used for routingNew IP addresses are used for routing
FragmentationLess likely due to lower overheadMore likely due to increased packet size
PerformanceGenerally higher due to lower overheadMay be lower due to higher overhead and possible fragmentation
ComplexitySimpler to implement for end-to-end scenariosMore complex due to encapsulation of the entire packet

References