A data center interconnect using a VXLAN was recently implemented. A network engineer observes slow performance and fragmentation on the…

Virtual Extensible LAN (VXLAN) is an encapsulation protocol that tunnels Layer 2 frames over a Layer 3 network. This encapsulation adds new headers (VXLAN, UDP, IP), increasing the original frame size by 50 bytes or more. If the underlying network (the data center interconnect) uses the standard Ethernet Maximum Transmission Unit (MTU) of 1500 bytes, these larger encapsulated packets will exceed the MTU and must be fragmented. Packet fragmentation and reassembly introduce significant processing overhead and latency, leading to the observed slow performance. Enabling jumbo frames increases the MTU of the network devices in the path (e.g., to 9000 bytes), allowing the larger VXLAN packets to be transmitted without fragmentation, thereby resolving the performance degradation.

A. 802.1Q tagging is a protocol for creating Virtual LANs (VLANs) and does not address the fundamental issue of packet fragmentation caused by VXLAN's large encapsulation overhead.

B. Spanning tree is a Layer 2 loop-prevention protocol. It is unrelated to packet size, MTU, or performance issues caused by fragmentation in an overlay network.

C. Link aggregation combines multiple physical links to increase bandwidth and provide redundancy but does not change the MTU, so fragmentation will still occur on the aggregated link.

1. Cisco Systems

Inc. (2023). VXLAN Overview: Cisco Nexus 9000 Series Switches

F55123. "The encapsulation adds a total of 50 bytes to the original frame... To prevent fragmentation in the transport network

it is recommended to enable jumbo frames in the core to accommodate the larger packet size." (See section: "VXLAN Encapsulation and Packet Format").

2. VMware

Inc. (2023). vSphere Networking Documentation. "When you use VXLAN

the virtual machine traffic is encapsulated

and the packet size increases. If the MTU is not set correctly

the virtual machine traffic might be dropped... To avoid performance problems with VXLAN

set the MTU size on the vSphere Distributed Switch and the physical network infrastructure to 1600 or higher." (See section: "System Requirements for VXLAN").

3. IETF RFC 7348. (August 2014). Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. Section 5

"Packet Format

" details the 8-byte VXLAN header

which

when added to the outer UDP (8 bytes) and IP (20 bytes) headers

confirms the source of the packet size increase that necessitates a larger MTU.