A /8 route that swallowed cloud traffic

Walkthrough

Click through the incident. Use Next and Back to follow the investigation.

Overview

A group of sites reached the internet through gateways that also carried VPN traffic. To make VPN return traffic work, I had added a route for 172.0.0.0/8 that pointed into the VPN tunnel.

Problem statement

A cloud-hosted business application and some consumer services failed at several sites at once, while most other browsing worked. Partial failures like this look random, which is why they take longer to diagnose than a clean outage.

Investigation

1. Look for what the failures share. The same symptom at several unrelated sites ruled out a single access link or ISP. It pointed at something every site had in common: the gateway configuration.
2. Compare a working and a failing destination. Both should have been reachable, so I asked the gateway how it would route each one.

ip route get <working-destination>
ip route get <failing-destination>

The working destination left through the WAN gateway. The failing one resolved to the VPN tunnel interface.

1. Find the route that matched.

ip route show | grep '^172\.'

That showed the broad 172.0.0.0/8 route.

Root cause

Only 172.16.0.0/12 is private address space. A /8 also covers every public address from 172.0.0.0 to 172.15.255.255 and from 172.32.0.0 to 172.255.255.255, including ranges used by large cloud and consumer services. Routers choose the longest matching prefix, so any destination in that range went into the VPN tunnel instead of out to the internet, and never arrived. I had scoped the route too widely.

Resolution

I added a specific host route for each affected destination through the WAN gateway. A /32 is the longest possible prefix, so it wins over the /8.

ip route add <destination>/32 via <wan-gateway>

Verification

I re-ran ip route get for the failing destination and confirmed it now left through the WAN gateway, then tested the application and the affected websites from clients at each site.

Lessons learned

• Scope VPN routes to the exact prefixes that need the tunnel. A /8 added for convenience is a future outage.
• Check every new route against public address space before it goes live.
• Write the incident up with a timeline and root cause so clients can follow it without being network engineers.

Skills demonstrated

Confidentiality: all identifying information, credentials, customer data, internal addresses and proprietary configurations have been removed or replaced with illustrative examples.