This article addresses intermittent timeout errors that may be seen while running Traceroute tests from the Google Cloud nodes.
Explanation
Traceroute may work sporadically because TTL does not expire on some hops. This occurs due to the way we do encapsulation on the Google Network. The inconsistencies we see are actually dependent on the cloud region and network tier.
Traceroute to external IP addresses
For internal reasons, Google Cloud increases the TTL counter of packets that traverse next-hops in Google's network. Tools like traceroute and mtr might provide incomplete results because the TTL doesn't expire on some of the hops. Hops that are inside and outside of Google's network might be hidden in these circumstances:
- When you send packets from Compute Engine instances to external IP addresses, including external IP addresses of other Google Cloud resources or destinations on the internet.
- When you send packets to the external IP address associated with a Compute Engine instance or other Google Cloud resource.
The number of hidden hops varies based on the instance's Network Service Tiers, region, and other factors. If there are only a few hops, it's possible for all of them to be hidden. Missing hops from a traceroute or mtr the result doesn't mean that outbound traffic is dropped.
There is no workaround for this behavior. You must take it into account if you configure third-party monitoring that connects to an external IP address associated with a VM.
Important: Probe loss statistics are a component of traceroute tests, but care must be taken when analyzing test results. traceroute and mtr by default utilize ICMP-based probing. ICMP probe response generation is typically rate-limited (or disabled) in routers that reside in the network path of your probing and can result in missing probe responses. When this behavior occurs, you may see probe loss in intermediate routing hops, but this should not reflect end-to-end performance. If looking for packet loss, the only hop that generally matters is the destination hop.