I have what appears to be a networking issue, but our network guys and VMware haven't been any help. Maybe somebody here has seen this.
Background:
We have 4 ESX 5i clusters totaling about 30 hosts. We moved one of the clusters to a new building and within a day, the hosts started getting disconnected from vCenter. Rebooting fixed the problem for a while and then it started to come back. The hosts drop out one at a time and we lose the ability to communicate with them from vCenter. vCenter can talk to everything else on that subnet and everything on that subnet can talk to vCenter. After looking at the routing, we found that the traceroute changes over time.
The first route looks like
10.5.5.1 (Default router)
10.5.5.251 (MPLS gateway)
192.1.1.1 (Remote firewall)
192.1.1.5 (Destination)
and works fine. After a short time, it changes:
10.5.5.251 (MPLS gateway)
192.1.1.1 (Remote firewall)
192.1.1.5 (Destination)
It still works, but not it's no longer hitting the default router. Then, at some point days or weeks later, it changes again:
10.5.5.254 (Firewall)
10.5.5.251 (MPLS gateway)
192.1.1.1 (Remote firewall)
192.1.1.5 (Destination)
At that point, we lose connectivity. I can ping and hit any port on vCenter from the ESX host, but not vice versa. Rebooting fixes it.
Has anybody ever seen anything like this? Cisco points to VMware as the problem and VMware points to Cisco. Maybe the community can be of more help.
Thanks
-mS