Details changed as allways

So the customer (local admin for that site) called on a saturday night and complained to me about a VM that no one was able to login via ssh.

The services on that vm were still up and working as expected and a login via the KVM of the virtualizer was no problem.

What followed were basic checks like:

  • sshd config
  • iptables
  • routes
  • tcpdump

And there it got interesting for a second.
The problem only existed if you connected to the site via VPN.
So I verified the routes and everything were fine.

Even tcpdump on the vm showed the pakets arriving at the system, but never leaving it.

Turns out the customer had a 172.30.0.0/16 network assinged to the VPN clients and routes accordingly at the gateway level.
When I first checked the routes on the problematic VM I missread one of the lines.

The VM had a route for 172.30.0.0/16 but via the docker0 interface.

Someone had installed docker on this VM and by chance it had chosen the same network as VPN Clients,
and since the VPN Clients were routed and not NATed the pakets entered the VM and dissapeared into the void behind the docker0 interface.

Since the local admin might never be able to get a hold of the infrastructure I recommended to NAT the VPN Clients, and told him to expect more fun like this unless he finds a solution so that no one randomly installs something somewhere.