VPN Failover Between VPS Nodes

TL;DR: Automatic multi-hop failover can introduce critical security risks if a tunnel failure silently reverts egress traffic to an unintended public IP. In a fail-closed architecture, the default response to a tunnel outage is a complete network blackhole. Recovery requires controlled self-healing or deliberate operator-initiated migration.

This article is Part 6 of a 7-part series on building a multi-hop, fail-closed VPS VPN chain with Ansible, LUKS encrypted vaults, Telegram monitoring, and zero-log policy routing.

1. Part 1: VPS Baseline: Install, Harden, and Disable Logs
2. Part 2: Encrypted Vault for VPN Secrets and Residual Logs
3. Part 3: Telegram Alerts Without Leaking the Exit IP
4. Part 4: Ansible Playbook for Edge Hop Installation
5. Part 5: VPN Chain and End-to-End Encryption Without Persisted Logs
6. Part 6: VPN Failover Between VPS Nodes
7. Part 7: Operational Tasks: Heal, Add, Remove, and Replace Hops

If a multi-hop tunnel dies and system scripts automatically reroute fleet traffic out an intermediate WAN interface, your egress IP silently changes. Downstream firewalls or IP allowlists break, and traffic is exposed on an unapproved egress path.

As detailed in Part 5, the system operates on a strict fail-closed paradigm:

Tunnel (tun1) Active   -> Policy Table 200 via-route active (Traffic exits vpn-2)
Tunnel (tun1) Down     -> Only metric 1000 blackhole default remains (Traffic drops)
Fleet Virtual Machines -> Zero internet connectivity, zero WAN leaks

Automated recovery scripts must re-establish the exact declared path, never silently select an arbitrary exit node without operator authorization.

Network flaps happen. A watchdog systemd timer running on vpn-1 checks every minute whether tun1 is active and policy routes are correctly populated.

The watchdog verifies and restores missing policy routing elements:

1. Checks if ip rule show contains iif tun0 lookup 200.
2. Verifies that routing table 200 has both default via 10.200.0.1 dev tun1 and blackhole default metric 1000.
3. Re-applies iptables MASQUERADE rules on tun1.
4. Restarts openvpn-client@uplink-vpn2 if the client process died.

Important: Never restart the primary OpenVPN server container on vpn-1 during routine health checks. Restarting the server terminates all connected gateway clients simultaneously.

If vpn-2 suffers a prolonged outage and you explicitly decide to allow temporary egress directly from vpn-1:

# Temporarily suspend policy routing to the chain
rm -f /etc/vpn-uplink-enabled
ip rule del iif tun0 lookup 200

# Route tun0 traffic into vpn-1's default eth0 NAT table
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Fleet VMs will now exit through vpn-1 (203.0.113.10). Notify operators of the expected temporary IP shift.

Once vpn-2 returns to service, re-enable the multi-hop path:

touch /etc/vpn-uplink-enabled
systemctl restart openvpn-client@uplink-vpn2
ip rule add iif tun0 lookup 200 priority 100

Verify table 200 states:

ip route show table 200

Confirm that the via 10.200.0.1 route is active alongside the metric 1000 blackhole.

When vpn-2 is unrecoverable, introduce replacement exit node vpn-3 (203.0.113.30) configured with custom SSH port 52322:

              [vpn-2: DEAD]
                    x
gateway -> vpn-1 --+--> vpn-3 (NEW EXIT) -> Internet

1. Probe Target Host: Perform 10 consecutive TCP connection probes to 203.0.113.30:52322.
2. Provision Baseline: Add vpn-3 to Ansible hosts.yaml with custom SSH port 52322 and unique OpenVPN subnet 10.300.0.0/24.
3. Execute Playbooks: Run baseline, vault, security, and VPN playbooks on vpn-3.
4. Update Entry Hop Client Config: Point vpn-1’s uplink client config to vpn-3’s IP address and install the new client credentials.
5. Restart Tunnel: Restart openvpn-client@uplink on vpn-1.
6. Verify Fleet Egress: Run curl -4 ifconfig.me from a fleet VM and verify it returns vpn-3’s public IP (203.0.113.30).
7. Verify Management Path: Confirm curl -4 ifconfig.me on vpn-1 still returns vpn-1’s IP (203.0.113.10).

Use this policy matrix when implementing health-check scripts:

Next in the series: Part 7: Operational Tasks: Heal, Add, Remove, and Replace Hops - The complete Day-2 operational runbook for maintaining, scaling, and repairing a multi-hop VPN chain.