Operational Tasks: Heal, Add, Remove, and Replace Hops

Oct 13, 2026 5 min read

TL;DR: Day-2 operation of a multi-hop VPS VPN chain requires clear, repeatable procedures for diagnostics, path healing, node expansion, node removal, and server replacement. This runbook provides step-by-step procedures to maintain chain integrity and verify fail-closed routing.

This article is Part 7 of a 7-part series on building a multi-hop, fail-closed VPS VPN chain with Ansible, LUKS encrypted vaults, Telegram monitoring, and zero-log policy routing.

Initial Status Diagnostics

Before modifying configuration, inspect host state across all nodes over non-standard SSH ports (28422, 41922):

# Check entry hop (vpn-1) status
ssh -p 28422 [email protected] \
  'hostname; cryptsetup status vpn-vault; ip -br link; ip rule show; ip route show table 200'

# Check exit hop (vpn-2) status
ssh -p 41922 [email protected] \
  'hostname; cryptsetup status vpn-vault; systemctl is-active docker; ss -ulnp | grep 51821'

Diagnostic Decision Matrix:

Observed Symptom	Underlying Cause	Immediate Remediation
SSH Connection Timeout on port `28422`	Network outage, provider issue, or host crash	Access host out-of-band console
SSH active, but LUKS vault `/vault` unmounted	Server rebooted; key file or crypttab failed	Unlock volume via `cryptsetup open`
`tun1` down, routing table `200` has blackhole only	Uplink connection to exit node dropped	Restart uplink client; check exit node
`tun1` up, but table `200` missing `via` route	Policy routing rule drift	Re-apply `ip route add default via ...`
Fleet VM exits via wrong public IP	Routing leak or manual fallback left active	Re-enforce policy routing and blackhole

Procedure 1: Path Healing (No Re-provisioning)

If a host is running but traffic is blocked, follow this healing order:

Verify Vault Mounts: If closed, mount the LUKS container:

cryptsetup open /var/lib/vpn-vault.img vpn-vault
mount -a

Restart Uplink Tunnel: If tun1 is down, restart the client daemon:
```
systemctl restart openvpn-client@uplink-vpn2
```

Re-apply Policy Routing: Re-inject declared policy rules:

ip rule add iif tun0 lookup 200 priority 100 2>/dev/null || true
ip route replace default via 10.200.0.1 dev tun1 metric 0 table 200
ip route replace blackhole default metric 1000 table 200

Verify Exit IP: Confirm egress IP from a fleet VM:
```
curl -4 ifconfig.me
```

Note: Do not re-run Ansible installation playbooks to debug minor routing issues. Re-creating Docker containers drops active client connections.

Procedure 2: Adding a New Node to the Chain

To insert a new intermediate or exit hop (vpn-3 at 203.0.113.30 on port 52322):

TCP Probe Test: Verify 10 sequential TCP connection probes to port 52322.
Bootstrap SSH: Install operator public keys, configure ListenStream=52322 in systemd ssh.socket, and disable port 22.
Inventory Update: Add vpn-3 to Ansible hosts.yaml with a unique internal subnet (10.300.0.0/24).

Execute Playbooks:

ansible-playbook -i hosts.yaml playbooks/vpn-node.yaml -e target_vpn=vpn-3
ansible-playbook -i hosts.yaml playbooks/vpn-node-vault.yaml -e target_vpn=vpn-3
ansible-playbook -i hosts.yaml playbooks/vpn-node-security.yaml -e target_vpn=vpn-3 -e deadman_enabled=false
ansible-playbook -i hosts.yaml playbooks/vpn-node-traffic.yaml -e target_vpn=vpn-3
ansible-playbook -i hosts.yaml playbooks/vpn-node-vpn.yaml -e target_vpn=vpn-3

Update Upstream Route: Point vpn-2’s uplink client config to vpn-3 and activate policy routing.
Update Topology Manifest: Document the new path in /etc/vps-chain/chains.conf.

Procedure 3: Removing a Node

When decommissioning an exit node (vpn-2):

Shift Egress Traffic: Re-point vpn-1’s uplink tunnel to another valid exit node or temporarily revert to vpn-1 egress.
Verify Egress Path: Confirm from fleet VMs that traffic no longer flows to vpn-2.
Terminate Uplink: Stop the uplink client daemon on vpn-1:
```
systemctl stop openvpn-client@uplink-vpn2
```
Scrub Credentials: Remove vpn-2 certificates and keys from vpn-1’s vault.
Decommission Node: Cancel the VPS instance with the hosting provider. (Because disk logging was disabled, no persistent client IP logs remain on the disk).

Procedure 4: Replacing a Node (Add + Cut + Remove)

To replace vpn-2 with vpn-3 without downtime:

Step 1: Provision vpn-3 in parallel with vpn-2
Step 2: Configure uplink client on vpn-1 to connect to vpn-3
Step 3: Verify fleet VM exit IP equals vpn-3's public IP
Step 4: Decommission vpn-2

Never delete an existing node before the replacement node has passed verification tests.

Post-Change Verification Checklist

Before declaring any maintenance task complete, execute this verification checklist:

/etc/vps-chain/chains.conf matches live kernel routing tables.
Running curl -4 ifconfig.me from a fleet VM returns the public IP of the final exit hop.
Running curl -4 ifconfig.me on vpn-1 returns vpn-1’s own public IP.
Disabling the uplink tunnel activates the blackhole default metric 1000 route, causing client requests to drop without WAN leakage.
Port 22 is confirmed closed (ss -tlnp | grep ':22 ' returns empty). Custom SSH ports (28422, 41922) answer correctly.
LUKS vault (/vault) mounts automatically upon host reboot.
Telegram alert notification is received upon opening an SSH session.
Re-running Ansible playbooks returns changed=0, failed=0.

Series Conclusion

This concludes the 7-part series on Building a VPS VPN Chain. By combining non-standard SSH ports, volatile in-memory logging, LUKS2 encrypted vaults, Telegram security notifications, idempotent Ansible playbooks, and fail-closed policy routing, operators can deploy resilient multi-hop egress infrastructure without sacrificing security or privacy.

Series Navigation: Building a VPS VPN Chain