Skip to content

Add a guide for serving one VIP from several hosts - #389

Draft
JackDoan wants to merge 2 commits into
mainfrom
vip-ecmp-guide
Draft

JackDoan wants to merge 2 commits into
mainfrom
vip-ecmp-guide

Conversation

@JackDoan

@JackDoan JackDoan commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Adds a guide, Serving one address from several hosts (docs/guides/service-vips/), for putting a service behind a virtual IP that several Nebula hosts answer for. It needs no Nebula changes; it combines features that already exist:

  • Each backend's certificate carries the VIP as an unsafe network, and the VIP sits on the backend's loopback interface. No IP forwarding or NAT.
  • Each client has an ECMP unsafe_routes entry for the VIP with one gateway per backend, so each client picks the backend for each connection. Replies come straight back from the backend, like direct server return without a director.
  • Backends see the client's real overlay address and certificate, so group/host/CA firewall rules work as usual.

The examples use IPv6. A section lists the IPv4 equivalents.

Field test

Ran on three real Linux hosts with nebula master (5739273f): one client and two backends. Every row below was run on both IPv4 (IPv4 overlay and underlay) and IPv6 (IPv6 overlay and underlay). Extra rounds covered IPv4 with v1 certificates and one dual-stack backend pair serving an IPv4 VIP and an IPv6 VIP together.

Check Result
Spread over 100 connections 50/50 (IPv4), 49/51 (IPv6). A model of the port hash predicted the backend for 40/40 connections.
Reply source The backend sends from the VIP out the Nebula interface, and the client receives replies from the VIP.
weight: 2 / 1 over 300 connections 201/99 (IPv4), 190/110 (IPv6)
Backend rule without local_cidr Requests never reach the backend's tun; only connections hashing to it time out
Backend cert without the VIP Only connections hashing to it time out (24/40 and 16/40, every one predicted)
Client prefix that contains the VIP Nebula exits: entry 1.route in tun.unsafe_routes is contained within the configured vpn networks
Stop Nebula on a backend 30/30 new connections go to the other backend immediately
Backend comes back A keep-alive connection that failed over is reset once the tunnel is back
Connection opened while one backend's tunnel is still coming up (backend just restarted) SYN goes to the other backend; data goes to the hashed one, which sends a RST; 3/3 per family
Cold-started client Its first packet starts handshakes to every gateway at once, so the reset window above is about 1 ms

The last two rows corrected my first draft, which said a client cold start triggers the reset.

docusaurus build passes with no broken links or anchors.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Kk4zXXpTWPoEBaSRdtvVr

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for docs-nebula ready!

Name Link
🔨 Latest commit 53001b0
🔍 Latest deploy log https://github.com/DefinedNet/nebula-docs/actions/runs/40367211106
😎 Deploy Preview Url https://0219fc36.docs-nebula.pages.dev
😎 Deploy Preview Alias https://vip-ecmp-guide.docs-nebula.pages.dev

@jasikpark

Copy link
Copy Markdown
Collaborator

Nice guide. I checked its claims against nebula master and they match the code: ECMP hashes ports only (routing/balance.go), the client falls back to the first other gateway in list order that has a tunnel and starts handshakes to all of them when none do (inside.go), and the client's outbound firewall drops packets to a gateway whose cert lacks the VIP (ErrInvalidRemoteIP).

I put my suggestions in #392, stacked on this branch. Merge it in if it looks right:

  • Mobile clients can't use this. Mobile Nebula only takes a single string via, so a via: list fails with via was not a string.
  • local_cidr is required, except when firewall.default_local_cidr_any: true.
  • Fixed-source-port clients stick to one backend, since the hash uses only the ports (some UDP clients, for example).
  • Cross-links: this guide is now linked from the tun ECMP section, the firewall local_cidr field, and the unsafe_routes guide's tips. Config options in the guide link to their reference sections, per the repo's CLAUDE.md.

Nit, not in #392: sidebar_position: 12 puts this last. It might sit better right after the unsafe_routes guide (8), since that's a prerequisite, but that means renumbering. Your call.

— Caleb + Cache

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants