Skip to content

ansible: bound silently dead ssh sessions - #316

Merged
berendt merged 1 commit into
mainfrom
ssh-server-alive
Oct 3, 2026
Merged

berendt merged 1 commit into
mainfrom
ssh-server-alive

Conversation

@ideaship

Copy link
Copy Markdown
Contributor

ssh_args enables connection multiplexing but no server-alive probes. When the transport under an ansible session dies without the client being told, the play waits until the kernel's TCP keepalive notices, which is 7200 s by default in the osism-ansible container.

It bites when bootstrap's package tasks upgrade systemd, or a library its daemons link against, so that the upgrade re-executes systemd and restarts systemd-networkd while the task is running. One such update is enough: nodes installed from an image built minutes before the update was published are affected too, and the likelihood grows with the image's age. Killing the wedged ssh -o ControlMaster clients in osism-ansible unblocks the play immediately, and the tasks complete normally.

This PR adds -o ServerAliveInterval=30 -o ServerAliveCountMax=10. The client then drops a session whose peer has not answered for 5 minutes. That is what the manual recovery does, but bounded and automatic.

  • sshd answers the probes at protocol level even while a command runs silently, so long quiet tasks (tempest, large upgrades) are not affected. Only a dead transport, or a host whose sshd has not answered for 5 minutes, is dropped.
  • Reboots are not affected. The reboot playbook fires shutdown asynchronously and reconnects with fresh connections (wait_for_connection or ansible.builtin.reboot), so no session has to survive a reboot.
  • The values are generous (5 minutes rather than, for example, 45 s) to leave margin for loaded hosts and lossy links.
  • Operators can still override ssh_args in environments/ansible.cfg, which the inventory reconciler merges over this file.

🤖 Generated with Claude Code

ssh_args enables connection multiplexing but no server-alive probes.
When the transport under an ansible session dies without the client
being told, the play waits until the kernel's TCP keepalive notices,
which is 7200 s by default in the osism-ansible container.

It bites when bootstrap's package tasks upgrade systemd and the
libraries its daemons link against, so that the upgrade re-executes
systemd and restarts systemd-networkd while the task is running. That
happens whenever the package source carries an update of systemd, or of
a library its daemons link against, that the nodes do not have yet. One
such update is enough: nodes installed from an image built minutes
before the update was published are affected too, and the likelihood
grows with the image's age. Killing the wedged `ssh -o ControlMaster`
clients in osism-ansible unblocks the play immediately, and the tasks
complete normally.

Add `-o ServerAliveInterval=30 -o ServerAliveCountMax=10`. The client
then drops a session whose peer has not answered for 5 minutes, which
is what the manual recovery does, but bounded and automatic.

- sshd answers the probes at protocol level even while a command runs
  silently, so long quiet tasks (tempest, large upgrades) are not
  affected. Only a dead transport, or a host whose sshd has not
  answered for 5 minutes, is dropped.
- Reboots are not affected. The reboot playbook fires shutdown
  asynchronously and reconnects with fresh connections
  (wait_for_connection or ansible.builtin.reboot), so no session has
  to survive a reboot.
- The values are generous (5 minutes rather than, for example, 45 s)
  to leave margin for loaded hosts and lossy links.
- Operators can still override ssh_args in environments/ansible.cfg,
  which the inventory reconciler merges over this file.

Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship ideaship self-assigned this Sep 30, 2026
@ideaship
ideaship marked this pull request as ready for review September 30, 2026 13:41
@ideaship
ideaship requested a review from berendt October 3, 2026 10:31
@berendt
berendt merged commit 5f96583 into main Oct 3, 2026
3 checks passed
@berendt
berendt deleted the ssh-server-alive branch October 3, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants