Repository navigation
ansible: bound silently dead ssh sessions - #316
Merged
Merged
Conversation
ssh_args enables connection multiplexing but no server-alive probes. When the transport under an ansible session dies without the client being told, the play waits until the kernel's TCP keepalive notices, which is 7200 s by default in the osism-ansible container. It bites when bootstrap's package tasks upgrade systemd and the libraries its daemons link against, so that the upgrade re-executes systemd and restarts systemd-networkd while the task is running. That happens whenever the package source carries an update of systemd, or of a library its daemons link against, that the nodes do not have yet. One such update is enough: nodes installed from an image built minutes before the update was published are affected too, and the likelihood grows with the image's age. Killing the wedged `ssh -o ControlMaster` clients in osism-ansible unblocks the play immediately, and the tasks complete normally. Add `-o ServerAliveInterval=30 -o ServerAliveCountMax=10`. The client then drops a session whose peer has not answered for 5 minutes, which is what the manual recovery does, but bounded and automatic. - sshd answers the probes at protocol level even while a command runs silently, so long quiet tasks (tempest, large upgrades) are not affected. Only a dead transport, or a host whose sshd has not answered for 5 minutes, is dropped. - Reboots are not affected. The reboot playbook fires shutdown asynchronously and reconnects with fresh connections (wait_for_connection or ansible.builtin.reboot), so no session has to survive a reboot. - The values are generous (5 minutes rather than, for example, 45 s) to leave margin for loaded hosts and lossy links. - Operators can still override ssh_args in environments/ansible.cfg, which the inventory reconciler merges over this file. Assisted-by: Claude:claude-opus-5.5 Signed-off-by: Roger Luethi <luethi@osism.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ssh_argsenables connection multiplexing but no server-alive probes. When the transport under an ansible session dies without the client being told, the play waits until the kernel's TCP keepalive notices, which is 7200 s by default in the osism-ansible container.It bites when bootstrap's package tasks upgrade systemd, or a library its daemons link against, so that the upgrade re-executes systemd and restarts systemd-networkd while the task is running. One such update is enough: nodes installed from an image built minutes before the update was published are affected too, and the likelihood grows with the image's age. Killing the wedged
ssh -o ControlMasterclients in osism-ansible unblocks the play immediately, and the tasks complete normally.This PR adds
-o ServerAliveInterval=30 -o ServerAliveCountMax=10. The client then drops a session whose peer has not answered for 5 minutes. That is what the manual recovery does, but bounded and automatic.wait_for_connectionoransible.builtin.reboot), so no session has to survive a reboot.ssh_argsinenvironments/ansible.cfg, which the inventory reconciler merges over this file.🤖 Generated with Claude Code