Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/ci/Dockerfile.trixie
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libc6-dev \
make \
pandoc \
python3 \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /work
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ Gitlink even when the two Spawn Manager commits are not direct ancestors.
Abuild integration and GitHub overlay branches as frozen history.
- Replaced Codelabs synchronization with local release-tag integrity checks.

### Fixed

- Catchable manager shutdown now waits for its executor to reap request
descendants before exiting, including after the leader identity has been
retired. A stalled reap acknowledgement fail-stops within a fixed bound.
Shell and structured requests retain the existing wire format and results.

## [0.2.0] - 2026-09-27

### Added
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ tests: $(POSIX_TEST) $(PROTOCOL_FAILURE_MANAGER) spawn_tests spawn_manager
@$(OBJDIR)/spawn_manager 47 $(OBJDIR)/spawn_manager_min_spawn &
@$(OBJDIR)/test_runner
@tests/check_signal_shutdown.sh "$(CURDIR)"
@python3 tests/check_signal_reaping.py "$(abspath $(OBJDIR))/spawn_manager" 1
@tests/check_adaflags.sh "$(CURDIR)"
@tests/check_install.sh "$(CURDIR)"

Expand Down
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ Building Spawn Manager requires:
- GNU Make.

The tests additionally require
[Ahven](https://www.stronglytyped.org/ahven/), Bash
[Ahven](https://www.stronglytyped.org/ahven/), Bash, Python 3
and standard Linux process utilities. Coverage needs `lcov` and `genhtml`.
Building the optional HTML documentation requires
[Pandoc](https://pandoc.org/).
Expand All @@ -119,6 +119,12 @@ its process and log ordering remains deterministic:
make tests
```

The signal integration check uses its own fallback subreaper. It detects
children abandoned by a terminated manager even when a container init would
otherwise collect them. Shell and structured requests exercise both SIGINT
and SIGTERM; direct C fault tests also cover delayed and stalled reaping.
A fixture self-test checks recovery across newly adopted generations.

Run the comparative execution benchmarks:

```sh
Expand Down
17 changes: 14 additions & 3 deletions doc/lifecycle-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,20 @@ pre-exec `Close_Descriptors` stage instead of proceeding to `execve`. The Work
Queue retains the missing unlimited-limit fault injection and boundedness
evidence.

The attached manager signal handler invokes only the synchronized C
containment hook and immediate OS exit. The pool owns socket-path cleanup after
the manager has terminated.
The attached manager signal handler invokes the synchronized C shutdown hook,
then exits. The hook closes the fork gate and kills the still-pinned group.
It then waits for the normal executor to acknowledge complete leader and
descendant reaping; it does not introduce a competing `waitpid` owner. A
mutex-protected in-progress flag spans the interval after the numeric group
identity is retired but before descendant collection finishes.

The reap acknowledgement has a fixed one-second wait, below the pool's
three-second manager wait. If the executor cannot finish, the manager
fail-stops with status 125 instead of waiting indefinitely or reporting normal
shutdown. Executor-side containment failure does not wait for itself. A
fail-stop cannot promise that uncollectable children have been reaped; their
eventual collection belongs to the next subreaper or host init. The pool owns
socket-path cleanup after the manager has terminated.

## Failure classification

Expand Down
62 changes: 56 additions & 6 deletions src/spawn-posix.c
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
#define DESCRIPTOR_CEILING_UNINITIALIZED -2
#define POLL_SLICE_MS 10
#define TERMINATION_GRACE_MS 100
#define SHUTDOWN_REAP_WAIT_MS 1000

/*
* ERROR_FD is the sole descriptor intentionally kept across child setup. The
Expand All @@ -68,6 +69,12 @@ extern char **environ;
static pthread_mutex_t group_state_lock = PTHREAD_MUTEX_INITIALIZER;
static pid_t active_group;
static int shutdown_requested;
/*
* Protected by group_state_lock. Unlike the process-group identity this stays
* set after leader retirement until descendant reaping has also completed.
* The interrupt task waits on this acknowledgement, never on a reusable PGID.
*/
static int request_in_progress;

/*
* The manager opens every persistent descriptor before its request loop.
Expand Down Expand Up @@ -680,13 +687,11 @@ static int terminate_group(
}

/*
* GNAT interrupt-task hook: prevent a later fork, kill the currently published
* request group and preserve errno. The Ada handler exits the manager after
* this returns; normal execution remains the sole reaper meanwhile.
* Prevent a later fork and kill the still-pinned request group. Do not wait:
* executor-side fail-stop also uses this hook and must not await itself.
*/
void spawn_posix_terminate_current(void)
static void request_shutdown(void)
{
int saved_errno = errno;
pid_t group;

lock_group_state();
Expand All @@ -695,6 +700,48 @@ void spawn_posix_terminate_current(void)
if (group > 0)
(void)kill(-group, SIGKILL);
unlock_group_state();
}

/*
* Let the sole executor finish reaping without holding its ownership mutex.
* A fixed one-second deadline stays below the pool's three-second wait; a
* stalled executor fail-stops rather than deadlocking an interrupt-server task.
*/
static void wait_for_request_reaping(void)
{
int64_t started = monotonic_milliseconds();
int64_t deadline;

if (started < 0)
_exit(CONTAINMENT_FAILURE_EXIT);
deadline = started > INT64_MAX - SHUTDOWN_REAP_WAIT_MS
? INT64_MAX : started + SHUTDOWN_REAP_WAIT_MS;
for (;;) {
int pending;
struct timespec pause = { 0, POLL_SLICE_MS * 1000 * 1000 };

lock_group_state();
pending = request_in_progress;
unlock_group_state();
if (!pending)
return;
if (remaining_milliseconds(deadline) == 0)
_exit(CONTAINMENT_FAILURE_EXIT);
(void)nanosleep(&pause, NULL);
}
}

/*
* GNAT interrupt-task hook: prevent another fork, kill the published group and
* await the executor's final reap before the Ada handler exits the manager.
* Waiting uses no Ada runtime or competing waitpid owner and preserves errno.
*/
void spawn_posix_terminate_current(void)
{
int saved_errno = errno;

request_shutdown();
wait_for_request_reaping();
errno = saved_errno;
}

Expand Down Expand Up @@ -820,6 +867,7 @@ int spawn_posix_execute(
if (setpgid(pid, pid) < 0 && errno != EACCES && errno != ESRCH)
group_setup_error = errno;
active_group = pid;
request_in_progress = 1;
unlock_group_state();
if (set_manager_signal_mask(
SIG_SETMASK, &original_signal_mask, NULL) < 0) {
Expand Down Expand Up @@ -978,7 +1026,7 @@ int spawn_posix_execute(
}
if (containment_failed) {
/* Never resume the manager after losing request-group ownership. */
spawn_posix_terminate_current();
request_shutdown();
_exit(CONTAINMENT_FAILURE_EXIT);
}
if (manager_signals_blocked
Expand All @@ -990,6 +1038,8 @@ int spawn_posix_execute(
unlock_group_state();
_exit(CONTAINMENT_FAILURE_EXIT);
}
/* All leader and descendant waits precede this shutdown acknowledgement. */
request_in_progress = 0;
unlock_group_state();
if (pidfd >= 0)
(void)close(pidfd);
Expand Down
5 changes: 3 additions & 2 deletions src/spawn-posix.h
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,13 @@ int spawn_posix_execute(
struct spawn_posix_result *result);

/*
* Kill the active request group, if any.
* Kill the active request group, if any, and await its executor's final reap.
*
* This function is called by GNAT's SIGINT/SIGTERM interrupt-server task. It
* synchronizes through a process-local mutex with fork publication and exact
* leader reaping; it is not a raw POSIX signal handler. The normal execute path
* remains the sole reaper.
* remains the sole reaper, including descendants after leader retirement.
* A stalled executor fail-stops the manager after a one-second reap wait.
*/
void spawn_posix_terminate_current(void);

Expand Down
Loading
Loading