Skip to content

feat: add per-invocation executor isolation - #735

Draft
zhongkechen wants to merge 13 commits into
fix/lmi-invocation-lifecyclefrom
feature/invocation-executor-isolation
Draft

zhongkechen wants to merge 13 commits into
fix/lmi-invocation-lifecyclefrom
feature/invocation-executor-isolation

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Issue Link, if available

Partial fix for #726, stacked on #733. Arbitrarily nested bounded-pool starvation remains unresolved.

Description

Concurrent LMI invocations sharing a bounded executor can occupy every worker with roots that wait for queued steps. withInvocationExecutorFactory(...) gives each invocation a fresh executor for its root, steps, child contexts, and coordinators. Existing withExecutorService(...) executors remain caller-owned and shared.

The updated stack preserves main's handler-owned Start/End lifecycle. Both plain and plugin-enabled roots use the selected executor. The runtime caller shuts down an invocation-owned executor after normal root/End completion, so cleanup cannot wait for its own worker. Termination uses the remaining cleanup budget; the additional reserve applies only to owned executors. User code that ignores interruption can outlive that budget.

DurableConfig.toBuilder() preserves the factory and resolved plugin configuration. The local test runner therefore obtains a new executor on replay/resume without rediscovering providers. The factory must return a fresh, non-null, usable executor; overlapping reuse of an executor identity and configuring both executor modes are rejected.

The LMI fixed2 fixture uses per-invocation fixed(2) pools. The shared nested2 fixture remains available for diagnosis and is not presented as solved by invocation isolation.

Validation

  • Full clean Java 25 reactor on the final Java source: 3,036 passed / 34 conditional skips, zero failures/errors across all ten modules.
  • Final Java 17 focused ownership, cancellation, lifecycle, configuration-copy, and replay controls: 96 passed, including all three opt-in LMI lifecycle regressions. The full unfiltered Spotless check passes.
  • Updated Python LMI harness: 180 passed, including delayed cohort evidence, original healthy-request acceptance, post-cleanup request-failure auditing, and bounded post-stop residual-work settlement, and lifecycle acceptance for every case request, and final immutable request/JVM/cohort identity checks. Original deadline/recovery/progress acceptance is unchanged. The final parent refresh changes only these Python harness/test files and their README; Java sources are identical to the validated tree.
  • Current head 11b17912 includes upstream main 67bad28c and declared base fix: apply invocation deadlines to SDK waits and cleanup #733 242b88fc; all original PR history is retained.

The focused public controls verify that plugin Start/End run on the owned worker, the caller waits for End before shutdown, rejected roots close their owned executor with or without plugins, and replay/resume preserves the configured factory. Inherited observation gates recognize the real budgeted waits; their lifecycle, MDC, and admission assertions are unchanged.

No new deployed LMI acceptance run is claimed for this implementation. This PR remains Draft, targeting #733's fix/lmi-invocation-lifecycle branch; merge the stack in order before retargeting this PR to main.

Checklist

  • Shared executors remain caller-owned.
  • Invocation-owned executors are shut down by the runtime caller with the remaining cleanup budget.
  • Factory identity and configuration-copy behavior are covered.
  • Draft state and the partial-mitigation boundary are retained.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@zhongkechen
zhongkechen added this pull request to stack #732 September 25, 2026 17:28
@zhongkechen
zhongkechen force-pushed the feature/invocation-executor-isolation branch from 5433d65 to 703c7e0 Compare September 29, 2026 17:49
@zhongkechen
zhongkechen force-pushed the feature/invocation-executor-isolation branch from 703c7e0 to 241c152 Compare September 29, 2026 22:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant