[KAJdev]

Part of CON-1833.

Deploy and invoke registered Flash tasks using endpoint or pod runtime credentials. Deployment resolves task resources under deployer credentials, and workers invoke tasks by name with JSON arguments. Recursive calls, cancellation cleanup, task-only apps, and per-call deployment output are supported.

Requires RunPod#6686, ai-api#1176, and sdk-runtime#4. Deployed workers require app/environment/resource/build identity; dev retains its current source-based execution.

[donovanclarke]

approved under standing grant [approve-clean-reviews]

No critical or high findings. The SDK side of the CON-1833 contract lines up with RunPod#6686 (x-runpod-endpoint-id / x-runpod-pod-id + Bearer, exactly one source header, flashTaskCreate selection set within runtimeFields, token placed in RUNPOD_TASK_TOKEN, the four identity vars set on the task pod), with sdk-runtime#4 (/submit accepts only args/kwargs/timeout, the runner never sets RUNPOD_RUNTIME_KIND itself so dev task pods stay on PodTarget, the bootstrap apt-installs manifest systemDependencies and puts the artifact env first on sys.path), and with ai-api#1176's header constants. resolve_mounts returns bindings in spec.mounts order, so both zips in deploy.py are aligned, and resolve_registry_auth(None) short-circuits without an API call. One medium on release sequencing.

SEVERITY - medium
IMPACT - finalizeFlashArtifactUpload { … resourceSpec } plus the resourceSpec.tasks gate make every rp flash deploy, task-less apps included, fail against today's production schema (RunPod#6686 is still open) after the full vendor and upload; worker sentinel calls likewise need ai-api#1176 live.
SOLUTION - release only after #6686 and ai-api#1176 are deployed, or select and require resourceSpec only when the manifest declares tasks so task-less deploys keep working against either backend.

Non-blocking (4)

SEVERITY - low
IMPACT - task env keys with RUNPOD_/FLASH_ prefixes (or PYTHONPATH/PYTHONHOME/LD_PRELOAD) and mounts under /root/.runpod, /runpod-volume, /proc, /sys, /dev pass dev and the full vendor+upload, then fail at finalize with BAD_USER_INPUT (taskPolicy.ts); only endpoints get the local reserved-env check.
SOLUTION - mirror the reserved-env and reserved-path checks for tasks in build_manifest, before build_environment runs.

SEVERITY - low
IMPACT - _deployed_endpoint_input now raises on a user-supplied RUNPOD_API_KEY; worker code that used the previously injected key for wider SDK calls has no explicit opt-in path, and the error does not name the offending key.
SOLUTION - allow explicit user values (the silent injection was the bug), or at least name the key in the error and call out the break in the release notes.

SEVERITY - low
IMPACT - _validate_timeout accepts 0 but flashTaskCreate rejects executionTimeoutSeconds <= 0, so DeployedTaskExecution(..., timeout=0) fails at create; latent today, since no public caller passes a timeout.
SOLUTION - require a positive timeout for deployed executions, or map 0 to None.

SEVERITY - low
IMPACT - _prepare_task_resources and reconcile_endpoints each build their own VolumeResolver, so a volume shared by a task and an endpoint is listed and placement-solved twice per deploy.
SOLUTION - build one resolver in deploy_app and pass it to both.

Unaddressed (0)

Round 1: no prior findings from this reviewer. The four CodeQL / code-quality threads on 8c59d5b are resolved and outdated, and both fixes are in the diff (_TaskTarget base replaces the PodTarget subclassing; asyncio.wait_for replaces the bare await).

Review metadata

review 3 · fable · e49e739

RAN - full diff + 19 changed files read | handles, app, spec, volume, registry, invoker, context, images, secret, events, build, dev, shim, graphql cross-checked at head | RunPod#6686 contract cross-checked (tasks.ts, taskPolicy.ts, taskProvision.ts, taskBootstrap.ts, runtimeFlashAuth.ts, schema diff) | sdk-runtime#4 (bootstrap, task/deployed.py, runner, executor) and ai-api#1176 header constants cross-checked | reviewThreads via GraphQL (4 resolved, outdated) | CI 10/10 success at head
NOT RUN - pytest locally (no checkout) | end-to-end deploy against a #6686 stage
RELIED ON - CI green at e49e739 | bot-thread fixes verified by reading the diff

[Copilot]

🔵 Needs a closer look

The authentication boundary needs coordinated backend integration review, and reserved-field validation still permits partially applied deployments.

1 open issue

• Validate reserved fields before build activation · New

What changed in this PR

This PR enables deployed Flash tasks to run by name with JSON arguments and runtime-scoped credentials, removing automatic developer-key injection from deployed endpoints while preserving source-based development execution.

Changes:

• Adds deployed task invocation, public Task stubs, and shared execution cleanup.
• Resolves task resources during deployment and verifies backend registration.
• Adds runtime identity checks, deployment coverage, and task-aware CLI output.

File	Description
tests/​test_cli/​test_deploy_command.py	Tests task-only deployment output.
tests/​test_apps/​test_dispatch.py	Supplies deployed identity in dispatch tests.
tests/​test_apps/​test_deployed_tasks.py	Tests credentials, task lifecycle, and registration.
tests/​test_apps/​test_deploy.py	Updates backend response fixtures and formatting.
runpod/​rp_cli/​main.py	Displays all deployed resources.
runpod/​rp_cli/​console.py	Labels registered per-call tasks.
runpod/​apps/​utils/​network.py	Selects runtime-scoped authentication headers.
runpod/​apps/​tasks.py	Shares task transport and adds deployed execution.
runpod/​apps/​targets.py	Adds JSON-only deployed task targets.
runpod/​apps/​stubs.py	Introduces the public task client.
runpod/​apps/​handles.py	Adjusts worker self-call routing.
runpod/​apps/​deploy.py	Prepares task resources and validates registration.
runpod/​apps/​context.py	Validates deployed worker identity.
runpod/​apps/​app.py	Routes deployed tasks to named targets.
runpod/​apps/​api.py	Adds task creation and termination operations.
runpod/​apps/​__init__.py	Exports Task from the apps package.
runpod/​api/​mutations/​apps.py	Requests registered resource definitions.
runpod/​api/​graphql.py	Supports additional authentication headers.
runpod/​__init__.py	Exposes the top-level Task import.

_{Review effort: Balanced}

[donovanclarke]

approved under standing grant [approve-clean-reviews]

No critical or high findings. The rework since e49e739 closes every round-1 item and both human threads: the task reserved-env and reserved-mount checks now run in build_manifest and mirror the backend's isReservedFlashTaskEnv (trim().toUpperCase(), RUNPOD_/FLASH_ prefixes, PYTHONPATH/PYTHONHOME/LD_PRELOAD) and its reserved path list exactly; the endpoint reserved-field check runs before vendoring and before any API call (Copilot's thread); DeployedTaskExecution rejects a zero timeout; one VolumeResolver is shared by _prepare_task_resources and reconcile_endpoints; finalize_artifact_upload selects resourceSpec only for manifests that declare tasks, so task-less deploys work against either backend; validate_runtime_target fails cross-scope stub calls before any request, and PodTarget enforces the same JSON contract as DeployedTaskTarget (deanq's two threads). The sdk-runtime executor on main already dispatches on serialization_format, so dev task pods accept the new JSON payload, and rp flash dev workers carry no identity vars, so deployed_identity() stays None there. One medium: the PR's own test is why CI is red.

SEVERITY - medium
IMPACT - test_api_json_and_text_responses_share_transport[target0-*] fails in CI (2 failed on 3.14, rest of the matrix fail-fast cancelled, validation red): LiveTarget("dev") is built at collection time and its QueueClient captures the original _headers function (targets.py:614), so patch("runpod.apps.targets._headers") never reaches QueueClient.http; the real _headers() falls through to _api_key(), which raises without RUNPOD_API_KEY, and the test passes only where runpod.api_key is populated from a local config.
SOLUTION - patch runpod.apps.targets._api_key or monkeypatch.setenv("RUNPOD_API_KEY", ...), or build the LiveTarget inside the patch context; the CodeQL and Analyze reds are the org IP allow list, not this PR.

Non-blocking (3)

SEVERITY - low
IMPACT - release coupling narrowed from round 1 but still live for the SDK release: RunPod#6686 and ai-api#1176 merged 2026-10-28 and are in no release yet (latest v1.753.0 and v1.291.0 predate them), sdk-runtime#4 is still open; against today's prod a task-app deploy fails at finalize after the upload, deployed worker sentinel calls need ai-api#1176 live, and a flashTaskCreate pod without the runtime would hold the caller up to READY_TIMEOUT (600s) before terminating.
SOLUTION - keep the CHANGELOG gate as written and hold the PyPI release until all three are deployed.

SEVERITY - low
IMPACT - a queue/api env= carrying FLASH_APP_NAME, FLASH_ENVIRONMENT, FLASH_RESOURCE_NAME or RUNPOD_BUILD_ID is silently overwritten by _deployed_endpoint_input (the identity keys follow the spread), unlike every other reserved key and unlike any FLASH_* key on a task, so the user's value vanishes with no error.
SOLUTION - add the four identity keys to the _validate_endpoint_env reserved set.

SEVERITY - low
IMPACT - _validate_task_payload reports "deployed tasks accept only JSON args and kwargs" on the local and dev PodTarget path too, serialization.py's module docstring still says args cross the wire as base64 cloudpickle, and build_manifest keeps an excluded_packages kwarg no caller passes any more.
SOLUTION - drop "deployed" from the message, update the docstring, remove the dead kwarg.

Unaddressed (0)

Round 1 had no inline threads from this reviewer; its medium and four lows were body-level and are closed above. All seven threads on the PR are resolved with their fixes in the diff.

Review metadata

review 6 · fable · f1f5df0

RAN - full diff + 23 changed files read at head | targets, deploy, tasks, api, handles, stubs, context, app, serialization, protocol, volume, spec, build, invoker, job, shim, secret, dev, graphql cross-checked | reviewThreads via GraphQL (7 resolved) | CI logs for test (3.14), codeql, Analyze (python), validation | RunPod main taskPolicy.ts reserved env and path rules | sdk-runtime main executor and runner, #4 head task/deployed.py | RunPod#6686, ai-api#1176, sdk-runtime#4 merge and release state
NOT RUN - pytest locally (no checkout) | end-to-end deploy against a #6686 stage
RELIED ON - CI log trace for the test-failure root cause | round-1 cross-repo contract verification (taskProvision, runtimeFlashAuth, bootstrap)

[donovanclarke]

approved under standing grant [approve-clean-reviews]

No critical, high or medium findings. 341706e closes the round-2 medium and two of its three lows; the third (release coupling) is carried below.

• medium (transport-parity test red in CI): fixed. test_api_json_and_text_responses_share_transport now patches target._client._headers, the instance attribute QueueClient.__init__ stores (targets.py:385) and that QueueClient.http calls (targets.py:504), so the patch reaches both LiveTarget (targets.py:614) and SentinelTarget (targets.py:530) without the account key. LiveTarget._sync_source returns early when no source is attached, so its separate module-level _headers() call (targets.py:657) never runs in this test.
• low (identity keys silently overwritten): fixed. FLASH_APP_NAME, FLASH_ENVIRONMENT, FLASH_RESOURCE_NAME and RUNPOD_BUILD_ID are now in the _validate_endpoint_env reserved set. That check runs in build_manifest before vendoring and again in _deployed_endpoint_input, and the parametrized pre-vendor test plus the updated test_deploy_environment_is_used_for_nested_calls cover it.
• low (message, docstring, dead kwarg): fixed. The message now reads "tasks accept only JSON args and kwargs", the serialization.py docstring describes the JSON contract, and build_manifest no longer takes excluded_packages. Both callers (build_artifact, deploy_app) set excludedPackages after vendoring, and test_build_artifact_preserves_excluded_packages and test_taskless_deployment_accepts_finalize_without_task_policy cover it.

Non-blocking (1)

SEVERITY - low
IMPACT - carried from round 2: the SDK release still depends on RunPod#6686, ai-api#1176 and sdk-runtime#4 being deployed; otherwise task-app deploys fail at finalize and worker sentinel calls 401.
SOLUTION - keep the CHANGELOG gate and hold the PyPI release until all three are live.

Unaddressed (0)

All seven threads on the PR are resolved, and none of them is this reviewer's. Round 2's medium and three lows were in the review body and are closed out above, apart from the carried release-coupling low.

Review metadata

review 7 · opus · 341706e

RAN - rework diff since f1f5df0 read | targets.py at head (QueueClient, SentinelTarget, LiveTarget init/_sync_source/request) | dev.py reserved-env callers | reviewThreads via GraphQL (7 resolved)
NOT RUN - pytest locally (no checkout) | CI at 341706e (all 8 checks still pending when this review was posted)
RELIED ON - round-2 cross-repo contract verification | static trace of the patched _headers path

[donovanclarke]

approved under standing grant [approve-clean-reviews]

No critical or high findings. The code rework since f1f5df0 (341706e) holds up at full scope: the four identity keys (FLASH_APP_NAME, FLASH_ENVIRONMENT, FLASH_RESOURCE_NAME, RUNPOD_BUILD_ID) sit in the _validate_endpoint_env reserved set, which runs in build_manifest before vendoring and again in _deployed_endpoint_input; deploy.py is the only reference to excluded_packages in the repo and both callers (build_artifact, deploy_app) set excludedPackages after vendoring; dev.py injects only RUNPOD_API_KEY on dev workers and never passes through the endpoint validator, so rp flash dev is unaffected; the transport-parity test patches the _headers instance attribute QueueClient stores and calls, and the full test matrix plus e2e passed at 341706e. One medium on the new commit, 7bff1ea.

SEVERITY - medium
IMPACT - 7bff1ea deletes the repo's CodeQL job and its weekly schedule to route around an org-level IP allow list that broke every CodeQL upload on 2026-10-29 (codeql and the GitHub-managed Analyze (python) both passed at e49e739 on 10-21 and the 10-26 scheduled run; both fail at f1f5df0 and 341706e with "the runpod organization has an IP allow list enabled"), yet it fixes nothing: Analyze (python) comes from dynamic/github-code-quality/codeql, not ci.yml, so it stays red and the PR stays UNSTABLE, and main has no required status checks, so neither red ever blocked merge; what is lost is the repo's own PR and weekly SAST, in a SEC ticket PR, with no mention in the description or CHANGELOG.
SOLUTION - drop 7bff1ea from this PR; fix the cause at the org level (allow GitHub Actions / installed-app traffic through the IP allow list) or, if retiring the in-repo CodeQL job is a deliberate call, land it as its own ci: PR with security sign-off so it is reviewable on its own.

Non-blocking (1)

SEVERITY - low
IMPACT - carried from rounds 2 and 3, still live: RunPod#6686 (merged 2026-10-28) is not in any RunPod release (latest v1.753.0, 2026-10-27), ai-api#1176 (merged 2026-10-28) is not in any ai-api release (latest v1.291.0, 2026-10-26), and sdk-runtime#4 is still open; against today's prod a task-app deploy fails at finalize after the upload and deployed worker sentinel calls 401.
SOLUTION - keep the CHANGELOG gate and hold the PyPI release until all three are deployed.

Unaddressed (0)

All seven threads on the PR are resolved and none is this reviewer's. Round 3's single carried low (release coupling) is restated above with current release state; its medium and the other two lows were closed in 341706e.

Review metadata

review 9 · fable · 7bff1ea

RAN - rework diff since f1f5df0 (341706e + 7bff1ea) read | ci.yml and .github/workflows at head | check-runs and workflow runs at e49e739, f1f5df0, 341706e, 7bff1ea, main push and schedule runs | codeql and Analyze (python) job logs at 341706e | branch protection and rulesets for main | code search for build_manifest / excluded_packages callers | dev.py reserved-key injection at head | reviewThreads via GraphQL (7 resolved) | RunPod and ai-api release lists, RunPod#6686, ai-api#1176, sdk-runtime#4 state
NOT RUN - pytest locally (no checkout) | CI at 7bff1ea (test matrix and e2e in progress at post time)
RELIED ON - CI green for the test matrix and e2e at 341706e | rounds 1 and 2 cross-repo contract verification