Skip to content

Support inference servers other than Triton, starting with Nereid - #88

Open
kondratyevd wants to merge 20 commits into
mainfrom
inference-server-abstraction
Open

Support inference servers other than Triton, starting with Nereid#88
kondratyevd wants to merge 20 commits into
mainfrom
inference-server-abstraction

Conversation

@kondratyevd

@kondratyevd kondratyevd commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Makes the chart server-agnostic and tests it with Nereid as a second inference server.

Inference servers

The chart no longer assumes Triton: a server that speaks the KServe v2 gRPC protocol is configured through inferenceServer.image, command/args, ports and probes. values/values-nereid.yaml runs Nereid with Triton's port numbers, so Envoy, the scaling metric, scaleFromZero and the ServiceMonitor work unchanged; like Triton's command line, its nereid.yaml is written in args. Chart additions:

  • The readiness probe accepts httpGet in place of the exec command (the Nereid image has no curl).
  • New configMap model repository type, used by the CI test.

Breaking

Before Now
Values key triton: inferenceServer:
Resources <release>-triton <release>-inference-server
Component label triton inference-server
Envoy clusters triton_grpc_service, triton_admission inference_server_grpc_service, inference_server_admission
  • Rename the top-level triton: key in values files; helm install fails on a leftover one.
  • After upgrading, restart the bundled Prometheus (<release>-prometheus-server): it loads its scrape configuration only at startup and otherwise drops the renamed pods, which the scaling metric reads as no load. An external Prometheus that selects pods by the component label needs the new value.
  • A custom Envoy configuration must use the new cluster names: the default scaling metric selects inference_server_grpc_service.
  • The Deployment is renamed, so an upgrade replaces it (brief interruption).

Nereid status

Also included

  • Envoy pods restart when their configuration changes (checksum annotation); Envoy reads it only at startup.
  • The preStop hook of the inference server never ran its wait. A terminating pod now keeps serving for 40 seconds while Envoy stops routing to it, then gets SIGTERM and 20 seconds to exit, instead of failing the requests that still reach it.

🤖 Generated with Claude Code

@kondratyevd
kondratyevd force-pushed the inference-server-abstraction branch 2 times, most recently from 6d29461 to ae2f4bb Compare September 5, 2026 20:56
@kondratyevd
kondratyevd force-pushed the inference-server-abstraction branch from 21d1eef to 14e1d10 Compare October 8, 2026 19:43
kondratyevd and others added 18 commits October 15, 2026 16:57
@kondratyevd @claude
Rename Triton-specific chart identifiers to inference-server
316a01f 
Preparation for supporting inference servers other than Triton: every
chart identifier that named Triton specifically is renamed to a
server-neutral one. Triton remains the default and only supported
server; this commit changes no rendered behaviour.

  values key    triton:                    -> inferenceServer:
  templates     templates/triton/          -> templates/inference-server/
  helper        supersonic.tritonName      -> supersonic.inferenceServerName
  resources     <release>-triton           -> <release>-inference-server
  label         component: triton          -> component: inference-server
  container     triton-server              -> inference-server
  envoy cluster triton_grpc_service        -> inference_server_grpc_service
                triton_admission           -> inference_server_admission
  sidecar env   GPU_DEPLOYMENT             -> INFERENCE_SERVER_DEPLOYMENT

Verified by rendering the chart against all 8 values files plus chart
defaults, before and after: applying the rename map above to the
baseline reproduces the new output exactly, compared as an unordered
set of manifests (the template directory rename reorders the stream).

The rename is breaking for existing values files, so a stale top-level
`triton:` key now fails the render with a message naming the new key --
Helm would otherwise ignore it and silently deploy chart defaults. Note
that `helm lint` reports template `fail` calls as INFO, so this surfaces
on `helm install`/`helm template` rather than in the lint job.

Upgrading renames the Deployment, so Helm replaces rather than patches
it: expect a brief interruption, not an immutable-selector error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Make the chart server-agnostic and add the Nereid server type
6d396d8 
Adds `inferenceServer.type` (triton | nereid) and the pieces a non-Triton
server needs, without changing anything for existing Triton deployments.

Nereid speaks the same KServe v2 gRPC protocol as Triton, and its
bind_addr is configurable, so the rendered nereid.yaml keeps Triton's
port numbers. Envoy, the Service, the ServiceMonitor, the Prometheus
scrape config and the default scaling metric therefore need no
per-server branching at all. What is left differing is the image, the
model configuration and the probe endpoints:

- Probes now accept any one of `command` (exec shorthand), `exec`,
  `httpGet` or `tcpSocket` instead of hardcoding exec for readiness and
  httpGet for startup. Nereid's image ships no curl, so it needs httpGet
  against the KServe v2 health endpoints.

  Setting two handlers is an error rather than a precedence rule. Helm
  merges values with the chart defaults, so adding `httpGet` does not
  remove the inherited `command`; picking a winner would silently probe
  the wrong endpoint, so switching handler must be explicit
  (`command: null`).

- `command`/`args` are omitted when empty, so a server configured by
  file can use its image's entrypoint.

- `inferenceServer.nereid.config` renders to a ConfigMap mounted at
  nereid.yaml via subPath, so it does not shadow the model folders that
  share Nereid's working directory.

- An unknown `type` fails the render rather than producing a
  half-configured Deployment.

yaml-to-schema.py now drops genson's inferred "required" lists. genson
only ever sees values.yaml, so "required" was a copy of the defaults --
which Helm merges into every release, making the constraint unable to
catch a missing key while rejecting a values file that deliberately
clears one. Nulling a probe handler was blocked by exactly this.

values/values-nereid.yaml is a complete worked example. Its image
reference is a placeholder: nereid-server has no releases or tags yet
and its GHCR package is unpublished.

Verified: all 8 existing values files plus chart defaults render
semantically identically to before this commit (parsed and compared as
Kubernetes objects, so probe key reordering is not mistaken for change).
helm lint passes on all 9 values files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Add CI coverage for the Nereid inference server
4159ce4 
A kind-based job that installs the chart with type: nereid and drives
one inference through Envoy.

The client is a pip-installed tritonclient rather than the one in the
Triton SDK image: the claim under test is that an unmodified KServe v2
client works against Nereid, and building it from Triton's own proto
stubs is what makes that a cross-implementation check rather than a
self-consistency one. Tensor names and shapes come from ModelMetadata
instead of being hardcoded, so the metadata path is covered too.
perf_analyzer is not used -- it needs the model-config and statistics
RPCs, which Nereid does not implement.

The model is nereid-server's own ml-backends/model3 example (a 10 KB
TorchScript MLP) vendored into a ConfigMap, so the job needs no model
volume, no CVMFS and no network fetch. It is served by the Rust torch
backend, which is in the image's default feature set and needs no
writable model directory -- unlike the Python backend, which builds a
venv/ inside each model folder.

Mounting it required a `configMap` modelRepository storageType, which
also gives small models and test deployments a first-class option.
mountPropagation is omitted for it, since the API server rejects that
field on a configMap volume.

The job is gated on the Nereid image actually being published: the
repository has no tags yet and its GHCR package is unpublished, so the
gate skips with a GitHub notice rather than failing for a reason
unrelated to this chart. Remove the gate once a release exists.

Verified: all 8 existing values files plus chart defaults still render
semantically identically; helm lint passes on all 11 values files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Document inference server selection and generalize Triton wording
a820ec7 
Adds a "Select an Inference Server" section to the configuration guide
comparing the two types, with a worked Nereid snippet and the two
Triton-only caveats: Nereid exports no nv_gpu_* metrics (so the GPU
dashboard panels and metricsCollector do not apply), and its Python
backend needs a writable model folder, so a read-only volume serves
only .pt/ONNX/TensorFlow models.

The rest of the guide keeps saying "Triton" only where it really means
Triton -- image selection, the model repository layout, the
tritonserver command line. Everywhere the chart is now server-agnostic
the wording follows. Also documents the new configMap model repository
option, and fixes the section numbering, which had two sections
numbered 1.

Dashboard panels retitled from "Number of Triton Servers"; the queries
are untouched, verified by diffing the rendered dashboard ConfigMap.

helm/supersonic/README.md is the CI-maintained copy of the root README,
refreshed here so the tree is self-consistent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Drop the OpenTelemetry repo from the Nereid CI job
9a44fcd 
Chart.yaml no longer declares an opentelemetry-collector dependency
after #87, so `helm dependency build` has nothing to resolve from that
repository.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Adopt the shared CI job-wait and diagnostics helpers in the Nereid job
5933fd9 
#86 replaced `kubectl wait --for=condition=complete` with
scripts/wait-for-job.sh, which polls for Complete OR Failed so a failed
Job reports immediately instead of idling until the timeout, and
extracted failure dumps into the k8s-diagnostics composite action. This
job was written against the older conventions and was the only one left
hand-rolling both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Point the Nereid notes at the upstream RepositoryIndex PR
fe902ad 
ngpaladi/nereid-server#20 implements the RepositoryIndex RPC that
scaleFromZero triggers on, so it is no longer an assumption with nothing
behind it. Its proto is the same vendored `inference.GRPCInferenceService`,
so the path Envoy routes and the Lua filter matches is identical for both
server types; the chart needs no change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kondratyevd @claude
Condense comments, the Nereid example and its CI job
9fd888e 
Shorten the inline comments and failure messages added by this branch.
values/values-nereid.yaml keeps only what differs from the chart
defaults, and the CI job installs that file directly (model source
overridden with --set), replacing tests/values-nereid-ci.yaml.

No change to rendered manifests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Condense the inference server documentation
76ca316 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Restart Envoy and Nereid pods when their configuration changes
c1f8f5e 
Envoy reads its configuration, Lua filter and admission script only at
startup, and nereid.yaml is a subPath mount that never updates in place,
so a ConfigMap change did not reach running pods. A checksum annotation
on the pod template now rolls them.

This also makes the rename in this PR safe to upgrade into: without
scaleFromZero the Envoy pod template was otherwise unchanged, so Envoy
kept routing to the deleted <release>-triton Service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Fail on a probe without a handler and on Nereid without models
e7e9478 
A probe whose default handler was nulled without a replacement rendered
timings only, which the API server rejects with a generic error. And
`type: nereid` on its own rendered a Triton pod next to a nereid.yaml
with no models, which Nereid does not start with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Fix the inference server preStop hook
89c6be4 
The hook passed its command as separate arguments, so `sh -c` ran a bare
`touch`, failed, and the 60-second wait never happened: a pod removed by
scale-down got SIGTERM at once and requests still routed to it failed.

A terminating pod now keeps serving for 60 seconds while Envoy drops it
from the upstream set. The hook does not depend on the server type, so
the documentation no longer points at a Triton flag for draining.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Gate the Nereid CI job once and wait on rollouts
b6b3d96 
The image check moves to its own job, so the deployment job carries a
single job-level condition instead of one per step. Readiness is waited
on with `kubectl rollout status`: `kubectl wait pod -l ...` right after
`helm upgrade --install` fails outright if the pod does not exist yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Keep a values key named "required" in the generated schema
4005409 
drop_required removed every "required" entry, including the schema of a
property with that name; only genson's lists are meant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Simplify the Nereid support
192b55e 
- The Deployment is main's template again plus what Nereid needs: the
  nereid.yaml mount, an httpGet alternative for the readiness probe and
  the configMap model repository. The probe helper, the type helpers and
  their validations are gone; the API server already rejects a probe
  with no or two handlers.
- The Nereid example overrides command and args with empty lists instead
  of null, so the schema generator and the "required" lists in
  values.schema.json stay as they are on main.
- The leftover `triton:` check is one line in NOTES.txt.
- inferenceServer.nereid.configPath is dropped: Nereid reads nereid.yaml
  from its fixed working directory.

Rendered output for Triton is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Shorten the Nereid CI job and its test
274d11f 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Drop wording-only changes from internal code, CI and docs structure
26d4797 
Comments in the Lua filter, the admission script and the metric
template, and step names in the workflows that deploy Triton, keep
main's text; only identifiers and selectors change there. Docs sections
1 and 2 keep main's structure with the parameter rename and one Nereid
paragraph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Configure Nereid through the generic server parameters
9723599 
The chart no longer knows about Nereid: inferenceServer.type, the
inferenceServer.nereid values and the nereid.yaml ConfigMap are removed.
Like Triton's command line, Nereid's configuration now lives in `args`:
the example writes nereid.yaml to the server's working directory and
then starts it. Config changes therefore roll the pods without a
checksum annotation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd
kondratyevd force-pushed the inference-server-abstraction branch from 4bb3b12 to 9723599 Compare October 15, 2026 20:57
kondratyevd and others added 2 commits October 15, 2026 17:03
@kondratyevd @claude
Shorten the preStop wait to 40 seconds
b4e9281 
The 60-second wait used up the whole termination grace period, so the
server was killed instead of exiting. 40 seconds still outlasts the
time Envoy may keep routing to a terminating pod (CoreDNS caches
answers for up to 30 s and Envoy re-resolves every 5 s) and leaves the
server 20 seconds to exit after SIGTERM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kondratyevd @claude
Name Nvidia Triton as an example inference server
6aabd7e 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kondratyevd