quench

Post-training for tool-using agents, on AWS, with reproducibility built in.

quench takes raw agent traces — what your tool-using model did, step by step — and turns them into fine-tuned models you can deploy. Supervised fine-tuning, preference optimization, and verifiable-reward RL, all from one library, all reproducible, all runnable locally or on SageMaker.

  raw agent traces  →  rewards  →  training data  →  fine-tuned model
       (your data)     (rule + verifiable        (LoRA/QLoRA)    (on S3)
                        + LLM-judge)              (SFT/DPO/GRPO)

If you've ever wanted to fine-tune an agent and found that the existing tools either don't speak agent traces, don't handle tool calls, don't produce trustworthy synthetic data, or don't reproduce — this is for you.

What you can do with it

The v0.1 alpha is in active development. Most of the pipeline runs end-to-end today; the commands below mark each command as ✅ (shipped) or 🚧 (planned). See Status and roadmap for the full picture, including which WBS ambition items are still pending.

pip install -e ".[dev,train,judge]"

# 1. Validate trajectories + check data quality
quench data validate traces.jsonl                                            # ✅
quench data describe traces.jsonl                                            # ✅
quench data quality-check pairs.jsonl --config recipes/quality-gate.yaml     # ✅ (§12.6 CHES check)

# 2. Build training-ready datasets from trajectories
quench build sft trajectories.jsonl --out datasets/sft.jsonl                 # ✅
quench build preference trajectories.jsonl --out datasets/pairs.jsonl        # ✅

# 3. Filter preference pairs through the §3.12 CHES gate before DPO
quench data preference-check pairs.jsonl --reference-model Qwen/Qwen2.5-0.5B-Instruct   # ✅
quench data preference-filter pairs.jsonl --out pairs.clean.jsonl ...                   # ✅

# 4. Train a small model with LoRA — emits a run manifest (§8.1)
quench train sft --config recipes/01_sft_lora_local/config.yaml              # ✅
quench train dpo --config recipes/02_dpo_lora_local/config.yaml              # ✅ (§5.4; §12.6 gate enforced unless --allow-unfiltered-preferences)
quench train grpo-tools --config ...                                         # 🚧 (§5.5)
quench train sft --config ... --backend sagemaker                            # 🚧 (§7/§14)

# 5. Compare base vs fine-tuned on a held-out slice (§11.1+§11.3 slice)
quench eval compare Qwen/Qwen2.5-0.5B-Instruct runs/latest --dataset eval.jsonl   # ✅

# 6. Calibrate an LLM judge against the bundled golden set (§3.10)
quench rewards calibrate-judge \
    --examples goldens/calibration/tool_use_v1/examples.jsonl \
    --rubric goldens/calibration/tool_use_v1/rubric.md \
    --rubric-id tool_use_v1                                                   # ✅

# 7. Inspect provenance + ship a release card
quench graph trace eval:compare-<id>                                         # ✅ (§8.7)
quench graph render -o provenance.mmd                                        # ✅
quench release-card generate runs/latest --human-fields human.yaml           # ✅ (§8.9)

# 8. Synthetic data + Docent bridge — planned
quench data synthesize --skills chat --count 4000 --teacher claude-opus      # 🚧 (§13)
quench trace push-to-docent traces.jsonl --collection my-collection          # 🚧 (§14)

Every shipped command emits structured artifacts. Training runs write a runs/<id>/manifest.json that pins the dataset path + train/val sizes, the recipe config, the base model, the seed, every installed library version, and a host descriptor. The Provenance Graph (§8.7) materialises those manifests + eval reports into a typed, queryable graph without a new on-disk format — quench graph trace walks the ancestry of any node, and quench release-card generate reads the same artifacts into an FMTI-inspired model release card.

What it isn't

Not a SaaS. It's a Python library + CLI. You run it on your machine, your AWS account, or your CI.
Not a full eval framework. It ships minimal eval (held-out validation, one reference benchmark, checkpoint comparison) — enough to trust the models it produces, not enough to replace Inspect AI, lm-eval-harness, or HELM. The companion eval repo is separate.
Not multi-cloud. AWS-first via a thin ComputeBackend abstraction. Local works everywhere. Other backends are a v1.0 question.
Not a Hugging Face replacement. Models live on HF; this library uses HF tokenizers and datasets. It's the post-training layer on top.
Not a generic LLM fine-tuner. Axolotl, TRL, Unsloth, and LLaMA-Factory are excellent at chat-model SFT. quench exists because tool-using agents need different schemas, different rewards, and different evals.

How it differs from the things you know

✓ = shipped in quench today · ◐ = partial in quench · — = not present

	`quench`	Axolotl	TRL	Alignment Handbook	open-instruct
Trajectory-aware schema (tool calls, step-level rewards)	✓	partial	partial	—	partial
Skill catalog + skill annotations (§1.9)	✓	—	—	—	—
LLM-judge with mandatory calibration (§3.9 + §3.10)	◐ Anthropic only	—	—	—	—
Bundled hand-labeled judge calibration set	✓	—	—	—	—
CHES preference-pair gate (§3.12)	✓	—	—	—	—
Quality gate as a recipe-blocking step (§12.6)	✓	—	—	—	—
Provenance graph derived from manifests (§8.7)	✓	—	—	—	partial
Model Release Transparency Card (§8.9)	✓	—	—	—	—
Content-addressed run manifest (§8.1)	✓	—	—	—	partial
Synthetic data generation (Instruct-SkillMix, §13)	🚧 unblocked	—	—	—	—
Verifiable-reward RL (GRPO with tools, §5.5)	🚧	partial	partial	—	✓
AWS SageMaker recipes (§7/§14)	🚧	—	—	—	—
Behavior analytics bridge (Docent, §14)	🚧	—	—	—	—

The differentiation is in the rows that matter for agentic post-training specifically — not chat, not general SFT, but training agents that use tools well.

Quickstart

Status: alpha (v0.1, in active development). The schema, data pipeline, reward primitives, judge calibration, CHES gate, SFT-LoRA + DPO-LoRA recipes, minimal eval, run manifest, provenance graph, and release card are all shipped and tested. The GRPO recipe, the §13 SkillMix synthesis, AWS backend, and Docent bridge are still planned. Items marked 🚧 below are the latter.

Install

pip install -e ".[dev,train]"     # core dev + training stack
pip install -e ".[dev,train,judge]"  # add the Anthropic SDK for §3.10 calibration

End-to-end local pipeline (everything below is shipped)

# 1. Validate trajectories against the schema (§1.7).
quench data validate examples/valid_trajectory.jsonl

# 2. Build an SFT dataset (messages-format, TRL-ready).
quench build sft examples/valid_trajectory.jsonl --out datasets/sft.jsonl

# 3. Fine-tune a small model with LoRA — emits adapter + tokenizer + manifest.
quench train sft --config recipes/01_sft_lora_local/config.yaml

# 4. Compare base vs fine-tuned on a held-out slice.
quench eval compare Qwen/Qwen2.5-0.5B-Instruct runs/latest \
    --dataset datasets/sft.jsonl --max-samples 64

# 5. Inspect what fed this run.
quench graph trace run:$(basename $(readlink runs/latest))

# 6. Ship a release card with auto-populated provenance + human-authored sections.
cat > human.yaml <<'EOF'
model_name: my-agent-v0
intended_use: Research only.
limitations: 0.5B base; smoke fixture.
out_of_scope_use: Do not deploy.
license: Apache-2.0
EOF
quench release-card generate runs/latest --human-fields human.yaml

# 7. Calibrate an LLM judge against the bundled golden set (needs ANTHROPIC_API_KEY).
quench rewards calibrate-judge \
    --examples goldens/calibration/tool_use_v1/examples.jsonl \
    --rubric goldens/calibration/tool_use_v1/rubric.md \
    --rubric-id tool_use_v1

What 🚧 still requires planned work

quench data synthesize --skills chat --count 4000 --teacher claude-opus   # §13 SkillMix
quench train grpo-tools --config ...                                       # §5.5
quench train sft --config ... --backend sagemaker                          # §7 / §14
quench trace push-to-docent traces.jsonl --collection my-collection        # §14

Full quickstart (local) — what the post-v0.1 surface looks like

The fastest path from zero to a fine-tuned model. Uses tiny inputs throughout; the goal is to verify the pipeline, not to train a competitive model.

pip install quench

# Pull a starter dataset (10 tool-use traces, shipped with the package)
quench data fetch starter-traces --out traces.jsonl

# Validate the schema
quench data validate traces.jsonl
# → OK: 10 trajectories, schema_version 1.0.0

# Quality-check (will pass on starter data)
quench data quality-check traces.jsonl

# Build an SFT dataset
quench build sft traces.jsonl --out datasets/sft.jsonl

# Train (uses Qwen2.5-0.5B-Instruct by default; takes ~5 minutes on a laptop)
quench train sft --config recipes/01_sft_lora_local/config.yaml

# Evaluate against the base model
quench eval compare \
  --base Qwen/Qwen2.5-0.5B-Instruct \
  --fine-tuned ./runs/latest \
  --eval ifeval-mini

# Inspect the run manifest
cat ./runs/latest/manifest.json

Quickstart on AWS — when the AWS backend is complete

# Configure AWS once
quench aws configure

# Run the same recipe on SageMaker
quench train sft \
  --config recipes/01_sft_lora_local/config.yaml \
  --backend sagemaker

# Watch the job
quench aws status

# Artifacts land in S3 with the same lineage and manifest format

The same recipe runs locally and on SageMaker. The ComputeBackend abstraction means the recipe doesn't know where it's running.

The shape of the library

src/quench/
├── schemas/        # Trajectory, ToolCall, ChatMessage — see ADR-003, ADR-004
├── data/           # Loaders, validators, redaction, dedup, splits, quality gate
├── rewards/        # Rule, verifiable, tool-use, process, LLM-judge primitives
├── builders/       # SFT, preference, reward-labeled dataset builders
├── training/       # Recipe abstraction; SFT, DPO, GRPO-tools recipes
├── agents/         # Tool registry, executor, agent loop, recorder
├── eval/           # Held-out validation, reference benchmarks, comparison
├── synthesize/     # Instruct-SkillMix; skill library
├── analytics/      # Docent bridge
├── runs/           # Manifest, artifact bundle, lineage graph
├── compute/        # LocalBackend, SageMakerBackend  (ADR-002)
└── cli/            # Typer-based CLI

Concepts you'll meet

Trajectory — one agent attempt at one task. The atomic unit of data. Same schema whether it's an SFT pair, a preference example, or a multi-step agentic rollout. (ADR-003)
Skill catalog + skill annotations — first-class Skill objects with (skill_id, version) identity (ADR-008); SkillTag, SkillPair, MissingSkillTag live in annotations and are excluded from payload_hash so post-hoc skill tagging never invalidates trajectory identity.
Run manifest — self-describing JSON record next to every recipe run; pins dataset path and sizes, recipe config, base model, seed, every installed library version, host descriptor. Lives in runs/<id>/manifest.json.
Provenance graph — typed graph of Dataset → Run → Checkpoint → EvalVerdict materialised at query time from manifests + eval reports (§8.7). No new on-disk format. quench graph trace walks ancestry; ... render emits Mermaid; ... diff compares two runs.
Release card — FMTI-inspired Model Release Transparency Card (§8.9). Auto-populates from the manifest + graph; refuses to finalize human-authored fields (intended use, limitations, out-of-scope) when blank. MD doubles as a HF model card.
Recipe — opinionated, parameterized training pipeline. SFT-LoRA and DPO-LoRA (local backend) shipped; GRPO-tools planned. Each recipe declares its primary metric for §5.8 best-checkpoint selection.
Reward primitive — a composable scoring component. Rule-based, exact-match, format, tool-use shipped; verifiable (sandboxed code execution) and process planned.
LLM-judge — judges a candidate against a rubric and returns {pass, warn, fail} + score
- rationale (§3.9). Calibrated against a hand-labeled set (§3.10) — goldens/calibration/ ships a starter set with 12 examples covering the main judgment edges.
Quality gate — pre-training filter (§12). At v0.1 alpha runs the §12.6 CHES check against preference datasets; §12.1 brevity, §12.2 LLM-judge scan, §12.3 distribution check plug into the same framework. quench train dpo refuses to start on a dataset that hasn't been checked against a configured gate, and refuses to start on one that's blocked, unless --allow-unfiltered-preferences is passed (the bypass is logged; ADR-009).
CHES — Centered Hidden Embedding Similarity for DPO preference pairs (§3.12). Flags pairs at risk of likelihood displacement; uses the DPO reference model for embeddings.
Skill library — versioned named-skill ontology used for synthetic data generation (§13, planned). Pre-built libraries for chat, code, customer support.
ComputeBackend — runs a recipe locally or on SageMaker (ADR-002; SageMaker path planned).
Trace adapter — converts external trace formats (Inspect AI, OpenAI Agents SDK, Anthropic Agent SDK) into the library's schema (v0.2).

Status and roadmap

v0.1 alpha — re-anchored June 2026. The build no longer tracks the W1-W16 calendar in the WBS line-for-line; five of six v0.1 ambition items are now code-complete and the §5.2 SFT-LoRA + §11 minimal-eval slices are shipped. The authoritative status column lives in WBS §6.

v0.1 ambition (per WBS §10)

WBS item	What it is	Status
§1.9 Skill catalog + skill annotations	Schema 1.0.0 → 1.1.0 bump; `Skill`/`SkillTag`/`SkillPair`/`MissingSkillTag`; hash-invariant under skill annotations	done
§3.9 LLM-judge primitive	Judge protocol + provider impls; response caching	partial — Anthropic provider shipped; OpenAI + Bedrock + caching planned with §3.13
§3.10 Judge calibration	`quench rewards calibrate-judge` + golden set `goldens/calibration/tool_use_v1/` (12 hand-labeled examples)	done
§3.12 CHES preference-pair gate	`quench data preference-check / preference-filter`	done
§12.6 CHES wired into quality gate	`quench data quality-check` + `assert_preferences_passed_gate` helper, enforced by the §5.4 DPO recipe	done
§8.7 Provenance Graph	`quench graph trace / render / diff` — derived from manifests + reports at query time	done
§8.9 Model Release Transparency Card	`quench release-card generate` — auto + human-authored sections, MD/JSON/HTML output	done
§13 SkillMix synthetic data	`quench data synthesize`	planned — unblocked now that §3.9 + §1.9 ship

Slices pulled forward as alpha-validation

WBS item	What's shipped	What's still on the calendar
§5.2 SFT-LoRA recipe	LoRA-only over a small HF model via TRL; `recipes/01_sft_lora_local/`; emits an §8.1 manifest	§5.3 QLoRA, §5.7 resume, §5.8 best-checkpoint selection
§5.4 DPO-LoRA recipe	LoRA-only DPO via TRL; `recipes/02_dpo_lora_local/`; §12.6 preference-quality gate enforced by default (bypass via `--allow-unfiltered-preferences`, logged per ADR-009); device-aware precision (CPU/MPS/CUDA); emits an §8.1 manifest	§5.3 QLoRA, §5.7 resume, §5.8 best-checkpoint selection, measured RESULTS run
§11 Minimal eval	§11.1 held-out loss/PPL + §11.3 `quench eval compare` + §11.4 primary metric	§11.2 reference benchmark, §11.5 regression check

Still planned for v0.1

§3.3 verifiable (sandboxed) rewards · §3.5 process rewards · §5.5 GRPO recipe · §12.1 brevity detector · §12.2 LLM-judge quality scan · §12.3 distribution check · §7/§14 AWS SageMaker path · §14 Docent bridge · §8.8 run comparison

v0.2

Skill Intelligence (§18 — extraction, Missing-Skill Profile/STAT, LESS-v0 selector) · Feedback Simulation Sandbox (§19) · §3.13 cost-aware judge router (also adds the OpenAI + Bedrock judge providers) · §13.7 Self-Instruct baseline · §8.10 Open Experiment Registry · Parquet writer · KTO/ORPO recipes · Inspect AI / OpenAI / Anthropic SDK trace adapters · Docker image · PyPI · recorded-tool mode.

Explicitly not planned

Web UI · multi-tenancy · multi-cloud orchestration · full benchmark suite (companion repo) · production inference layer · deep evaluation framework (delegated to AgentEvalOps; see docs/BOUNDARY.md).

Contributing

See CONTRIBUTING.md. The short version: open an issue first for anything non-trivial; PRs need tests; new public APIs need ADRs. The foundational ADRs in docs/adrs/ describe the project's philosophy.

License

Apache 2.0 — see LICENSE.

Acknowledgments

This library stands on:

Hugging Face — TRL, Alignment Handbook, datasets, tokenizers
Axolotl — used under the hood for SFT recipes
Ai2 — open-instruct, the RLVR lineage
Volcano Engine — verl for GRPO
Princeton + Meta — Instruct-SkillMix paper for the synthetic-data approach
Transluce — Docent, the behavior analytics backend
Nathan Lambert's RLHF Book — the conceptual foundation

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
analysis		analysis
docs		docs
evals		evals
examples		examples
goldens/calibration/tool_use_v1		goldens/calibration/tool_use_v1
infra		infra
recipes		recipes
src/quench		src/quench
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
README_reanchored.md		README_reanchored.md
RESULTS.md		RESULTS.md
SECURITY.md		SECURITY.md
pyproject.toml		pyproject.toml

Navigation Menu

Repository navigation

About

Repository files navigation

quench

What you can do with it

What it isn't

How it differs from the things you know

Quickstart

Install

End-to-end local pipeline (everything below is shipped)

What 🚧 still requires planned work

Full quickstart (local) — what the post-v0.1 surface looks like

Quickstart on AWS — when the AWS backend is complete

The shape of the library

Concepts you'll meet

Status and roadmap

v0.1 ambition (per WBS §10)

Slices pulled forward as alpha-validation

Still planned for v0.1

v0.2

Explicitly not planned

Contributing

License

Acknowledgments

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

quench

What you can do with it

What it isn't

How it differs from the things you know

Quickstart

Install

End-to-end local pipeline (everything below is shipped)

What 🚧 still requires planned work

Full quickstart (local) — what the post-v0.1 surface looks like

Quickstart on AWS — when the AWS backend is complete

The shape of the library

Concepts you'll meet

Status and roadmap

v0.1 ambition (per WBS §10)

Slices pulled forward as alpha-validation

Still planned for v0.1

v0.2

Explicitly not planned

Contributing

License

Acknowledgments

About

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages