Skip to content
mlnomadpyPublic

About

End-to-end JAX/Flax NNX LLM training, evaluation, inference, checkpointing, and TPU/GPU execution harness

Resources

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

flaxchat

Native JAX/Flax NNX training, recovery and inference tools for TPU language models and the GOAT multilingual encoder family.

GOAT multilingual encoder

GOAT model family

GOAT mmBERT YAT Base MLM 10B is a 22-layer, 768-dimensional encoder with GOAT-input attention and YATGLU feed-forward blocks. It was initialized from scratch and pretrained with masked language modeling, including complete coverage of a 10,038,104,948 nonpadding-token prepared corpus after an earlier MLM stage. The published checkpoint is 379079; it has no contrastive or downstream fine-tuning.

Load the native model directly from Hugging Face through FlaxChat:

pip install -e ".[encoder]"
from flaxchat.hub import from_pretrained

model = from_pretrained(
    "mlnomad/goat-mmbert-yat-base-mlm-10b",
    revision="a0279613f6a2cca746847fbba1672a6f0b407ea4",
)
# model.encode(token_ids): token representations
# model.pool(token_ids): pooled, normalized representations

The model card includes tokenization and batching examples. Model execution and numerical acceptance use the documented accelerator runtime; metadata-only tests do not qualify inference or training.

Benchmark GOAT MLM-only base Previous YAT embedding v1, fine-tuned mmBERT-base paper, embedding fine-tuned
English MTEB, 41 tasks / seven-category mean 36.84 52.78 53.9
Multilingual MTEB, 131 tasks / eight-category mean 31.77 48.21 54.1
CoIR, ten-task mean 9.91 43.45 42.2
MIRACL hard-negative, 18-language mean 1.82 46.37 —

Scores use the 0–100 display scale. GOAT measurements use the complete pinned task inventories, length512, nonpadding mean pooling including special tokens, FP32 L2 normalization and no prompts. Previous-model results are historical published receipts; mmBERT paper embedding results follow fine-tuning. These columns have different training histories. See full results and protocol.

Contrastive pilot status — October28,2026: eight-chip parent import and four hard-negative loss/gradient checks passed. The real-data recovery qualification stopped before optimizer updates because its development sample was too small to cover every language. The local sample-budget correction passed model-free admission; physical revalidation and production contrastive training remain pending. Protected MLM checkpoints and failure evidence are retained, and the attempt's compute resources were independently verified removed. See current model and training status.

Latest reconciled audit: all open issues, current evidence and tool priorities. Earlier dated findings below remain historical where superseded.

Reproducibility and checkpoint contracts are documented in docs/REPRODUCIBILITY.md and docs/CHECKPOINT_FORMAT.md. Release gates and compatibility are documented in docs/RELEASES.md. Exact accelerator measurements and their limitations are in docs/RESULTS.md. The current engineering risks and prioritized backlog are in the September 30 system audit.

The October 1 follow-up audit records the remaining recovery, admission and release blockers in the new local fixes. The complete open-issue checklist identifies the remaining acceptance for all 29 issues. For continued YAT embedding representation training, use the training runbook, tool inventory, and versioned training skill. The September 3 audit is historical. The Spot TPU validation report records the prepared-data training fixes, interruption recovery, and complete TPU test inventory. The GCP multi-host campaign records physical worker acceptance, FSDP recovery, scale coverage, and remaining limits. The follow-up audit fixes cover update rollback, artifact identity, streaming shards, CI and bounded Spot supervision. The training-readiness follow-up records tied-head initialization, checkpoint overhead, four-host launcher fixes, physical qualification, and posted billing evidence. Supported Python and accelerator combinations are in docs/COMPATIBILITY.md.

A minimal, end-to-end LLM training harness for Google Cloud TPU pods, built on JAX/Flax NNX.

JAX/Flax NNX adaptation of nanochat, with TPU-oriented execution and speculative decoding.

pixi install
pixi run test
pixi run test-e2e                 # offline end-to-end acceptance smoke
python -m scripts.run_tinystories # pinned TinyStories end-to-end pipeline

What is this?

flaxchat is the complete LLM pipeline running natively on TPUs and GPUs with automatic data parallelism:

Stage Script Description
Tokenizer scripts/tok_train.py Train BPE, or select training-free ByT5/MrT5 bytes
Pretrain scripts/pretrain.py Pretrain GPT on ClimbMix-400B or TinyStories
Encoder MLM scripts/train_encoder.py ModernBERT/mmBERT continued masked pretraining; guide and limits
Encoder embeddings scripts/train_yat_embedding_finetune.py Continue trained YAT weights with retrieval pairs/triplets; quality gates and pending upgrades
SFT scripts/sft.py Supervised fine-tuning on conversations
RL scripts/rl.py GRPO/REINFORCE on GSM8K with tool use
Eval scripts/eval.py CORE metric, MMLU, ARC, GSM8K, HumanEval
Chat scripts/chat_web.py FastAPI WebSocket chat UI
Local scripts/run_tinystories.py Full pipeline on TinyStories (laptop or GPU)
Export scripts/convert_to_tflite.py LiteRT/TFLite export for edge deployment

The repository contains shared training services, encoder and decoder models, cloud adapters, evaluation tools, and a PyTorch export implementation.

Architecture

The GPT model faithfully replicates every feature from nanochat:

  • Rotary Embeddings (RoPE) with 100K base theta
  • Group-Query Attention (GQA) via jax.nn.dot_product_attention (hardware-adaptive)
  • QK Normalization with 1.2x scaling for sharper attention
  • ReLU^2 MLP (squared ReLU activation)
  • Value Embeddings (ResFormer-style, alternating layers with gating)
  • Sliding Window Attention per-layer configurable (SSSL pattern)
  • Per-layer Residual Scaling (resid_lambdas + x0_lambdas)
  • Smear — cheap bigram-like token mixing from previous position
  • Backout — subtract mid-layer residual to remove low-level features
  • Logit Soft-capping via tanh(x/15)*15
  • Gradient Checkpointing (nnx.remat, dots_saveable policy)

Optimizer

Mixed Muon + AdamW (ported to optax):

Group Optimizer Notes
Attention/MLP matrices Muon Polar Express orthogonalization + NorMuon variance reduction
Embeddings (wte) AdamW b1=0.8, b2=0.995
LM head AdamW Lower LR for stability
Value embeddings AdamW Half embedding LR
Per-layer scalars AdamW Separate groups for resid_lambdas and x0_lambdas
Smear/Backout AdamW No weight decay

LR schedules: warmup (40 steps) -> constant -> warmdown (65% of total, final 5% of peak). Falls back to pure AdamW on Flax 0.11 where NamedTuple state has issues.

Inference Engine

Four generation modes with increasing performance:

Mode Function Speed Use Case
Padded generate() Measured per run Testing, debugging
KV-cached generate_with_cache() Measured per run Production, Python loop
Fully JIT generate_fast() Measured per run TPU inference via jax.lax.while_loop
Speculative generate_speculative() Measured per pairing Large model + small draft model

Tool Use

The Engine class provides streaming generation with automatic tool execution:

engine = Engine(model, tokenizer)
for token_column, masks in engine.generate(prompt_ids, num_samples=3, max_tokens=256):
    print(tokenizer.decode([token_column[0]]), end="")

When the model outputs <|python_start|>2+2<|python_end|>, the engine:

  1. Tries the safe calculator (use_calculator) for math and string.count()
  2. Keeps generated Python disabled by default; reviewed code can opt into the best-effort reliability guard (execute_code(..., trusted=True))
  3. Injects <|output_start|>4<|output_end|> tokens back into the stream

Speculative Decoding

Use a smaller draft model to propose tokens, verified in batch by the main model:

from flaxchat.engine import generate_speculative

# draft_model: 2-layer, model: 12-layer (same vocab)
tokens = generate_speculative(model, draft_model, prompt_ids, draft_steps=4)

Guarded Code Execution

For HumanEval evaluation and RL tool use:

from flaxchat.execution import execute_code

result = execute_code("print(sum(range(10)))", timeout=5.0, trusted=True)
# ExecutionResult(success=True, stdout="45\n", stderr="", error=None)

This trusted-code reliability guard is not a security sandbox. Untrusted generated code stays disabled unless a digest-pinned Docker/Podman boundary is configured; see docs/SECURITY.md.

Parallelism (built-in, not optional)

  • compute_init() creates a mesh over ALL available devices automatically
  • Data parallelism: with_sharding_constraint(data, P('data')) in every train step
  • FSDP: shard_model_fsdp() for models exceeding single-device memory
  • Multi-host: jax.distributed.initialize() + jax.make_array_from_process_local_data()
  • No manual all-reduce — JAX SPMD compiler handles gradient synchronization

Configuration

Single-dial depth-based auto-config — all hyperparameters derive from depth:

from flaxchat.config import FlaxChatConfig

config = FlaxChatConfig.from_depth(
    depth=12,            # 12 layers
    aspect_ratio=64,     # base_dim = 12 * 64 = 768
    head_dim=128,        # n_heads = 768 / 128 = 6
    max_seq_len=2048,
    window_pattern="SSSL",
)
# -> 12 layers, 768 dims, 6 heads, ~79M params

Evaluation Tasks

Task Type Source
MMLU Categorical (4-choice) cais/mmlu
ARC-Challenge Categorical allenai/ai2_arc
GSM8K Generative (math + calculator) openai/gsm8k
HumanEval Generative (code execution disabled by default) openai/humaneval
SpellingBee Generative (tool use) Built-in (30+ templates)
SmolTalk Conversation quality HuggingFaceTB/smol-smoltalk
CORE ICL benchmark (DCLM paper) Hellaswag, ARC, PIQA, Winogrande

Quick Start

Install

pixi install    # or: pip install -e ".[dev]"

Train locally on TinyStories

python -m scripts.run_tinystories \
  --layers=4 --embedding-dim=256 \
  --pretrain-steps=1000 --sft-steps=100 --rl-steps=100

The dataset is streamed from an immutable revision. The output directory contains the trained tokenizer, complete Orbax state, generated sample, stage metrics, source/config/data hashes, licensing metadata, and run_manifest.json. Use --smoke for the committed offline corpus.

FineWeb from prepared tokens

Prepare the bounded token pool on CPU before allocating a TPU. The preparation process needs datasets and transformers; the trainer reads only local files.

python -m scripts.prepare_token_pool \
  --output artifacts/fineweb-pool \
  --revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9 \
  --tokenizer-revision 607a30d783dfa663caf39e06633721c8d4cfcd7e \
  --train-tokens 1000001 --validation-tokens 65537
python -m scripts.train_gpt2 \
  --token-manifest artifacts/fineweb-pool/manifest.json \
  --depth 12 --batch-per-device 1 --seq-len 1024 \
  --tokens 1M --warmup-steps 10

Use --resume with the same training configuration and checkpoint directory. --stop-after stops at an absolute update count without changing the planned learning-rate schedule. This prepared-token trainer currently supports one physical host; its batch is sharded over all devices on that host. Larger token budgets require preparing enough data first; exhaustion is a hard error.

Full pipeline on TPU pod

python -m scripts.pretrain --depth=24 --num-iterations=50000
python -m scripts.sft --base-model=d24
python -m scripts.rl --model=d24
python -m scripts.eval --model=d24 --tasks=all
python -m scripts.chat_web --model=d24

Remote execution

# Bundle the full suite into one Kaggle v5e-8 allocation and wait for artifacts.
python -m scripts.kaggle_tpu_tests \
  --kernel-id OWNER/flaxchat-tpu-tests --wait

# GCP TPU (via tpuz)
python - <<'PY'
from tpuz import TPU
tpu = TPU("my-tpu", accelerator="v6e-8")
tpu.up()
tpu.setup(extra_pip="flaxchat")
tpu.run("python -m scripts.pretrain --depth=12", sync=".")
PY

Project Structure

flaxchat/
├── flaxchat/                  # Core library (~3,500 LOC)
│   ├── gpt.py                 # GPT model (all nanochat features)
│   ├── optim.py               # Mixed Muon+AdamW optimizer (optax)
│   ├── engine.py              # Inference: padded, cached, JIT, speculative, tool use
│   ├── execution.py           # Sandboxed Python code execution
│   ├── eval.py                # CORE metric + BPB evaluation
│   ├── dataloader.py          # BOS-aligned best-fit packing
│   ├── tokenizer.py           # BPE (rustbpe/HF) + ByT5-compatible UTF-8 bytes
│   ├── config.py              # Depth-based auto-config
│   ├── common.py              # Mesh, distributed, logging
│   ├── checkpoint.py          # Orbax checkpoint manager
│   ├── report.py              # Training reports
│   └── dataset.py             # Parquet file listing
├── scripts/                   # Executable scripts (~2,500 LOC)
├── tasks/                     # Evaluation tasks (MMLU, ARC, GSM8K, HumanEval, ...)
├── tests/                     # unit, integration, sharding, and accelerator tests
├── docs/                      # GitHub Pages documentation
├── configs/                   # YAML configuration templates
└── runs/                      # Launch scripts

Test Suite

The test suite covers model semantics, four generation modes, optimizer schedules and numerical safety, guarded execution, tokenizers, exact dataloader resume, checkpoint integrity, configuration, evaluation protocols, reports, datasets, sharding, and TPU attention parity.

pixi run test-quick       # deterministic CPU developer loop
pixi run test-coverage    # branch coverage with a 65% floor
pixi run test-multidevice # eight virtual CPU devices
pixi run lint             # fatal Python correctness/static errors
pixi run typecheck        # typed configuration and pipeline contracts
pixi run audit            # declared dependency vulnerability audit

Verified Results

The canonical, machine-readable acceptance records are indexed in benchmarks/results/provenance-index.json, with a readable summary in docs/RESULTS.md. The latest verified TPU bundle is tied to the full source SHA and distinguishes acceptance tests from scaling or quality claims. Matched nanochat/MaxText measurements remain explicitly pending; no comparison is presented as apples-to-apples until every protocol record validates.

Comparison with nanochat

nanochat flaxchat
Framework PyTorch JAX/Flax NNX
Hardware NVIDIA GPU (8xH100) TPU pods + GPUs
Distributed DDP + torch.distributed JAX SPMD mesh (automatic)
Compile torch.compile jax.jit / nnx.jit
Attention Flash Attention 3 jax.nn.dot_product_attention
Precision bf16/fp16/fp8 bf16 (TPU native)
Optimizer Custom MuonAdamW Custom optax Muon+AdamW
Checkpointing Pickle-based Orbax (async, cloud-friendly)
Generation KV-cache + Python loop 4 modes: padded, cached, JIT, speculative
Tool use Calculator + Python REPL Calculator + opt-in guarded REPL
Remote execution N/A Kaggle CLI + TPU (tpuz)
Config Manual Depth-based auto-scaling

Acknowledgments

This project is part of the 2026 Q1 TPU Sprint, supported by the Google AI Developer Programs team.

We gratefully acknowledge:

  • Google AI Developer Programs for issuing GCP credits that made large-scale training experiments possible
  • TPU Research Cloud (TRC) for providing free access to Cloud TPU v4, v5e, and v6e accelerators
  • Kaggle for providing free TPU v5e access for prototyping and validation

Built on:

License

MIT

The four-chip Spot benchmark and recovery pilot measured 72.5%/82.0% lower estimated invocation compute cost at 1K/4K than the matched 16-chip run, and exact recovery after a step-200 process kill. The sampled ARC-Easy result does not establish above-chance downstream quality.

About

End-to-end JAX/Flax NNX LLM training, evaluation, inference, checkpointing, and TPU/GPU execution harness

Resources

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages