Native JAX/Flax NNX training, recovery and inference tools for TPU language models and the GOAT multilingual encoder family.
GOAT mmBERT YAT Base MLM 10B is a 22-layer, 768-dimensional encoder with GOAT-input attention and YATGLU feed-forward blocks. It was initialized from scratch and pretrained with masked language modeling, including complete coverage of a 10,038,104,948 nonpadding-token prepared corpus after an earlier MLM stage. The published checkpoint is 379079; it has no contrastive or downstream fine-tuning.
Load the native model directly from Hugging Face through FlaxChat:
pip install -e ".[encoder]"from flaxchat.hub import from_pretrained
model = from_pretrained(
"mlnomad/goat-mmbert-yat-base-mlm-10b",
revision="a0279613f6a2cca746847fbba1672a6f0b407ea4",
)
# model.encode(token_ids): token representations
# model.pool(token_ids): pooled, normalized representationsThe model card includes tokenization and batching examples. Model execution and numerical acceptance use the documented accelerator runtime; metadata-only tests do not qualify inference or training.
| Benchmark | GOAT MLM-only base | Previous YAT embedding v1, fine-tuned | mmBERT-base paper, embedding fine-tuned |
|---|---|---|---|
| English MTEB, 41 tasks / seven-category mean | 36.84 | 52.78 | 53.9 |
| Multilingual MTEB, 131 tasks / eight-category mean | 31.77 | 48.21 | 54.1 |
| CoIR, ten-task mean | 9.91 | 43.45 | 42.2 |
| MIRACL hard-negative, 18-language mean | 1.82 | 46.37 | — |
Scores use the 0–100 display scale. GOAT measurements use the complete pinned task inventories, length512, nonpadding mean pooling including special tokens, FP32 L2 normalization and no prompts. Previous-model results are historical published receipts; mmBERT paper embedding results follow fine-tuning. These columns have different training histories. See full results and protocol.
Contrastive pilot status — October28,2026: eight-chip parent import and four hard-negative loss/gradient checks passed. The real-data recovery qualification stopped before optimizer updates because its development sample was too small to cover every language. The local sample-budget correction passed model-free admission; physical revalidation and production contrastive training remain pending. Protected MLM checkpoints and failure evidence are retained, and the attempt's compute resources were independently verified removed. See current model and training status.
Latest reconciled audit: all open issues, current evidence and tool priorities. Earlier dated findings below remain historical where superseded.
Reproducibility and checkpoint contracts are documented in docs/REPRODUCIBILITY.md and docs/CHECKPOINT_FORMAT.md. Release gates and compatibility are documented in docs/RELEASES.md. Exact accelerator measurements and their limitations are in docs/RESULTS.md. The current engineering risks and prioritized backlog are in the September 30 system audit.
The October 1 follow-up audit records the remaining recovery, admission and release blockers in the new local fixes. The complete open-issue checklist identifies the remaining acceptance for all 29 issues. For continued YAT embedding representation training, use the training runbook, tool inventory, and versioned training skill. The September 3 audit is historical. The Spot TPU validation report records the prepared-data training fixes, interruption recovery, and complete TPU test inventory. The GCP multi-host campaign records physical worker acceptance, FSDP recovery, scale coverage, and remaining limits. The follow-up audit fixes cover update rollback, artifact identity, streaming shards, CI and bounded Spot supervision. The training-readiness follow-up records tied-head initialization, checkpoint overhead, four-host launcher fixes, physical qualification, and posted billing evidence. Supported Python and accelerator combinations are in docs/COMPATIBILITY.md.
A minimal, end-to-end LLM training harness for Google Cloud TPU pods, built on JAX/Flax NNX.
JAX/Flax NNX adaptation of nanochat, with TPU-oriented execution and speculative decoding.
pixi install
pixi run test
pixi run test-e2e # offline end-to-end acceptance smoke
python -m scripts.run_tinystories # pinned TinyStories end-to-end pipelineflaxchat is the complete LLM pipeline running natively on TPUs and GPUs with automatic data parallelism:
| Stage | Script | Description |
|---|---|---|
| Tokenizer | scripts/tok_train.py |
Train BPE, or select training-free ByT5/MrT5 bytes |
| Pretrain | scripts/pretrain.py |
Pretrain GPT on ClimbMix-400B or TinyStories |
| Encoder MLM | scripts/train_encoder.py |
ModernBERT/mmBERT continued masked pretraining; guide and limits |
| Encoder embeddings | scripts/train_yat_embedding_finetune.py |
Continue trained YAT weights with retrieval pairs/triplets; quality gates and pending upgrades |
| SFT | scripts/sft.py |
Supervised fine-tuning on conversations |
| RL | scripts/rl.py |
GRPO/REINFORCE on GSM8K with tool use |
| Eval | scripts/eval.py |
CORE metric, MMLU, ARC, GSM8K, HumanEval |
| Chat | scripts/chat_web.py |
FastAPI WebSocket chat UI |
| Local | scripts/run_tinystories.py |
Full pipeline on TinyStories (laptop or GPU) |
| Export | scripts/convert_to_tflite.py |
LiteRT/TFLite export for edge deployment |
The repository contains shared training services, encoder and decoder models, cloud adapters, evaluation tools, and a PyTorch export implementation.
The GPT model faithfully replicates every feature from nanochat:
- Rotary Embeddings (RoPE) with 100K base theta
- Group-Query Attention (GQA) via
jax.nn.dot_product_attention(hardware-adaptive) - QK Normalization with 1.2x scaling for sharper attention
- ReLU^2 MLP (squared ReLU activation)
- Value Embeddings (ResFormer-style, alternating layers with gating)
- Sliding Window Attention per-layer configurable (SSSL pattern)
- Per-layer Residual Scaling (
resid_lambdas+x0_lambdas) - Smear — cheap bigram-like token mixing from previous position
- Backout — subtract mid-layer residual to remove low-level features
- Logit Soft-capping via
tanh(x/15)*15 - Gradient Checkpointing (
nnx.remat,dots_saveablepolicy)
Mixed Muon + AdamW (ported to optax):
| Group | Optimizer | Notes |
|---|---|---|
| Attention/MLP matrices | Muon | Polar Express orthogonalization + NorMuon variance reduction |
| Embeddings (wte) | AdamW | b1=0.8, b2=0.995 |
| LM head | AdamW | Lower LR for stability |
| Value embeddings | AdamW | Half embedding LR |
| Per-layer scalars | AdamW | Separate groups for resid_lambdas and x0_lambdas |
| Smear/Backout | AdamW | No weight decay |
LR schedules: warmup (40 steps) -> constant -> warmdown (65% of total, final 5% of peak). Falls back to pure AdamW on Flax 0.11 where NamedTuple state has issues.
Four generation modes with increasing performance:
| Mode | Function | Speed | Use Case |
|---|---|---|---|
| Padded | generate() |
Measured per run | Testing, debugging |
| KV-cached | generate_with_cache() |
Measured per run | Production, Python loop |
| Fully JIT | generate_fast() |
Measured per run | TPU inference via jax.lax.while_loop |
| Speculative | generate_speculative() |
Measured per pairing | Large model + small draft model |
The Engine class provides streaming generation with automatic tool execution:
engine = Engine(model, tokenizer)
for token_column, masks in engine.generate(prompt_ids, num_samples=3, max_tokens=256):
print(tokenizer.decode([token_column[0]]), end="")When the model outputs <|python_start|>2+2<|python_end|>, the engine:
- Tries the safe calculator (
use_calculator) for math andstring.count() - Keeps generated Python disabled by default; reviewed code can opt into the
best-effort reliability guard (
execute_code(..., trusted=True)) - Injects
<|output_start|>4<|output_end|>tokens back into the stream
Use a smaller draft model to propose tokens, verified in batch by the main model:
from flaxchat.engine import generate_speculative
# draft_model: 2-layer, model: 12-layer (same vocab)
tokens = generate_speculative(model, draft_model, prompt_ids, draft_steps=4)For HumanEval evaluation and RL tool use:
from flaxchat.execution import execute_code
result = execute_code("print(sum(range(10)))", timeout=5.0, trusted=True)
# ExecutionResult(success=True, stdout="45\n", stderr="", error=None)This trusted-code reliability guard is not a security sandbox. Untrusted generated
code stays disabled unless a digest-pinned Docker/Podman boundary is configured;
see docs/SECURITY.md.
compute_init()creates a mesh over ALL available devices automatically- Data parallelism:
with_sharding_constraint(data, P('data'))in every train step - FSDP:
shard_model_fsdp()for models exceeding single-device memory - Multi-host:
jax.distributed.initialize()+jax.make_array_from_process_local_data() - No manual all-reduce — JAX SPMD compiler handles gradient synchronization
Single-dial depth-based auto-config — all hyperparameters derive from depth:
from flaxchat.config import FlaxChatConfig
config = FlaxChatConfig.from_depth(
depth=12, # 12 layers
aspect_ratio=64, # base_dim = 12 * 64 = 768
head_dim=128, # n_heads = 768 / 128 = 6
max_seq_len=2048,
window_pattern="SSSL",
)
# -> 12 layers, 768 dims, 6 heads, ~79M params| Task | Type | Source |
|---|---|---|
| MMLU | Categorical (4-choice) | cais/mmlu |
| ARC-Challenge | Categorical | allenai/ai2_arc |
| GSM8K | Generative (math + calculator) | openai/gsm8k |
| HumanEval | Generative (code execution disabled by default) | openai/humaneval |
| SpellingBee | Generative (tool use) | Built-in (30+ templates) |
| SmolTalk | Conversation quality | HuggingFaceTB/smol-smoltalk |
| CORE | ICL benchmark (DCLM paper) | Hellaswag, ARC, PIQA, Winogrande |
pixi install # or: pip install -e ".[dev]"python -m scripts.run_tinystories \
--layers=4 --embedding-dim=256 \
--pretrain-steps=1000 --sft-steps=100 --rl-steps=100The dataset is streamed from an immutable revision. The output directory
contains the trained tokenizer, complete Orbax state, generated sample, stage
metrics, source/config/data hashes, licensing metadata, and run_manifest.json.
Use --smoke for the committed offline corpus.
Prepare the bounded token pool on CPU before allocating a TPU. The preparation
process needs datasets and transformers; the trainer reads only local files.
python -m scripts.prepare_token_pool \
--output artifacts/fineweb-pool \
--revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9 \
--tokenizer-revision 607a30d783dfa663caf39e06633721c8d4cfcd7e \
--train-tokens 1000001 --validation-tokens 65537
python -m scripts.train_gpt2 \
--token-manifest artifacts/fineweb-pool/manifest.json \
--depth 12 --batch-per-device 1 --seq-len 1024 \
--tokens 1M --warmup-steps 10Use --resume with the same training configuration and checkpoint directory.
--stop-after stops at an absolute update count without changing the planned
learning-rate schedule. This prepared-token trainer currently supports one
physical host; its batch is sharded over all devices on that host. Larger token
budgets require preparing enough data first; exhaustion is a hard error.
python -m scripts.pretrain --depth=24 --num-iterations=50000
python -m scripts.sft --base-model=d24
python -m scripts.rl --model=d24
python -m scripts.eval --model=d24 --tasks=all
python -m scripts.chat_web --model=d24# Bundle the full suite into one Kaggle v5e-8 allocation and wait for artifacts.
python -m scripts.kaggle_tpu_tests \
--kernel-id OWNER/flaxchat-tpu-tests --wait
# GCP TPU (via tpuz)
python - <<'PY'
from tpuz import TPU
tpu = TPU("my-tpu", accelerator="v6e-8")
tpu.up()
tpu.setup(extra_pip="flaxchat")
tpu.run("python -m scripts.pretrain --depth=12", sync=".")
PYflaxchat/
├── flaxchat/ # Core library (~3,500 LOC)
│ ├── gpt.py # GPT model (all nanochat features)
│ ├── optim.py # Mixed Muon+AdamW optimizer (optax)
│ ├── engine.py # Inference: padded, cached, JIT, speculative, tool use
│ ├── execution.py # Sandboxed Python code execution
│ ├── eval.py # CORE metric + BPB evaluation
│ ├── dataloader.py # BOS-aligned best-fit packing
│ ├── tokenizer.py # BPE (rustbpe/HF) + ByT5-compatible UTF-8 bytes
│ ├── config.py # Depth-based auto-config
│ ├── common.py # Mesh, distributed, logging
│ ├── checkpoint.py # Orbax checkpoint manager
│ ├── report.py # Training reports
│ └── dataset.py # Parquet file listing
├── scripts/ # Executable scripts (~2,500 LOC)
├── tasks/ # Evaluation tasks (MMLU, ARC, GSM8K, HumanEval, ...)
├── tests/ # unit, integration, sharding, and accelerator tests
├── docs/ # GitHub Pages documentation
├── configs/ # YAML configuration templates
└── runs/ # Launch scripts
The test suite covers model semantics, four generation modes, optimizer schedules and numerical safety, guarded execution, tokenizers, exact dataloader resume, checkpoint integrity, configuration, evaluation protocols, reports, datasets, sharding, and TPU attention parity.
pixi run test-quick # deterministic CPU developer loop
pixi run test-coverage # branch coverage with a 65% floor
pixi run test-multidevice # eight virtual CPU devices
pixi run lint # fatal Python correctness/static errors
pixi run typecheck # typed configuration and pipeline contracts
pixi run audit # declared dependency vulnerability auditThe canonical, machine-readable acceptance records are indexed in
benchmarks/results/provenance-index.json,
with a readable summary in docs/RESULTS.md. The latest
verified TPU bundle is tied to
the full source SHA and distinguishes acceptance tests from scaling or quality claims.
Matched nanochat/MaxText measurements remain explicitly pending; no comparison is
presented as apples-to-apples until every protocol record validates.
| nanochat | flaxchat | |
|---|---|---|
| Framework | PyTorch | JAX/Flax NNX |
| Hardware | NVIDIA GPU (8xH100) | TPU pods + GPUs |
| Distributed | DDP + torch.distributed | JAX SPMD mesh (automatic) |
| Compile | torch.compile |
jax.jit / nnx.jit |
| Attention | Flash Attention 3 | jax.nn.dot_product_attention |
| Precision | bf16/fp16/fp8 | bf16 (TPU native) |
| Optimizer | Custom MuonAdamW | Custom optax Muon+AdamW |
| Checkpointing | Pickle-based | Orbax (async, cloud-friendly) |
| Generation | KV-cache + Python loop | 4 modes: padded, cached, JIT, speculative |
| Tool use | Calculator + Python REPL | Calculator + opt-in guarded REPL |
| Remote execution | N/A | Kaggle CLI + TPU (tpuz) |
| Config | Manual | Depth-based auto-scaling |
This project is part of the 2026 Q1 TPU Sprint, supported by the Google AI Developer Programs team.
We gratefully acknowledge:
- Google AI Developer Programs for issuing GCP credits that made large-scale training experiments possible
- TPU Research Cloud (TRC) for providing free access to Cloud TPU v4, v5e, and v6e accelerators
- Kaggle for providing free TPU v5e access for prototyping and validation
Built on:
- nanochat by Andrej Karpathy
- JAX, Flax, Optax, Orbax
- tpuz for TPU VM management
- Kaggle CLI for headless accelerator execution
MIT
The four-chip Spot benchmark and recovery pilot measured 72.5%/82.0% lower estimated invocation compute cost at 1K/4K than the matched 16-chip run, and exact recovery after a step-200 process kill. The sampled ARC-Easy result does not establish above-chance downstream quality.
