scratchbench

Can the model actually implement it?

Machine learning machinery — attention masks, KV-caches, gradient accumulation, quantization, a fused Triton kernel, a Metal threadgroup reduction — written from a written specification and graded by hidden pytest suites. On the laptop you already own. Under five minutes a task. The CPU tier needs no GPU and no cloud bill; grading a real model is a paid API call, and three of the tasks want a GPU — both kept apart from the laptop rate rather than folded into it.

$ scratchbench run --model claude-haiku-4-5 --tasks softmax_stability
  softmax_stability ... pass  (29 passed)

claude-haiku-4-5  ·  2026-08-03T05:43:14+00:00  ·  task set v1

task               category  tier    diff  result  time   tests
-----------------  --------  ------  ----  ------  -----  ---------
softmax_stability  numerics  laptop  1     pass    0.34s  29 passed

laptop       100% (1/1)
10.3s wall clock  ·  1 attempt(s) per task  ·  $0.01

A model was handed a specification, wrote a file, and twenty-nine tests it never saw agreed. Ten seconds and eight tenths of a cent.

---

The gap this fills

I like MLE-bench. I like KernelBench. I could not run either of them.

MLE-bench asks agents to place in Kaggle competitions: real datasets, real training runs, hours of compute per task. KernelBench wants an NVIDIA GPU. SWE-bench and the harnesses around it grade something else entirely — fixing issues in an existing repository, which is a different skill from building the thing in the first place.

So there is a gap, and it is a specific one. What I could not find is a benchmark for whether a model can implement the machinery of machine learning — the causal mask, the KV-cache, the gradient accumulation, the quantization round-trip — correctly, from a specification, on a laptop, with nothing to copy from. The benchmarks above and their neighbours measure things next to it — Kaggle placements, GPU kernels, issue fixing; this is the corner I wanted and did not have.

And here is the part that makes closing it possible: you do not need a training run to grade an implementation. You need tests. A wrong attention mask fails a shape-and-values assertion in eleven milliseconds. The loss curve was never the referee.

The rule everything else follows from

Every task must be gradeable on a laptop, without a GPU, in under five minutes.

That single constraint is the whole design:

• Anyone can reproduce the leaderboard. If you cannot re-run a benchmark, you are trusting a number, and trusting numbers is how benchmarks rot.
• Grading is objective. Tests pass or they do not. No judge model, no rubric, no argument.
• A full sweep costs tens of dollars, not thousands — so it can be re-run the day a new model ships, not six months later.

Seventeen tasks and 808 hidden tests. Fourteen of them run on any laptop and need nothing but Python; the other three need a GPU (CUDA and Metal), are reported beside the headline with the hardware each one wants, and are never folded into it.

The first rows

Laptop tier, five independent draws each, one attempt per task, both models at their own defaults:

Model	Pass rate	Draws	Solved	Cost per draw	Wall clock
claude-opus-5	100%, every draw	5	40/40	$0.49 to $0.56	187 to 215s
claude-haiku-4-5	38% to 50%	5	18/40	$0.059 to $0.077	62 to 90s

Read that first row as a limit as much as a result: the laptop tier has stopped measuring at the top. The sampling is real and it shows up in the same files — the prompt is byte-identical each time so the input is fixed at 15858 tokens, while the output ranges over 16476 to 19070, a 16% spread in what the model actually wrote. Five materially different answers, one verdict every time. A ceiling hit once is a good day; a ceiling hit five times out of five is a ceiling.

The accelerated tier has not stopped measuring — but a fuller sweep moved which task does it. The first CUDA rows were one Triton task at five draws, Opus 1/5 and Haiku 3/5, where the finding was never the rate but the shape underneath it: Opus failed one test out of twenty-four, the same one every time; Haiku, when it failed, failed all twenty-four. A working kernel with an edge missed and a kernel that does not run are the same word under binary scoring and completely different results. The edge is that Triton specializes an integer argument whose value is 1 into a compile-time constant, so n_cols.to(tl.float32) compiles at every row width except a single-column one — which prompt.md licenses explicitly.

Finished at ten draws across both vendors on 2026-10-28, that task saturated at the top: Opus 5/10 (still only the single-column edge, five for five), Haiku 5/10, and claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna all 10/10. Two entries at 100% is exactly what the admission rule refuses, so the CUDA task is warmup now and the accelerated tier's discrimination has moved to the two Metal tasks, where it is still sharp — metal_cross_entropy_kernel goes Opus 23/30, Sonnet 19/30, Haiku 4/29. The next set has to earn its place is the whole story, and it is flash-attention's ejection a second time.

Every results file is checked in at leaderboard/ and every cost re-derives from the tokens beside it. The whole table cost $3.30. What all of it implies about the next task set is docs/V2_DESIGN.md.

Run it

git clone https://github.com/selimfedakar/scratchbench
cd scratchbench
pip install -e .

scratchbench validate                          # every task, both halves of the check
scratchbench run --model reference --tasks all # the control: known-correct solutions
scratchbench report                            # your results, next to everyone else's

Run it from the clone. The tasks are the benchmark and they live in the repository, not in a published wheel — pip install scratchbench on its own would install the harness with nothing to grade, so there is no PyPI release and the editable install above is the supported path. Run from a checkout and the CLI finds the tasks beside it; run from an installed wheel and it says so rather than failing obscurely.

To grade a real model, add the SDK and a key:

pip install 'scratchbench[anthropic]'
export ANTHROPIC_API_KEY=...
scratchbench run --model claude-opus-5 --tasks all

Start with --model reference, though, and not out of politeness. A harness that has never been proven against a correct solution will happily report that a model failed when what actually failed was the harness — and a benchmark gets exactly one chance to publish a wrong number about somebody else's work.

Running generated code

Grading a model runs code the model wrote. The harness strips credential-shaped environment variables from the grading process and only drops the hidden tests in after the solver has finished writing, so a solution cannot read a key or a test it should not — but that is the limit of it, and it is not a security sandbox. pytest runs the solution in the same interpreter as the tests, with this machine's filesystem and network. Generated code can read outside its working directory and reach the network, exactly as any Python you run can. Treat scratchbench run --model <real model> the way you would treat running any untrusted program: on a machine, or in a disposable container or VM, that you are willing to hand to it. The reference control run is your own code and carries none of this. A properly isolated grading path is on the roadmap and is not here yet.

Three things this found that a pass rate would not have

A percentage tells you a model failed. It does not tell you how, and how is where everything interesting in this repository has been.

A frontier model lost to a reserved word. On metal_cross_entropy_kernel, both of Opus 5's failures and five of Sonnet 5's six are the same compile error, on the same identifier:

E   SyntaxError: program_source:39:14: error: cannot combine with previous 'type-name' declaration specifier
E           uint half = (n + 1u) / 2u;
E                ^

half is a type in Metal Shading Language, so it cannot name a variable — and the threadgroup reduction underneath it is correct in every one of those draws. The model knew the algorithm and lost to the language. I made the identical mistake in the first draft of the reference, an hour before either model was asked (L34).

A model reproduced one of my invented wrong answers, exactly, twice. Before any model is asked anything, each task's reference gets deliberately mutated into the three or four wrong versions a real implementation would plausibly produce, and the hidden tests have to catch each one. On flash_attention_backward, Sonnet 5's two failures in ten draws on 2026-08-09 are the same two tests both times, and they are the signature of a mutant written weeks earlier: queries not put at the end of the key range. It happened again on custom_autograd_double_backward, where ten draws of ten from Haiku reproduced mutant three — saving the forward's sigmoid and reusing it in a backward where it arrives as a constant.

Both of those are dated on purpose. Re-drawn on 2026-09-17, Sonnet went 10 for 10 on flash_attention_backward and the signature did not reappear — which is the same sentence this page makes about every other number here, arriving about its own best story.

A kernel that ignores the GPU passes every test. metal_cross_entropy_kernel grades the answer, and a kernel that uses exactly one lane and loops serially over the row — no reduction, no threadgroup, nothing the task is about — passes all sixty-eight tests. The only assertion that would catch it is a wall-clock one, and those are banned in every tier here because timing on unknown hardware is not reproducible. So the hole is checked in as a mutant whose expected verdict is SURVIVES, which makes it a re-runnable fact rather than a footnote (L33). A correctness benchmark cannot see a slow kernel, and this one says so in its own test suite rather than in a disclaimer.

None of those three is visible in a score. All three are why the failure shapes are published beside every rate.

What "verified" means here, and why it is two things

Every task in this repository has to clear a bar with two halves:

1. the reference solution passes the hidden tests, and
2. the untouched starter fails them with real assertion errors — not import errors, not collection errors.

Half of that pair is worthless. A suite that passes against a correct solution proves nothing if it also passes against an empty one, and an import error and a wrong answer score identically while meaning completely different things.

This is the default command, run on 2026-09-17 in a fresh clone of this repository on a machine with no GPU:

$ scratchbench validate
task                             reference   untouched starter  time   verdict
-------------------------------  ----------  -----------------  -----  -------
activation_checkpointing_rng     34 passed   34 failed          7.87s  ok
attention_causal_mask            26 passed   26 failed          1.67s  ok
bpe_merge_order                  30 passed   30 failed          0.15s  ok
chunked_batchnorm_backward       44 passed   44 failed          1.03s  ok
chunked_batchnorm_reduction      57 passed   57 failed          1.05s  ok
custom_autograd_double_backward  116 passed  116 failed         1.39s  ok
flash_attention_backward         49 passed   49 failed          1.16s  ok
grad_accumulation                27 passed   27 failed          2.17s  ok
kv_cache_equivalence             21 passed   21 failed          0.20s  ok
online_softmax_attention         57 passed   57 failed          0.23s  ok
quantization_error_bounds        65 passed   65 failed          0.24s  ok
sharded_dataloader               53 passed   53 failed          0.32s  ok
softmax_stability                29 passed   29 failed          0.22s  ok
speculative_decoding_verify      24 passed   24 failed          4.11s  ok

14 task(s) validated: reference passes, starter fails cleanly.

Fourteen, not seventeen, and that is the tier separation rather than a gap: the three accelerated tasks are not in the default scope at all. Ask for them with --tier all and a task whose hardware is missing reports itself instead of being counted:

fused_rmsnorm_kernel          -          -          -      UNCHECKED  no cuda device on this machine

Not a pass, not a failure, and not silently absent either.

There is a third check that does not fit in a table, and it is the one that has found the most: the reference gets mutated on purpose. Passing tests prove the reference satisfies the contract they cover; only mutants prove the tests would catch a wrong version of it at all. Every wrong version a real implementation would plausibly produce — the off-by-one, the wrong axis, the missing rescale — gets written and run, and the suite has to catch each one.

That check has changed this repository three times, and docs/LESSONS.md has all three. The shortest version of why it exists: sharded_dataloader passed fifty-one tests, and a wrong implementation that gave each rank a contiguous slab instead of every world_size-th sample passed all fifty-one of them too.

Two claims, and the receipts

A GPU task does not get into the frozen set on the strength of its directory listing. fused_rmsnorm_kernel — a fused RMSNorm forward pass in Triton, one program per row, float32 accumulation under a float16 input — was written on a laptop with no CUDA device and marked frozen_set: unvalidated until it had run on hardware. When it finally did, on a rented RTX A4500, it came back with two real defects: a mutant that survived, and two tests that could not fail. Both are fixed, and the run that says so is re-runnable:

$ bash tools/verify_accelerated.sh
=== 1. hardware ===
NVIDIA RTX A4500 | torch 2.8.0+cu128 | triton 3.4.0
...
fused_rmsnorm_kernel  24 passed  24 failed          6.71s  ok
...
all 6 mutants caught

The mutant that survived the first time was the one the task is named after — a float16 accumulator instead of a float32 one. The test written to catch it was arithmetic about an implementation I was not running. docs/LESSONS.md L14 is the whole story, and the reason the accelerated tier exists at all is L10.

A cost on a leaderboard has to be reproducible, not plausible. Every results file carries the tokens the price was computed from — input, output, cache read, cache write — because a stale price table and a run that used more tokens than expected produce the same figure, and afterwards nothing tells them apart. Not for you and not for me. That gap was found by trying to check my own number and discovering I had not kept the means to.

$ python tools/check_cost.py
ok           claude-haiku-4-5-20260803.json  12103 in + 11947 out = $0.071838
ok           claude-opus-5-20260803.json  15858 in + 18453 out = $0.540615
...
251 file(s) checked: every cost reproduces from its own tokens.

CI runs that on every push, because a check done once by hand in a README is a check that stops running the day it is written.

How a task works

Each task hands the model a specification and a stub. It never sees the tests.

tasks/attention_causal_mask/
  meta.yaml          category, difficulty, time limit, what it probes
  prompt.md          the specification the model receives
  starter/           the files it edits — signatures fixed, bodies empty
  hidden_tests/      never shown to the model; the only thing that decides the score
  reference/         a correct solution, for maintainers only

Scoring is binary per task: every hidden test passes, or the task is failed. Partial credit hides exactly the failures worth knowing about.

The graded environment is pinned — one thread, fixed hash seed, no inherited PYTHONPATH — because numpy and torch change the order of their floating-point reductions with the number of cores available, and a task that flips from pass to fail depending on which machine graded it is a benchmark quietly lying in the direction that is hardest to notice.

See TASK_FORMAT.md for the full contract, and CONTRIBUTING.md if you want to add a task.

The task set

Task	Set	Category	Difficulty	Tests	Calibrated against
softmax_stability	v1	numerics	1	29	not calibrated
bpe_merge_order	v1	tokenization	2	30	not calibrated
grad_accumulation	v1	training	3	27	not calibrated
kv_cache_equivalence	v1	attention	3	21	not calibrated
online_softmax_attention	v1	attention	5	57	not calibrated
metal_cross_entropy_kernel ⚡	v2	kernels	4	68	claude-opus-5 23/30 · claude-sonnet-5 19/30 · claude-haiku-4-5 4/29 · gpt-5.6-sol 7/10 · gpt-5.6-terra 9/10 · gpt-5.6-luna 3/7
metal_softmax_backward_kernel ⚡	v2	kernels	4	81	claude-opus-5 16/20 · claude-sonnet-5 12/20 · claude-haiku-4-5 2/18 · gpt-5.6-sol 7/10 · gpt-5.6-terra 9/10 · gpt-5.6-luna 4/7
attention_causal_mask	warmup	attention	2	26	claude-opus-5 6/6 · claude-sonnet-5 5/5 · claude-haiku-4-5 4/6
activation_checkpointing_rng	warmup	training	4	34	claude-opus-5 10/10 · claude-sonnet-5 10/10 · claude-haiku-4-5 1/10
chunked_batchnorm_backward	warmup	training	4	45	claude-opus-5 10/10 · claude-sonnet-5 10/10 · claude-haiku-4-5 0/10
custom_autograd_double_backward	warmup	training	4	116	claude-opus-5 10/10 · claude-sonnet-5 10/10 · claude-haiku-4-5 0/10
fused_rmsnorm_kernel ⚡	warmup	kernels	4	24	claude-opus-5 6/15 · claude-sonnet-5 10/10 · claude-haiku-4-5 8/15 · gpt-5.6-sol 10/10 · gpt-5.6-terra 10/10 · gpt-5.6-luna 10/10
quantization_error_bounds	warmup	numerics	4	65	claude-opus-5 6/6 · claude-sonnet-5 5/5 · claude-haiku-4-5 6/6
sharded_dataloader	warmup	data	4	53	claude-opus-5 6/6 · claude-sonnet-5 5/5 · claude-haiku-4-5 6/6
speculative_decoding_verify	warmup	attention	4	24	claude-opus-5 15/15 · claude-sonnet-5 10/10 · claude-haiku-4-5 10/20
chunked_batchnorm_reduction	warmup	training	5	59	claude-opus-5 8/10 · claude-sonnet-5 9/10 · claude-haiku-4-5 0/10
flash_attention_backward	warmup	attention	5	49	claude-opus-5 25/25 · claude-sonnet-5 18/20 · claude-haiku-4-5 5/29 · gpt-5.6-sol 10/10 · gpt-5.6-terra 10/10 · gpt-5.6-luna 9/10

One bar per model per task, re-derived from the checked-in draws by tools/render_chart.py and CI-gated like every other number here. A row of full bars is a task that has stopped measuring; a row that spreads is one that still separates the field. The two Metal tasks and the ejected CUDA task are where the spread lives.

v1 is five laptop tasks and it was eight, plus the CUDA kernel, when the rows above were measured. attention_causal_mask, sharded_dataloader and quantization_error_bounds moved to warmup on 2026-08-13, on the draws already published rather than on new ones: three models, six draws each for two of them, and only Haiku ever failed one. A task that everything solves is not wrong, it has stopped measuring, and the rule that says so is below. fused_rmsnorm_kernel moved the other way on 2026-08-17, out of v1 and into v2 on Opus 1/5 — and then back to warmup on 2026-10-28 when its ten-draw sweep across both vendors put two models at the ceiling, the same rule removing it that admitted it. The round trip is below.

⚡ = accelerated tier: needs a GPU — CUDA for the Triton task, Metal for the other, which is to say any Apple silicon Mac. Reported beside the laptop rate and never folded into it. On a machine without the hardware it comes back needs_accelerator — not a pass, not a failure, an absence of evidence, and it stays out of every percentage rather than being rounded down to zero.

Read the last column before the difficulty column, because it is the one that qualifies the other. A difficulty here is not a property of a task, it is a statement about the models that were asked — and for most of the history of this repository those models were three Claudes. warmup does not mean "easy", it means the frontier we measured clears this, and a frontier is a specific list of models on a specific date. When the first OpenAI draws arrived they made the point at this repository's expense: gpt-5.6-luna, the cheapest model in that generation, solved all nine tasks it was asked, including the two this benchmark relies on to separate frontier models from each other. A task that separates one lab's models may not separate another's, and no warmup label in this table should be read as a claim about difficulty in general. docs/LESSONS.md L46.

The next set has to earn its place

Every difficulty number in the table above was assigned by how hard the task felt to write, before a single model had been asked. Then a model was asked, and Claude Opus 5 solved all eight laptop tasks in every one of five draws. Those numbers were a measurement of the author.

So from v2 onward a task carries what models actually scored on it, and the rule is mechanical rather than editorial:

calibration:
  - model: claude-opus-5
    draws: 15
    passed: 15
    date: 2026-08-09

A task the top of the field clears cannot be in a numbered set. The loader refuses it — not a warning, a TaskError — because a task that the frontier never fails costs a sweep and returns no information. It goes to frozen_set: warmup, where it still separates a small model from a large one and stays out of the headline. Concretely: a numbered set needs two calibration entries of five draws or more, and the highest two pass rates among them must not both be 100%. TASK_FORMAT.md has the table.

That rule read one entry until 2026-08-17 — the single best model — and this is what happened to the first three tasks written against it:

Task	Set	Tests	Opus 5	Sonnet 5	Haiku 4.5	What it probes
speculative_decoding_verify	warmup	24	15/15	10/10	10/20	accept-and-correct, graded on the output distribution
flash_attention_backward	v2	49	15/15	8/10	3/19	a derived backward, and the row term that is not blockwise
activation_checkpointing_rng	warmup	34	10/10	10/10	1/10	recomputation that puts the generator back where it found it

All three were refused. Two of them deserved it: Opus and Sonnet both sweep them, and a task two frontier models clear has stopped measuring the frontier. The middle one did not. Sonnet loses two draws of ten to it, which made it the only task on the laptop tier that separated one frontier model from another — and the rule threw it out on Opus's account, because it equated the top of the field with the strongest model in it. Reading the top two entries instead moves exactly one task in the whole repository, which is that one. The uncomfortable part — that a rule was rewritten after it refused something its author liked, and that an earlier journal had already decided the other way — is written up as docs/LESSONS.md L35 rather than smoothed over here.

And on 2026-09-17 the same rule took it back out, which is the most useful thing that has happened to this repository. Ten fresh draws per model, plus a second vendor for the first time: Opus went 10 of 10, Sonnet went 10 of 10, and gpt-5.6-sol went 10 of 10. Sonnet's two losses were the only thing holding the task in a numbered set, and they did not repeat. Cumulatively that is Opus 25/25, Sonnet 18/20, gpt-5.6-sol 10/10 — two entries at the ceiling, so _check_admission refuses it and it is warmup now.

The consequence is stated plainly because burying it would be worse: v2 therefore has no laptop task at all. Its laptop line reads no members, not 100%. The alternative was to leave the second vendor out of the calibration blocks, which would have kept the task in the set and would have meant choosing which evidence counts after seeing which way it went — the exact move L21, L35 and L43 are each about. The laptop tier is saturated at the top for two labs' frontier models, and that is a measurement rather than a mood. docs/ROADMAP.md §11.

And on 2026-10-28 the same rule did the same thing to the CUDA task, so v2 is two Metal tasks now. fused_rmsnorm_kernel was finished at ten draws per model across both vendors on a rented RTX 4090 — the last measurement standing between the set and a launch. It came back Opus 5/10 (all five losses the single-column edge, the August signature holding) and Haiku 5/10, but claude-sonnet-5 10/10 and all three OpenAI models 10/10. The top two measured rates are both 100%, so _check_admission refuses it exactly as it refused flash-attention, and v2 loses its CUDA member too. What is left under the headline is two Metal tasks — no laptop member, no CUDA member — and the honest reading is that this set's discrimination is Apple-silicon-only until a harder task is written. docs/LESSONS.md L55. The uncomfortable half is L46 wearing a new hat: gpt-5.6-luna, the cheapest model asked, going 10/10 is half the reason the top two saturate, and a task Opus still fails half the time is called warmup because of it.

Every draw behind those numbers is checked in under calibration/ and leaderboard/, and tools/check_calibration.py re-derives all forty-eight entries from the 362 files on every push. A figure that decides whether a task is allowed into a frozen set is not allowed to be the one figure nobody can reproduce.

One of those figures was re-drawn on 2026-09-04, and why is worth a paragraph. This repository is developed on an M1 Pro where every other process that touches torch.mps turns out to be blocked for the whole of its life — no custom shader and no failing test required, and nobody knew until it was measured. Under that fault a graded run could be killed at its time limit and recorded timeout: evidence, and a failure, when in fact nothing about the model had been measured. Exactly one draw in the published metal_cross_entropy_kernel evidence is such a timeout. It is still counted as a failure, because dropping a draw after seeing which way it went is how a benchmark starts choosing its own evidence, and because that direction can only understate a model rather than flatter one. Thirty fresh draws were taken with runner/mps_stall.py guarding every graded process — sixty gradings, no timeout, no mps_stalled — and the two Metal tasks now carry twenty and ten draws per model respectively. docs/LESSONS.md L42 and L43 are the account.

All three were built the way the design document asked for: the obvious answer forbidden, a property that is provable rather than plausible, mechanism that composes. speculative_decoding_verify never states the acceptance rule, only the guarantee it exists to provide, and grades the emitted tokens against the target distribution over two hundred thousand draws. flash_attention_backward hands the graded function one block of keys and never the rest, so the correction term that belongs to a whole query row cannot be accumulated from the block in hand. activation_checkpointing_rng compares generator state byte for byte, so an implementation that recomputes correctly and leaves the random stream rewound fails while its gradients are perfect. All three catch every wrong implementation written against them — twenty-eight mutants, twenty-eight caught.

And Opus 5 passed all three, forty draws out of forty. The other two did not, and the shape of how they failed is the part worth reading. On speculative_decoding_verify, Haiku's wrong versions run, accept at plausible rates, and emit plausible tokens — what fails is the distribution those tokens are drawn from, which no comparison against a reference array on one input would ever have noticed. On flash_attention_backward the three models fail at three different depths:

	fails	what the failure is
claude-opus-5	0 of 15	nothing
claude-sonnet-5	2 of 10	the decode-time query offset, the same two tests both times
claude-haiku-4-5	16 of 19	the row correction, or the 1/sqrt(d) scale

Sonnet's 2 failed, 47 passed is the signature of a mutant written for this task weeks before Sonnet was asked anything — queries not put at the end of the key range. A real model reproduced one of the invented wrong implementations, exactly, twice.

The fourth task, and what it taught the criterion

custom_autograd_double_backward (2026-08-21, 116 tests) is a hand-written autograd.Function whose backward has to be differentiable in turn. Opus 10/10, Sonnet 10/10, Haiku 0/10 — refused, warmup.

The refusal is not the interesting part. Haiku failed the same seventeen tests of 116 in every one of ten draws, and they are two test functions: the second-derivative ones. Everything else passed every time — forward, first-order gradients, broadcasting, return arity, the in-place detection. Ten draws out of ten saved the forward's sigmoid and reused it in the backward, where it arrives as a constant because a Function's forward runs with differentiation switched off. That substitution is mutant three for this task, written before any model was asked; this is the second time a real model has reproduced one of them verbatim.

And it sharpened the design rule. "The difficulty is an obscure fact about a tool" was too generous a sentence: PyTorch has a page about double backward in custom Functions, so every model at the top has read it. half being a type name in Metal Shading Language has no page, because it is not a topic — you learn it from a compiler. The question is now written down as does the framework have a page about exactly this mistake?, and it is answerable before spending anything on a calibration. docs/LESSONS.md L37.

The fifth task, and the half of the criterion that does the work

chunked_batchnorm_backward (2026-09-04, 44 tests) is batch normalisation's backward pass over micro-batches the caller chose. It was written against the other criterion — the one the laptop tier's single discriminating task actually satisfies — where difficulty comes from the shape of the interface rather than from an obscure fact: the graded function is handed one micro-batch and never another, so the two correction sums it needs, which run over the whole batch, are not available to it. Opus 10/10, Sonnet 10/10, Haiku 2/10 — refused, warmup.

The reason it failed to discriminate is worth more than the task, and it is a mistake in the criterion rather than in the code. The state the chunk was missing arrives as three of the function's own arguments, so nothing has to be reconstructed and one line of algebra is left. flash_attention_backward hands its block function the forward's output and the incoming gradient, and the difficulty is recognising that their row-wise dot product is the correction — a quantity nobody passes it. A decomposition the caller chose is necessary and not sufficient; the whole-unit quantity has to be recovered, not received.

The failure shapes say it independently, and the pass rate cannot. All eight of Haiku's losses fail in the case where the batch is a single chunk, which is exactly where the chunked mistake is correct by construction — so not one of the eight is the mistake the task was built around. The task measured arithmetic care and never once measured the thing it is named after. docs/LESSONS.md L44.

The first task the rule let in on its own numbers

metal_cross_entropy_kernel, written on 2026-08-13 against the corrected criterion — the difficulty is a fact about a tool, and an obscure one — is the first task in this repository that a frontier model fails.

Model	Passed	Cost per draw
claude-opus-5	8 / 10	$0.078 to $0.108
claude-sonnet-5	4 / 10	$0.033 to $0.091
claude-haiku-4-5	1 / 9	$0.006 to $0.117

Three models, three rates, in the order the models are usually put in — which no other single task here manages. The kernel is a Metal source string compiled at runtime by torch.mps.compile_shader, so the tier that needed a rented GPU box now has half of itself reproducible on any Apple silicon Mac. The harness compiles the string and dispatches it, one threadgroup per row, at a threadgroup size the kernel never chooses: from 1 to 1024, narrower than the row or wider than it, not necessarily a power of two, not necessarily a multiple of the simdgroup width.

And the failure has a name. Every one of Opus's two failures and five of Sonnet's six are the same compile error, on the same identifier:

E   SyntaxError: program_source:39:14: error: cannot combine with previous 'type-name' declaration specifier
E           uint half = (n + 1u) / 2u;
E                ^

half is a type in Metal Shading Language, so it cannot name a variable — and the reduction underneath it is correct in every one of those draws. I made the identical mistake in the first draft of the reference, an hour before either model was asked (docs/LESSONS.md L34). Haiku fails a third way: six draws call Metal functions that do not exist, and one wrote a fold whose stride never reaches zero, hung the GPU, and was killed by the task's own time limit.

That is exactly the difficulty L30 says survives, and exactly the difficulty L30 says is a moving target. This row is a measurement of one day. The full failure tables are in leaderboard/.

Forty out of forty is the result. Not a disappointment: it is the first number this repository has produced that it could not have produced by asking the author how hard something felt, and the tasks it refuses are ones their author spent a day on. What it says is in docs/LESSONS.md L30 and it is uncomfortable — the property that makes the one discriminating task hard is not on the list these three were built from, and naming it correctly makes the laptop tier's remaining difficulty look like unfamiliarity rather than depth.

Honesty about contamination

Any public benchmark leaks into the next training run. Pretending otherwise is worse than the leak.

Here "hidden" means hidden from the model at solve time, not secret. The prompts are public, the hidden tests are public, and anybody can read both — which is the price of the reproducibility promise above, because a benchmark you cannot inspect is a benchmark you have to trust. Every number here therefore has a shelf life.

So, the policy, in two parts and no more than the project actually does:

1. Every row is dated, and every published set's membership is frozen. A result records the task-set version and the date it was run, and which tasks a set contained when it was measured is pinned in leaderboard/SET_MANIFESTS.yaml: v1 names the eight laptop tasks it was measured over, forever, even after three of them move to warmup. A task's frozen_set label is mutable — it is where a task sits now — and set membership is not; the two were one field until they drifted apart, and a test re-derives the manifest from the published files so they cannot drift again unseen. You get to watch the evaluation drift without the historical rows moving under you. This is in force now.
2. A small private holdout is intended, and does not exist yet. The plan is a handful of tasks calibrated exactly like the public ones and never published, used only to check whether a public score has drifted away from a private one. It is scheduled after v2's first full sweep is out. Until those tasks exist this paragraph is a statement of intent and nothing else — claiming a holdout that does not exist would be worse than having none, because it would offer exactly the reassurance the reader cannot check.

The honest summary is that a score here is evidence about a model on a date, and the further you get from that date the more it is a claim about the training data instead. docs/LESSONS.md L30 is the entry where that stopped being theoretical.

Honesty about everything else

• Every generation is one independent attempt, not a converged number. A leaderboard row is the summary of however many draws it says it is, and a single draw is a sample: the same model on the same task passed one run and failed the next, four hours apart, same prompt, same settings. The harness is pinned to be deterministic; the model is sampled, and none of the pinning reaches it. Two models one task apart are not distinguishable, and this repository will not pretend otherwise until there are repeated runs. --repeat N runs N independent sweeps and report --variance prints how many of them each task passed; the two rows above predate it and have no error bar.
• The headline rate is the laptop tier and nothing else. Not an average over the tiers that happened to run: the accelerated tier is reported in full beside it, never inside it, and the results file names which tier the percentage came from.
• One attempt per task. A model that passes on the eleventh try is measuring the scaffolding, not the model. The number is written into every results file so nobody has to take it on trust. Repeated independent runs are a different thing from retries, and welcome.
• No sampling, thinking or effort knobs are set. Every model is asked at its own defaults, deliberately: a score at one effort setting and a score at another are not comparable, and nothing in a results file would say so.
• No server-side fallbacks. If the model that answers is not the model that was asked, the run raises rather than publishing one model's work under another's name.
• docs/LESSONS.md is the other half of this repository. Fifty-six entries, first person, newest first: what I expected, what happened, and what it cost. Including the ones where a wrong number nearly shipped and something mechanical caught it. A benchmark that is wrong looks exactly like a benchmark that is right, which is why the mistakes are documented rather than quietly fixed.
• A correctness benchmark cannot see a slow kernel. metal_cross_entropy_kernel grades the answer, and a kernel that ignores its threadgroup entirely — one lane, a serial loop over the row, no reduction — passes all sixty-eight tests. The only assertion that would catch it is a wall-clock one, and those are banned here in every tier because timing on unknown hardware is not reproducible. The degenerate kernel is checked in as a mutant with SURVIVES as its expected verdict, so the hole is a re-runnable fact rather than a footnote, and all ten Opus draws wrote a real threadgroup reduction anyway. docs/LESSONS.md L33.

What this repository does not contain

No course materials, no assignment code, no test suites from anyone's curriculum. Every task here is written from scratch for this repository. Inspiration is not redistribution, and the difference matters.

Who is writing this

I am Ahmet Selim Fedakar, a computer science undergraduate in Los Angeles. I co-founded EXCAR and Auris, two AI voice startups, where I build the machine learning and speech systems; separately I have shipped two apps to the App Store on my own.

This came out of working through Stanford's CS336 alone, and one recurring question: when a model tells me my implementation is wrong, is it right?

Now there is a way to find out.

License

MIT.

---

Bir başka gün, bir başka yerde, bir başka zaman ve bir başka mekânda, tekrar görüşünceye kadar kendinize çok iyi bakın.

Another day, another place, another time — take good care of yourselves until we meet again.