[grabbou]

Benchmarks Qwen3 0.6B on the on-device runtimes we work with, on one iPhone 17 Pro (iOS 27.0), to answer whether Core AI is as fast as MLX in practice. Results, methodology and run counts are in harness/benchmarks/RESULTS.md, and the raw data, one line per launch, is in harness/benchmarks/results/.

Results

Qwen3 0.6B, 256 output tokens, greedy decoding, thinking off. Each launch is 1 warmup and 3 measured runs per prompt. The table shows the median over launches that stayed at a nominal thermal state.

Runtime	Compute	Launches	Decode, 27-token prompt	Decode, 1k-token prompt	Prefill, 1k prompt	Peak memory above app baseline
Core AI, macOS export (dynamic 4-bit)	GPU	2	176 tok/s	135 tok/s	4,410 tok/s	264–1,090 MB
mlx-swift-lm 3.32.3 (4-bit)	GPU	3	163 tok/s	108 tok/s	4,821 tok/s	489–716 MB
ExecuTorch, MLX backend (int4)	GPU	1	150 tok/s	120 tok/s	4,799 tok/s	408–549 MB
llama.rn, Q4_0	GPU	1	137 tok/s	112 tok/s	3,443 tok/s	994–1,108 MB
Core AI, iOS export (static 4/8-bit)	Neural Engine	2	137 tok/s	77 tok/s	6,531 tok/s	1,292–1,428 MB
ExecuTorch, XNNPACK (8da4w)	CPU	1	83 tok/s	42 tok/s	562 tok/s	965–1,052 MB

Spread:

• Measured runs within a launch are within 3% of each other (4% for XNNPACK).
• Nominal launches agree within 2%.
• llama.rn and ExecuTorch also have one throttled launch each, listed separately in RESULTS.md.

Core AI cold start:

• Neural Engine export: on the first launch it waits about 103 s for its first token (2 launches). The compile happens on the first prediction, not in prepare(), and specialize() takes under 1 ms without changing that. After that, every load still waits 2.6 s for the first token.
• GPU export: about 2 s on the first launch and 0.12 s afterwards.

Is Core AI as fast as MLX?

• On the GPU, with a dynamic-shape export: yes. It decodes 8% faster on a short prompt and 26% faster at 1k tokens.
• The iOS export on the Neural Engine: it prefills fastest, but decodes 16–28% slower than MLX, starts much slower and uses 2–3× the memory.

RESULTS.md explains why: decode is one token per step, the export uses static shapes, and it keeps some weights at 8 bits.

What's in here

• harness/benchmarks/:

  • a harness benchmark per configuration: Core AI on the Neural Engine, Core AI on the GPU, Core AI cold start, llama.rn, and ExecuTorch with its MLX and XNNPACK backends;
  • shared measuring code (measure.ts) and the same prompts for everyone (prompts.ts);
  • run.sh, which runs the configurations one app launch at a time;
  • collect.ts, which builds the tables: the median over launches, with throttled launches left out unless --all is passed.

  The benchmarks only run with HARNESS_BENCHMARK=true.

• harness/benchmarks/mlx/: a minimal Swift app on mlx-swift-lm, built with XcodeGen, that measures the same things the same way.

• apps/expo-example/modules/device-stats: a local Expo module that reports the process footprint (with a sampled peak), the thermal state and Low Power Mode.

• react-native-executorch added to the example app. It's linked only with EXECUTORCH=1, and its native libraries are downloaded by hand, as the README describes.

Methodology notes

• Process isolation: each configuration runs in its own app launch, with loads measured 4 times.
• Thermal waits: each stage waits up to 5 minutes for a nominal thermal state, and records the state before and after. A pass without those waits throttled the phone to a third of these speeds and is not used.
• Builds: the React Native runtimes run in a debug build with native code compiled with optimization. MLX runs in a release build of its own app.
• Memory: it's compared above each app's baseline (about 200 MB for the example app, 10 MB for the MLX app).

Follow-ups

• More nominal launches of llama.rn and ExecuTorch, which needs longer cooldowns between configurations.
• Qwen3 4B.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
ai	Ready	Preview	Oct 27, 2026 1:41pm UTC

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.