Repository navigation
Add AI/ML API provider - #9158
Closed
hugoaimlapi wants to merge 37 commits into
Closed
hugoaimlapi wants to merge 37 commits into
hugoaimlapi wants to merge 37 commits into
Conversation
AI/ML API is an aggregator exposing many creators' models behind one OpenAI-compatible key, so it gets a sync provider rather than a hand-written model set. The public catalog needs no key. Only entries served as openai/chat-completions are taken, and of those only the ones whose output is purely text — an id can appear under several endpoint types, and its image or audio surface is a different product. Pricing is quoted as price-per-N-tokens and keyed by origin (provided / generated / cached), which maps onto input / output / cache_read. 163 of the 338 text-output chat models are published here. The rest are skipped by skippedNotice because the catalog does not yet expose a description or an output limit for them; inventing either would be worse than an absent entry. deleteMissing is off since the catalog lists far more than the chat surface, so one response omitting a model is not proof it is gone. Capability flags the catalog does not report (reasoning, tool_call, structured_output, open_weights) keep any value already in the repo instead of being overwritten with a guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add AI/ML API provider with catalog sync
…ng them
AI/ML API hosts other people's models and authors none of them, so every
entry has to point at the lab file. AGENTS.md makes that a blocker
("Non-lab hosts use base_model"), and the previous full inlines both
violated it and drifted from the repository's own data: 65 models
disagreed with their lab entry on reasoning, 75 on tool_call, 37 on
structured_output, 17 on attachment. openai/gpt-5 shipped here as
reasoning = false, tool_call = false while models/openai/gpt-5.toml says
both are true.
The cause was a fallback of the shape `existing?.reasoning ?? false`.
`existing` only ever reads this provider's own directory, which does not
exist for a new provider, so the guard never applied and every model was
written as false. The entries now carry no capability flags at all, so
there is nothing left to contradict.
What a file states is now only what is genuinely ours: cost, a limit when
it differs from the base, and reasoning_options. The limit no longer sets
`input`, because the catalog publishes a context window and an output cap
but no input cap, and equating input with the whole context overwrote the
lab's correct split (272k in + 128k out inside a 400k window).
reasoning_options is required whenever the base reasons, and no lab entry
carries it (0 of 245). It is read per model from the API's own published
request schema at /docs-json, so the declared control is the one the
endpoint actually accepts rather than an assumption.
Coverage is 108 of the 338 text-output chat models. 192 are skipped
because this repository has no lab entry to point at — authoring complete
lab files for them would mean sourcing capability metadata the catalog
does not publish — and 38 because their base reasons but the API does not
document the control. Both groups are reported through skippedNotice.
Verified: bun run validate exits 0 and resolves aimlapi/openai/gpt-5 to
reasoning = true, tool_call = true and the lab limit split, with our
price. bun test is 268 pass / 4 fail, the same four failures a pristine
tree produces. Re-running the sync reports 108 unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reference lab entries with base_model instead of inlining them
… from per-model schemas Addresses the review on this PR. Two separate causes, one in this sync and one upstream in the host's API. Image surfaces. `isChatTextModel` judged each record on its own modalities, but the catalog lists an id once per endpoint type and the CHAT record of an image model claims text output: `google/gemini-2.5-flash-image` appears both as `openai/image-generations` with `output = ["image"]` and as `openai/chat-completions` with `output = ["text"]`, and the same holds for the `gemini-3-pro-image` and `gemini-3.1-flash-image` families. The answer is not in the record, so the whole response is indexed first and an id this host serves on any media surface is refused regardless of what its chat record claims. The six image entries this admitted are deleted; `deleteMissing: false` retains files that merely fall out of the source, which is why removing them takes an explicit delete. Reasoning options. The uniform `["none", "low", "medium", "high"]` was not invented here — it was read faithfully from a host schema that published one boilerplate enum for every model. The host has since corrected that, and its published schemas now carry seven distinct ladders. Re-running the sync therefore produces per-model values: `none` disappears from GPT-5, Qwen3.8 2.4T A95B and Kimi K2 Thinking, and two models that previously documented no control at all (`alibaba/qwen3.6-max-preview`, `z-ai/glm-5.3-flash`) now do. Known remaining gap, stated rather than papered over: the host's per-model schema endpoint still understates GPT-5, publishing `low/medium/high` where the model demonstrably accepts `minimal` as well. That is a fix on the host side, not here; until it lands, this sync can only publish what the per-model schema states.
The per-model request body is a union of per-family variants, and more than one of them can describe `reasoning_effort` with a different ladder. The reader returned the first occurrence it walked into, which is whichever variant happens to sit earliest in the document rather than what the endpoint accepts. Measured against the live schemas, that dropped a real value on 9 of the 80 reasoning models: openai/gpt-5, -mini, -nano lost minimal anthropic/claude-sonnet-4.6 lost max anthropic/claude-opus-4.8 lost max google/gemini-2.5-flash-lite lost none google/gemini-3.6-flash lost none google/gemini-3.7-flash lost none z-ai/glm-5.3-flash lost minimal, medium The walk now visits every occurrence and unions them, and the result is ordered least-to-most effort instead of inheriting whatever order the schema happened to use. Regenerating also picks up 7 models added to the catalog since the last run: alibaba/qwen-plus, anthropic/claude-fable-5, anthropic/claude-sonnet-5, google/gemini-3.1-pro-preview, moonshot/kimi-k3, z-ai/glm-5-turbo and z-ai/glm-5v-turbo.
The previous commit took the union across the schema's `anyOf` variants,
which is right where every variant is a request shape the host will serve.
For two entries it was not: the union published values the API refuses.
Probed against production, reading the error body rather than the status
code — a low `max_tokens` returns 400 for an accepted value too, because
the model spends the budget on reasoning and returns nothing:
gemini-3.7-flash none -> "Reasoning is mandatory for this endpoint
and cannot be disabled"
gemini-3.7-flash minimal -> "Validation failed"
gemini-3.6-flash none -> "Validation failed"
So 3.7-flash serves low/medium/high/max and 3.6-flash adds minimal to
that. The other seven entries in the previous commit were re-probed the
same way and are correct as published.
…away The two gemini corrections in the previous commit lived only in the generated TOML, so the next `aimlapi:sync` would have read the schema and put the refused values straight back — silently, since nothing in the repository validates a published effort against the live endpoint. They now sit in the sync itself, next to the evidence for them: which values the endpoint served, when that was measured, and how, since the method is the part that is easy to get wrong. A reasoning model given a small `max_tokens` answers 400 for a value it accepts, so a probe that reads the status code and not the error body reports refusals that never happened. Only "Validation failed" — or gemini's explicit "Reasoning is mandatory for this endpoint and cannot be disabled" — is a real one. Regenerating now reports 0 updated for both entries.
The schema omits `max` for this model while offering it for the sibling opus-4.8, and the endpoint serves it on both — so the omission is in the description, not the capability. Understating costs a caller a level that works; it is the same class of error as the gemini entries, in the other direction. Probed the same way as the rest: every value sent to production with room in `max_tokens`, classified by the error body rather than the status code.
Six entries accept any string in `reasoning_effort` — `banana` and `zzz9` both return 200 — so the parameter is not honoured and the ladder beside it described a control that does not exist. Five of the six arrived with the previous regeneration, so this is a defect this branch introduced. They now publish `reasoning_options = []`: the model reasons, nothing about it is selectable. That is what the schema's empty list is for, and it is the honest answer where a ladder cannot be established at all. The check is cheap and worth repeating before trusting any of these numbers: send a deliberately invalid value. A 400 means the field is real and a probe of it means something; a 200 means the probe proves nothing, whatever the schema claims. All 87 entries carrying reasoning options were swept this way on 2026-09-09 and these six were the only failures — so the other 81 ladders in this PR rest on a control, not on the schema alone.
`-pro` and `-fast` ids factor onto the base model and inherited its display name, so the catalogue listed two rows called "GPT-5.6 Luna" and two called "Claude Opus 4.7" — distinct models, distinct prices, no way for a reader to tell which is which. The host already names them apart; the sync simply never carried the name through. It does now, and the override drops itself wherever the names already agree, so this adds a field only where one was missing. Six names were shared by thirteen ids before; the only remaining overlap is the dated gpt-4o snapshots, which the host itself calls "GPT-4o".
`anthropic/claude-opus-4.7-fast` is listed by `/v1/models` with a name and a full price table, and inference answers 404 "No endpoints found" for it. Publishing it hands a reader an id that fails on first use. Its siblings were checked the same way — claude-opus-4.8-fast and the three gpt-5.6 `-pro` aliases all answer 200 — so this is one broken row, not a broken family. The skip lives in the sync with that evidence beside it, and should be removed if the row starts working.
The check was added to the filter that decides which models get a schema probe, which is not the path that decides what ends up in the catalogue — so it prevented nothing. The entry stayed out of the last commit only because it had been deleted by hand. It now sits in `translateModel` and in the id listing, so an uncallable row is neither created nor updated. It still does not delete a file that is already on disk, because this sync runs with `deleteMissing: false`; the comment says so rather than implying more than the guard does.
Sets ten ladders to what the review asked for. They are kept in their own
table, apart from the measured ones, because they are not the same kind of
claim and should not be mistaken for each other later.
Stating the cost plainly, since it lands on callers rather than on us: the
review baselines a relay against the underlying vendor's own API, and this
host's accepted set differs in both directions. So these entries advertise
values the gateway refuses, and drop values it serves.
deepseek-v4-* now high,max — max and xhigh answer 400 here
gpt-5-pro now high — low and medium answer 200
gpt-5.4-pro now medium,high,xhigh — xhigh answers 400
o1, o3-mini xhigh removed — xhigh answers 200
claude-opus-4.x none removed,
xhigh added — none answers 200, xhigh 400
qwen3.8-max now low,medium,xhigh — none answers 200, xhigh 400
Probed against production on 2026-09-09, `max_tokens` high enough to leave
room for an answer, classified by the error body rather than the status
code — a reasoning model answers 400 for a value it accepts when the budget
runs out, so a status-only probe reports refusals that never happened.
Deleting an entry from that table restores the measured value.
Drops `REVIEWER_REQUESTED_EFFORTS`. The review is right that it was the wrong mechanism: it overrode live measurement with a vendor baseline, and for a relay that means publishing values this gateway refuses. `deepseek-v4-*` were advertising `max`, `gpt-5.4-pro` `xhigh`, `qwen3.8-max` `xhigh` — none of which appear in those models' request schema here, so a caller following the catalogue collects a 400. Every one of those ids now takes its ladder from the host again. The two rows the review was right about for a different reason stay at `low, medium, high`, and the reasoning is now recorded where it belongs. On `o1` and `o3-mini` this host answers 200 to `xhigh` — which is why a status-code probe put it in the catalogue and kept it there. Pinned to the direct hop on a prompt hard enough to show the difference, `xhigh` produced 0 reasoning tokens on o1 and 192 on o3-mini, against 10176 and 16000 for `high` and ~4500 for `low`. Accepted and dropped is worse advice than refused: the caller asks for more thinking, is told yes, and pays for a reply produced with less. `MEASURED_EFFORTS` gains both, and a note that `accepted` alone is not evidence a rung does anything. `claude-opus-4.8-fast` joins its 4.7 sibling as uncallable — absent from all 785 rows of `/v1/models`, and `/docs-json` 404s for it. Its file is deleted rather than left behind, since `deleteMissing: false` keeps stale rows alive. Each model is now published once. `/v1/models` returns a dotted id and a dashed one for the same model — `claude-opus-4.8` and `claude-opus-4-8`, `claude-sonnet-4.6` and `claude-sonnet-4-6` — because the dashed spelling is an alias its router also answers to. A sync that trusts the listing writes the model twice, which is what the previous run did before this guard existed. The rule is narrow on purpose: the dotted twin must be in the SAME payload and carry the SAME display name, so two genuinely different models that collide on spelling both survive. It catches 9 pairs and leaves 56 similarly shaped ids with no twin alone. Claude 4.7/4.8 keep `none` and still lack `xhigh`, which is what this host serves today: `none` answers 200, `xhigh` 400. That is a gateway bug rather than a catalogue one and the fix is merged upstream but not yet in production; when it lands, the schema will carry `xhigh` and this sync will pick it up with no table to edit. Sync is idempotent against production: 0 created, 0 updated, 110 unchanged. `validate` exits 0.
The gateway fix shipped, so the schema now carries what the review asked for and the sync takes it from there — no table to edit, which was the point of deriving these from `/docs-json` rather than pinning them. - `claude-opus-4.8` and `claude-sonnet-5` gain `xhigh`. `claude-sonnet-4.6` does NOT, and that is the control: Anthropic lists it for `max` and not `xhigh`, so a run that changed it too would have meant something else moved. - `openai/o1` loses its `name = "o1 Preview"` override. The host was mislabelling GA `o1` as the preview SKU — the review caught it from outside as an identity mismatch — and now that the host name matches the lab entry, the override has nothing left to override. - `openai/o1` and `openai/o3-mini` keep `low, medium, high`, now because the schema says so rather than because `MEASURED_EFFORTS` forces it. Those two entries are no-ops from here and can go once the values have held a while. Drops the `claude-opus-4.7` entry from `MEASURED_EFFORTS`. It was written when the schema omitted `max` that the endpoint served; with the schema corrected it had turned into the opposite problem, pinning a five-rung ladder over a six-rung one and hiding the `xhigh` production now serves. A pin that outlives the gap it patched is worse than no pin, since nothing fails when it goes stale. Also carries a DeepSeek price change that is NOT ours: production now reports `deepseek-chat` at 0.39 in / 1.56 out / 0.0078 cached per million, against the 0.364 / 0.546 / 0.0364 this catalogue held, and `deepseek-v4-flash` moves the same way. Read straight off `/v1/models` — the sync copies published prices and does not compute them — so the numbers are current rather than corrected. Sync: 0 created, 6 updated, 0 removed. `validate` exits 0 over 110 models.
Answers the review by measuring rather than by argument. Three of its items hold, three do not, and one of ours turned out stale. **Prices.** `cache_read = 0.0078000000000000005` was IEEE residue from `(price / per) * 1e6` serialized verbatim. `perMillion` now rounds to six decimals — finer than any published USD/MTok figure here — and 14 costs across the provider come out as stable decimals. **`moonshotai/kimi-k2-thinking` now publishes `[]`.** Measured on a prompt hard enough to show a difference: `low` produced 6723 and 8207 reasoning tokens across repeats, `high` produced 6508. `low` above `high` is not a delta in the wrong direction, it is the absence of an ordering. The lab entry says `[]` for the same reason — the model always reasons and the caller cannot steer it. That goes in a new `EFFORT_VALIDATED_BUT_INERT` rather than into `EFFORT_NOT_HONOURED`, because the evidence is a different kind and the two must not be confused. In the older set an invalid value returns 200, proving the field is never read. Here an invalid value is rejected — the field is read, parsed and enforced — and the levels still do nothing. **The existing `[]` entries were re-probed and hold.** The review argued they rested on "a single ignored OpenAI field" and might hide a control on another wire path. Five of six answer 200 to a deliberately invalid effort while answering 200 to the control: `gemini-3.1-pro-preview`, `gemma-4-26b-a4b-it`, `glm-5v-turbo`, `glm-5-turbo`, `qwen-plus`. A field that accepts `banana` is not being read, which is the affirmative evidence the review asked for. The sixth, `moonshot/kimi-k3`, no longer belongs there: its control answers 200 and `banana` answers 400, so the field IS validated now. Left untouched in this commit — swapping one unproven claim for another is the mistake this file exists to avoid — and being measured for whether its levels order. Three review items are not supported by measurement and stay as they are: `gemini-3.7-flash` `max` moves 2131 -> 3841 -> 6585 across low/high/max, so it is honoured; DeepSeek V4 orders 4374 -> 4936 between `low` and `high` with the field validated, so its ladder is real rather than boilerplate; and `none` on `claude-opus-4.8` produces 0 reasoning tokens against 191 at `low` and 1228 at `high`, so it is a working off switch on this host. `gpt-5-pro` could not be measured at all — every level, and the no-effort control, timed out at the gateway. Inconclusive stays inconclusive. `validate` exits 0.
Re-probing the six `EFFORT_NOT_HONOURED` ids turned up one that no longer belongs. `moonshot/kimi-k3` answers 200 to the control and 400 to a deliberately invalid effort, so the field is read, parsed and enforced — the one thing `reasoning_options = []` asserts is not true. It goes back to the ladder its own card declares. What is NOT claimed: that the levels order. `low` measured 919 and 1618 reasoning tokens across repeats and `high` measured 1943, so the spread inside `low` is nearly the whole distance from `low` to `high`, and `max` timed out at the gateway both times. That is too thin to assert an ordering and too thin to deny one, so the entry rests on the host's own declaration rather than on a claim of ours the measurement cannot carry. The other five hold and stay: `gemini-3.1-pro-preview`, `gemma-4-26b-a4b-it`, `glm-5v-turbo`, `glm-5-turbo` and `qwen-plus` each answer 200 to the control AND 200 to `banana`. `alibaba/qwen3.8-max` remains unmeasurable rather than agreed or refused. At `max_tokens: 4000` it spends the entire budget on reasoning; at 12000 every attempt times out at the gateway. Four runs, no usable pair, so the review's item on it is neither confirmed nor answered and the card is left alone. `validate` exits 0.
Three items across the last two rounds ask for `toggle` and `budget_tokens`
options because the labs behind these models expose them and a relay should
mirror the control family. The premise does not survive a control.
Probed against `deepseek-v4-pro`, `qwen-plus`, `kimi-k3` and `glm-5-turbo` on
the OpenAI-compatible path, each beside a no-field control that answered 200:
enable_thinking: false 200
thinking: {type: 'disabled'} 200
thinking_budget: 128 200
banana_toggle: true 200 <- a field that exists nowhere
An invented key is accepted exactly like the real ones. This surface discards
unknown top-level fields instead of forwarding them, so a 200 on
`enable_thinking` says nothing was listening — it is not evidence of a toggle.
Publishing one on that basis would hand callers a control the gateway throws
away, which is the failure this provider has spent four rounds removing.
`reasoning_effort` is different in kind, and that difference is the whole
method here: on the models that read it, an invalid value is REJECTED. That is
how a live control is told apart from a swallowed one.
`google/gemini-3.1-pro-preview` keeps `[]`, now on the token-scaling evidence
the review asked for rather than on status codes: `low` measured 3837 and 3841
reasoning tokens, `high` measured 3837 and 3838. Four runs, a spread of four
tokens, no movement between levels. Peers publish L/M/H for this id; this host
accepts the names and steers nothing.
Two items from the same rounds are already answered by the tree. `kimi-k3`
ships `["low", "high", "max"]`, not `[]` — that comment predates the commit by
two minutes and the later round sees the corrected file. And `gpt-5-pro` is
reachable after all: the earlier timeouts were the prompt, not the model. It
answers `low` and `high` on a light request, so the set is being measured
rather than trimmed to the lab baseline on the strength of a timeout.
No TOMLs change. `validate` exits 0.
Both were asked about on the ground that the lab publishes a narrower set. Measured, they answer differently, which is why the baseline could not settle either one. `gpt-5.4-pro` loses `low`. On a prompt hard enough to separate the levels it measured 2230 and 2273 reasoning tokens against `medium`'s 2180 and 2420 — overlapping ranges across repeats — while `high` sat clearly above at 2984. Two names for one behaviour is worse than one name. The review wanted this too, on the different ground that the lab omits `low`; the measurement is what the entry rests on. `xhigh` stays out: this host's schema stops at `high`. `gpt-5-pro` keeps `low, medium, high`. The review asked to trim it to `high` alone because an earlier round reported every probe timing out. Those timeouts were the prompt, not the model: on a lighter request it answers, and the levels order — 64, 256, 320 reasoning tokens across low/medium/high, and 128 against 192 on a repeat of the two ends. Trimming would take two working rungs away from callers to match a set this host does not have. That is also the correction to my own earlier note. "Inconclusive stays inconclusive" was right as a rule and wrong as a conclusion here — the probe had been built so that it could only ever time out. `validate` exits 0.
Follows the review on the ids where it and the measurements disagree, and records that this is a decision rather than a discovery. `REVIEW_BASELINE_EFFORTS` is kept apart from `MEASURED_EFFORTS` for that reason: one table says what was observed, the other says what was chosen. Six ids move to the lab/peer control family — `deepseek-v4-pro` and `-0813` to `high`, the Flash pair to `low, high`, `qwen3.8-max` to `low, medium`, `gpt-5-pro` to `high`. Every value is intersected with what the host accepts first, which is the one difference from the table this replaces. That earlier one took the baseline verbatim and published `max` on DeepSeek V4 and `xhigh` on `qwen3.8-max`, both of which answer 400 here, so a caller following the catalogue collected an error. Nothing here is a value this gateway refuses. What is given up is written down next to it. `deepseek-v4-pro` orders 4374 to 4936 reasoning tokens between `low` and `high` with the field validated, and `gpt-5-pro` ordered 64/256/320 and again 128/192 across repeats — both keep fewer rungs than those probes support, and callers lose working levels. Deleting the entry restores them. `gemini-3.7-flash` keeps `max`, now with the evidence recorded where the review asked for it: 2131 reasoning tokens at `low`, 3841 at `high`, 6585 at `max`. Also records why `deepseek-chat` and `deepseek-v4-flash` share prices, since that question cost a round: DeepSeek retired the former and routes the id to its Flash build, so the host bills it on the Flash grid. The check was still worth running — it turned up the row's stale V3 identity, which the host has since corrected. Sync: 6 updated. `validate` exits 0.
**`deepseek-chat` points at the Flash lab entry.** DeepSeek retired the model behind that name and routes the id to its current Flash build, so the chat lab entry was the wrong parent in every field that matters: a 128K window for a model with 1M, and `reasoning = false` for one that reasons. Its stale "DeepSeek V3" label is suppressed too — the host still publishes it while serving Flash, and republishing that beside a Flash `base_model` would put a contradiction in the catalogue. The upstream correction is merged; when it lands the suppression can go, because the host will be saying it itself. **DeepSeek V4 leaves `REVIEW_BASELINE_EFFORTS`.** The previous round asked for the lab set, this table gave it, and the next round objected that doing so "removes caller off/lower rungs that the host reportedly accepted" — which is what that entry's own note had said it cost. Re-probed on the objection: an invalid value is rejected, `none` returns 0 reasoning tokens across two runs, `low` returns 4175 and `high` 5662. An ordered ladder with a working off path, so the ids go back to the host's own set. **`qwen3.8-2.4t-a95b` keeps `high`.** The review asked for `xhigh` from the peer ladder; probed twice, it answers 400 with `Expected 'low' | 'medium' | 'high'`. Publishing it would hand callers an error. Whether the rungs order is still unmeasured — `high` times out at the gateway — but the enum is settled. **`none` on Claude is documented rather than dropped**, and the answer is per model because the routing differs. On `opus-4.7`/`4.8` every rung up to `high` reaches OpenRouter, where `none` measured 0 reasoning tokens against 191 at `low` and 1228 at `high` — an off switch. `sonnet-5` has no OpenRouter link and Anthropic has no off switch on adaptive models, so `none` there now maps to the cheapest real rung after an upstream fix; before it, dropping the field handed the request to Anthropic's default and returned an empty reply for a full budget. Both are worth publishing; they are not the same promise. Sync: 5 updated. `validate` exits 0.
…ladder **`claude-sonnet-5` loses `none`, and the review is right about why.** Every rung on that id reaches Anthropic natively, Anthropic has no off switch on its adaptive models, and the host therefore maps `none` to `low` — the cheapest real rung. Publishing it as `none` tells a caller the catalogue offers a way to turn reasoning off here, and what they would get is the minimum instead. `opus-4.7` and `4.8` keep `none`: their lower rungs reach OpenRouter, where it measured 0 reasoning tokens against 191 at `low` and 1228 at `high`. Same parameter, same family, different link, different promise — which is why this had to be decided per id rather than per family. **DeepSeek V4 is measured through.** `medium` was the last untested rung and the review objected to keeping it, so it was probed: 5563 reasoning tokens, between `low`'s 4175 and `high`'s 5662, with `none` at 0 across two runs and an invalid value rejected. Four rungs, in order, with a working off path. The "invented GPT-style ladder" named in three rounds is the one this host has. **`alibaba/qwen3.8-max` leaves the baseline table**, on the review's own ground: forcing `low, medium` dropped `high` without ever measuring a rejection of it, and `high` answers 200 and spends its whole budget reasoning. A rung that works is not ours to remove. Whether its rungs order remains unmeasurable — the model exhausts 4000 tokens on reasoning and times out at 12000 — and `xhigh`, the lab's top rung, answers 400 here. **`deepseek-chat` is finished.** Its `[limit]` still carried the old row's 128K context and 124K output beside a Flash `base_model` with 1M; the stale-row suppression now covers limits as well as the name, so the lab figures inherit until the host's own correction ships. Sync: 2 updated. `validate` exits 0.
…ek medium **`REVIEW_BASELINE_EFFORTS` is now empty.** The review offered a choice on the last entry — delete the `gpt-5-pro` pin or mark it an intentional lab match — and deleting is the better half. The pin held `["high"]` over a host that measured 64/256/320 reasoning tokens across low/medium/high and 128/192 on a repeat of the ends, so every sync discarded two working rungs by decision. The table stays in the file, empty, because the next round may want it again. **`alibaba/qwen3.8-max` loses `none`.** It is accepted, but nothing shows it turns reasoning off: two runs returned no `reasoning_tokens` field at all while still producing 1703 and 2000 output tokens, and `high` on the same model does report 2000. A rung whose semantics cannot be shown does not belong in a catalogue other people build against. **`deepseek-chat`'s retarget is confirmed rather than inferred.** Three calls to that id each answered `"model": "deepseek-flash"`, the same string `deepseek-v4-flash` returns. The review asked for exactly this check. **DeepSeek `medium` stays, and the note says why it is unsettled.** The objection was fair on one sample: 5563 against `high`'s 5662 is under two percent, and that same reasoning removed `low` from `gpt-5.4-pro`. Re-probed, `medium` came back 7204 — its own spread across runs is thirty percent, an order of magnitude wider than the gap being argued about, and `high` timed out on three of four attempts. The two cannot be shown to order and cannot be shown not to. Removing a rung the host accepts needs evidence, and noise larger than the effect is not evidence in either direction. `validate` exits 0.
Nine ids, an invalid-effort control on each, and the two questions the review raised for them. No TOML changes: every published set already matches what the host accepts, and nothing survived as a reason to change one. **The rungs the review wanted added are refused.** `xhigh` answers 400 on `gpt-5.2-pro`, `gpt-5.5-pro` and `gpt-5.3-codex`; `max` answers 400 on `gpt-5.6-luna`. Publishing the lab's wider sets would hand callers an error, which is the rule the review itself opened with. **The rung it worried was fake is real.** `none` measured 0 reasoning tokens on every id that offers it — `gpt-5.3-codex`, `gpt-5.6-luna`, `-sol`, `-terra`, `-terra-pro` — against non-zero at `high` on each. That is the same check `claude-sonnet-5` failed this morning, which is why it was worth running rather than assuming the family behaves alike. Ordering, on a light prompt so the batch could finish: `gpt-5.2-codex` 90/192, `gpt-5.1-codex` 0/103, `gpt-5.6-terra` 9/25 and `-terra-pro` 91/129 all separate; `gpt-5.6-sol` barely does at 0/9; `gpt-5.6-luna` (26/27) and `gpt-5.5-pro` (33/31) do not, and `gpt-5.2-pro` produced no reasoning at any rung at all. Those last three are left alone rather than trimmed. `gpt-5.2-pro` is the `kwaipilot` case this file's Method section already names: when the control produces nothing, the cells say nothing about the ladder. The other two need a harder prompt than the batch could afford, and a rung the host accepts is not removed on a probe that separated nothing. `validate` exits 0.
The review asked whether this gateway returns reasoning on a separate field. It does, and the catalogue never said so. The lab entries that declare `interleaved` are not inherited through `base_model`, so every aimlapi row built with it unset while the provider was emitting 20K characters of chain-of-thought beside the answer — 20114 on `deepseek-v4-pro`, 14150 on `v4-flash`, 6506 on `kimi-k3`. A client reading only this provider had nothing telling it that field was not the reply. Probed across all 81 ids that carry a reasoning ladder, by reading the keys actually on the response message rather than by family: 14 return `reasoning_content`, 44 return `reasoning_details`. Both are now declared, 58 files updated. Fifteen more return a bare `reasoning` and two Gemini Flash ids return `extra_content`. Neither name is in the schema's enum, so they stay unset and the map says why — most of that group is the Anthropic line, where the field carries the summarised thinking block. Two ladder corrections come with it, and both are this file being made consistent with itself rather than new findings. `gpt-5.2-pro` and `gpt-5.5-pro` drop `low`. Re-probed on a hard prompt at the review's suggestion, since the light one had `gpt-5.2-pro` not reasoning at all: `low` and `medium` come back indistinguishable on both — 1400 against 1311, 2345 against 2336 — while `high` sits above where it could be measured. `deepseek-v4-pro` drops `medium`, reversing a call made earlier today. It measured 5563 and then 7204 against `high`'s 5662, a spread inside the rung wider than any gap to the rung above, and that was recorded as "unsettled, so keep it". But publishing a rung is a claim too, and `gpt-5.4-pro` had the identical shape and lost `low` for it. Two rules for one situation is worse than either rule. `none` (0 twice) and `low` (4175 against 5662) stay measured. Sync: 58 updated. `validate` exits 0.
…d reasoning The review asked why `deepseek-chat`, the base `gpt-5.6-luna`/`-sol`/`-terra`, `o1` and `o3-mini` were left out of `INTERLEAVED_FIELD` when their `-pro` siblings got `reasoning_details`. Fair question: the sweep that placed them ran on a light prompt, and "no field" on a prompt the model did not reason on proves nothing. Re-probed with a prompt that forces reasoning and `effort: high`. They reason — `o1` spent 2500 tokens on it, `gpt-5.6-luna` 1023, `-sol` 1142, `-terra` 862 — and the message still carries nothing beside `content`. The count is in `usage`; the text is not exposed. The `-pro` siblings reach OpenRouter, which surfaces a channel; the base ids reach OpenAI natively, which does not. Same family, different link, different wire. `deepseek-chat` is non-thinking by default and produced no reasoning tokens at all, so there is nothing to declare on it either. `deepseek-v4-pro-0813` drops `medium`, measured rather than copied from its sibling: `medium` exhausted a 4000-token budget twice while `high` used 2318. A rung that spends more than the one above it is not a lower rung. The Pro family now reads the same, and for the same reason on each id. Sync: 1 updated. `validate` exits 0.
…h the host Adopts the review's baseline rule after seven rounds asking for it. The resolver now reads the lab's own first-party provider entry for the resolved `base_model`, keeps only the rungs this host accepts, and publishes that. A rung the host offers but the lab does not is dropped even where it measured as working. The table that hand-pinned the same outcome for a few ids is deleted; there is one mechanism now, and a re-sync cannot reintroduce a rung the lab does not publish. Three things the rule needed to be correct rather than merely narrow: - **`none` comes back only by measurement.** The lab sets never carry it, so the intersection strips it; `NONE_IS_REAL_OFF` restores it on the eight ids where a probe showed zero reasoning tokens against a non-zero count above, with an invalid value rejected. `deepseek-v4-pro-0813` was measured for this today rather than copied from its sibling: 0 and 0. `claude-sonnet-5` and `qwen3.8-max` stay out on purpose — on one `none` maps to `low`, on the other it never reported a count. - **A dated snapshot borrows its base's ladder.** `deepseek-v4-pro-0813` has no lab entry and was falling through to the host enum, which left it wider than the model it is a copy of. It now strips the date and reads `v4-pro`'s. - **The lab's ladder lives on its provider entry, not the model record.** `models/<lab>/<id>.toml` has no `reasoning_options`; the first attempt read it and changed nothing. `providers/<lab>/models/<id>.toml` is the file a reviewer means by "the lab publishes". An empty intersection — every lab rung refused here — keeps the host's enum rather than emitting `[]`, since that would claim no control exists. And `MEASURED_EFFORTS` can still narrow the result for an accepted-but-inert rung, but never widen it past the lab. What this costs is recorded on `labEffortValues` so it can be reversed with evidence in hand: `low` on `deepseek-v4-pro` (4175 reasoning tokens against 5662), `high` on `qwen3.8-max`, `max` on `gemini-3.7-flash` (6585 against 3841 — the largest effect measured anywhere in this file), and `low`/`medium` on `gpt-5-pro`. Callers of those ids lose levels the host honours. The hard-prompt probe the review wanted for `gpt-5-pro` timed out three of three, so the lab set decides it either way. Sync: 22 files. `validate` exits 0.
…nd test it The last round reviewed the intersection itself rather than its outputs, and found two things wrong with it. **The lab lookup used the wrong directory for two labs.** `base_model` is spelled with the metadata lab (`zhipuai/glm-4.7-flash`, `meta/...`) while a lab's ladder lives under its provider directory (`providers/zai`, `providers/llama`). For every GLM and Meta id the lookup missed, returned `undefined`, and the host enum stood as if the lab had agreed to it. Now resolved through the same table the OpenRouter sync uses, exported as `providerDirForMetadataLab`. **Flash lost its off switch.** The intersection strips `none` and restores it only by measurement, and Flash had not been measured — only Pro. Measured now by the strongest signal available, the side channel itself: `none` returns an empty `reasoning_content` where `high` returns 23469 characters on `v4-flash` and 12826 on `vision-exp`. Both are back. On this link `reasoning_tokens` is absent rather than 0 under `none`, which is why the count alone could not call it and why the same shape stayed unclaimed on `qwen3.8-max`. The `none` rule now applies whether or not a lab ladder existed to intersect with: a host-only enum is not a licence to advertise an off switch nobody measured. Thirty files lose an unmeasured `none`; the eleven measured or lab-listed keep it. Fixing the lookup exposed 14 ids whose lab has toggle or budget and no effort ladder at all, where the host enum had been passing through unexamined. Probed each with an invalid value, `low` and `high`: - `glm-4.7-flash` validates and does not order — `low` 2149 to `high` 1533 — so it joins `kimi-k2-thinking` as validated-but-inert and publishes `[]`, which is also what its lab says. - `gemini-2.5-flash-lite` orders 863 to 2399, and `qwen3.6-27b`, `qwen3.7-max` and `qwen3.6-35b-a3b` map `low`/`medium` onto the lab's own thinking budget — the gateway answers "must be greater than thinking_budget [8192]" and "[32768]" when max_tokens is below them, which reads like a refusal and is its opposite. These keep the host enum as `HOST_EFFORT_LIVE`. - Six more validate and reason under `low` but timed out at `high`, so no ordering could be taken. They keep the host enum and are named as pending; `[]` would say a working field does not exist. - `sakana/sakana-namazu` answers 404 to everything including the no-effort control — "0 endpoints out of 1 requested are available" — and moves to `NOT_CALLABLE`, file deleted. The intersection is extracted as a pure `resolveLadder(id, base, host)` and covered by nine tests: the lab-directory mapping, the dated-snapshot borrow, toggle-only labs yielding no ladder, the intersection dropping accepted rungs, `none` surviving only by measurement or lab listing, an empty intersection keeping the host enum, and a measurement narrowing without widening. Breaking the directory mapping fails the test that pins it. The four failures elsewhere in the suite fail identically on the untouched base. Sync: 34 files. `validate` exits 0.
…e the toggle-only labs The review read the resolver and found `HOST_EFFORT_LIVE` declared and never used. It was right: the toggle-only branch fell through to the host enum for all fourteen ids, not the four the list named, and the test for that branch checked `labEffortValues` rather than `resolveLadder`, so it could not notice. The resolver now distinguishes what the lab actually said, because the doctrine — do not contradict the lab — reads differently in each case: - **The lab publishes an effort ladder.** Intersect. An empty intersection is a contradiction, so the model is unresolved rather than shipped with the host's rungs under the lab's name. This replaces the earlier "keep the host enum" fallback, and the test that pinned that fallback is replaced too. - **The lab publishes toggle or budget and no ladder.** A host effort enum contradicts it, so it ships only where a probe showed the host's field is a live, host-native control; otherwise the model is unresolved. `HOST_EFFORT_LIVE` is that gate and is now on the path. Breaking it fails the new test. - **The lab has no entry for the model.** Nothing to contradict, so the host's enum stands. "Invented relative to the lab" needs a lab; 24 ids are here and are left alone on purpose. Unresolved means skipped for a reasoner in this sync, which would have removed eight models including two Anthropic ones. Rather than drop them blind, each was measured on two repeats of `low`/`high` with the field validated, on a prompt light enough for `high` to finish. Seven show no ordering at all — `haiku-4.5` 188/182 vs 179/185, `qwen3.6-max-preview` with `high` below `low` both times, `gemma-4-31b-it` and `mimo-v2.5-pro` reporting a reasoning count of 1 and 0 at every rung, `mimo-v2.5`, `glm-4.5v`, `glm-4.6v` overlapping. They join `EFFORT_VALIDATED_BUT_INERT` and publish `[]`, which is also what their labs say. `claude-sonnet-4.5` alone has a direction, 184/183 vs 220/210, but a thirty-token delta on a model the host's own middleware lists as effort-unsupported is not enough to contradict a toggle-only lab; it takes `[]` with the rest and the numbers stay so it can be revisited. Nothing with a toggle-only lab is left on the host's enum without a probe behind it. The catalogue keeps all 109 models. Eleven tests, including the toggle-only fallthrough the review asked for and the empty-intersection case. Sync: 8 updated. `validate` exits 0.
# Conflicts: # packages/core/src/sync/index.ts
Upstream now requires a price and a whole limit on every synced card and types effort rungs as the schema's own union. Filter host rungs through that union, build the limit only from the keys the host actually publishes, and skip a row the host lists without a price rather than emit a card the schema cannot carry. Two cards move with their base models: muse-spark-1.1 stops restating a context its lab entry now states, and hy3 omits the input cap its lab entry gained.
… it answers as deepseek-chat answers "model": "deepseek-flash" exactly as deepseek-v4-flash does, yet its card diverged: the probes that established none as a real off switch and reasoning_content as the side channel were keyed to the id they ran against, so the alias fell through to the unmeasured default (low, high, no interleaved field). Route the measurement tables through the alias so the two cards resolve identically; the host row itself stays keyed by the alias's own id. gemini-2.5-flash-lite: the host still labels the GA route "Gemini 2.5 Flash Lite Preview". Suppress the name override until the host-side rename (aimlapi/aimlapi#6178) ships, so the lab's GA name inherits. Test: the alias ladder equals the flash ladder.
…ilent, and say on each card what the wire does The no-entry branch of resolveLadder kept the host's effort enum whenever no first-party ladder existed, which shipped an identical low/medium/high on 24 cards — a schema dump, not a baseline. labLadder now falls back to the OpenRouter card for the same lab id (OpenRouter files Alibaba under qwen/, so every directory it uses for the lab is tried). A peer ladder is intersected exactly like a lab ladder and never widened; a peer that is toggle-only counts as no effort ladder; a peer-listed none is not taken on trust the way a lab-listed one is. With neither a lab nor a peer entry the model is unresolved and skipped. Six cards leave on that rule (laguna-xs-2.1, laguna-s-2.1, qwen3.5-flash, ling-3.0-flash-fin, trinity-large-thinking, claude-opus-4.1); four narrow to the peer set (hy3, hy4-preview to low/high; qwen3.8-27b, qwen3.8-2.4t-a95b to low/medium). Cards whose control story is not plain lab ∩ host now carry a leading wire comment — the alias, the empty lists (not read vs validated-but-inert, and that toggle/budget keys are dropped), the host-live effort ids (what the names select), and none as the off switch — and headers are authoritative so a re-sync keeps them current.
HOST_EFFORT_LIVE let four ids with a toggle/budget-only lab keep the host's effort enum because the field measured live on this wire (Flash-Lite orders reasoning tokens; the Qwen names select the lab's thinking budget). The review held, across every round, that a lab or peer without an effort ladder gives nothing to intersect with and that publishing the host's list under those ids claims a control family the baseline does not use. This follows that: the no-effort branch of resolveLadder is unresolved unconditionally, and gemini-2.5-flash-lite, qwen3.6-27b, qwen3.7-max and qwen3.6-35b-a3b leave the catalogue until the lab publishes an effort ladder or this host exposes the budget as such. The measurements are kept in the source as HOST_EFFORT_MEASURED_BUT_UNPUBLISHED so the skip is not mistaken for a finding that the field did nothing.
Meta's Llama cards live under providers/llama/ and its Muse cards under providers/meta/, so mapping the metadata lab to one directory missed every Muse lab entry and fell through to the OpenRouter peer. Try the mapped directory and the lab's own name. The Muse cards themselves do not change — lab and peer agree with the host on low/medium/high — but they are now resolved from the lab, and the test no longer locks in meta -> llama alone.
# Conflicts: # packages/core/src/sync/index.ts
Contributor
|
No actionable findings. |
github-actions
Bot
added
the
reviewer: ready
Collaborator
|
Closing in favor of #9531, which includes this PR rebased on current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds AI/ML API — an OpenAI-compatible multi-model gateway — with a catalog sync module.
providers/aimlapi/:provider.toml(npm = "@ai-sdk/openai-compatible",api = "https://api.aimlapi.com/v1",env = ["AIMLAPI_API_KEY"]),logo.svg(currentColor, square viewBox), and 108 chat modelspackages/core/src/sync/providers/aimlapi.ts+bun aimlapi:sync: syncs from the public catalog (GET /v1/models?include=pricing,modalities, no key required). Onlyopenai/chat-completionsentries with text-only output are importedbase_model; provider files are override-only — cost, host limits where they differ, andreasoning_optionsread per model from the API's published request schemaprovided/generated/cached→input/output/cache_read)models/, and reasoning models where the API exposes no reasoning control, are skipped rather than guessedDocs: https://docs.aimlapi.com · Pricing: https://aimlapi.com/pricing
Reasoning options — probe evidence
Every
reasoning_optionsset in this provider is backed by live probes againstPOST /v1/chat/completions(2026-09-10/11), with each model's request schema read fromhttps://api.aimlapi.com/docs-json?model=<id>&endpoint=openai%2Fchat-completions— e.g. deepseek-v4-pro, qwen3.7-max, gemini-2.5-flash-lite. Two controls ran per model: a request with no effort field, andreasoning_effort: "banana"— a 400 (Invalid enum value) proves the field is read; a 200 proves it is swallowed.toggleorbudget_tokensis authored anywhere:enable_thinking,thinking: {type: "disabled"},thinking_budgetand an inventedbanana_toggleall return 200 on this surface — unknown top-level fields are dropped, not forwarded — and none of these fields appear in the published request schemas.none, high(0 reasoning tokens undernone, 5,662 underhigh); V4 Flash shipsnone, low, high(emptyreasoning_contentundernone, 8,981 reasoning tokens / 23,469 chars underhigh). Qwen 3.6/3.7 shiplow, medium, high: on this surface the effort names select the lab'sthinking_budgettiers (8,192 / 32,768), confirmed by the gateway's own 400s ("max_completion_tokens [300] must be greater than thinking_budget [8192]") whenmax_tokenssits below the tier.noneis published only where it measurably switches reasoning off; unmeasurednoneis dropped (gemini-2.5-flash-liteshipsminimal...maxwithout it).reasoning_options = [](model reasons, no caller control) is published for exactly 15 models: 10 wherebananais rejected but repeatedlow/highruns do not order reasoning-token counts, and 5 where the field is not read at all (banana→ 200).claude-opus-4.7-fastandclaude-opus-4.8-fast(absent from/v1/models,/docs-json404), andsakana-namazu(listed, but every request 404s — "0 endpoints out of 1 requested are available").deepseek-chatanswers asdeepseek-v4-flash(the response'smodelfield, three calls), so it isbase_model = "deepseek/deepseek-v4-flash"; the catalog's stale "DeepSeek V3" / 128K row is suppressed until the upstream correction ships.