Repository navigation
release: website 5dee9d0 to production - #88
Merged
Merged
Conversation
* feat(proxy): forward cards and shares and name the caller to dfo-apis The public /api/v1 proxy now forwards /presentation/components and /presentation/shares, passing the write-token, ETag and CORS headers the share data route needs. Analysis, anchor, pin and derived reads already sit under the datasets surface. Every forward to dfo-apis carries x-dfo-caller and x-dfo-caller-principal (DFO_CALLER_PRINCIPAL), so dfo-apis can count allowances per client. The caller digest now hashes the folded address (IPv6 to its /64, v4-mapped to IPv4), the same preimage dfo-apis uses for direct callers. * feat(share): export shares for dfo-apis and keep users' write tokens scripts/export-shares.ts --since writes every share as one JSONL line in the format docs/share-export.md sets out: id, kind, mode with today's null-mode defaults applied, owner, created_at, title, a slot per referenced data card, and the website-owned payload that renders what slots cannot (search queries and parts, conversations, inline cards, canvas layout). It reads only, orders by created_at then id, and exits non-zero naming each share it cannot place. The share_token table (migration 0022) keeps a signed-in user's write token for a share held in dfo-apis. * feat(share): keep shares in dfo-apis and render them from its records Creating, fixing and deleting a share goes through dfo-apis' share routes with the caller named on every write. A signed-in owner's write token is kept in share_token and sent on later writes; guests may make live shares, which they cannot change afterwards. /s/<id> and /embed/<id> render the dfo-apis record: card shares from their slots, search, canvas and conversation shares from the stored payload. sharePayload places cards in slots for new shares and the exporter alike, and the exporter writes frame: null for a card whose frame the site does not hold. scripts/import-share-tokens.ts loads the token file dfo-apis' importer writes, idempotently. The per-guest share allowance moves to dfo-apis, so spendGuestShare is gone. * feat(share): draw live cards from dfo-apis' slot card route A live card on /s/<id> and /embed/<id> reads its slot's card from dfo-apis, drawn on the frame the slot serves, so the site needs no render principal; the refresh route is a GET that passes the ETag both ways, so the browser revalidates instead of redrawing. A live share's cards are checked before the share is created, so a failing card never leaves a share behind. Share create sends a minted write token and retries once with the same body, and a payload over dfo-apis' limit answers 413 payload_too_large. * feat(share): download a live card's data from its slot A live card's downloads, bundles and spreadsheets included, come from dfo-apis' slot data route in the format and bundle the reader asked for, so a download holds the data the card shows. The browser's refresh reads answer once, with no pending loop, and live shares need no credential. * fix(share): show a snapshot's recorded data and keep owners' tokens A snapshot slot draws and downloads the frame its slot recorded at the switch, through the same slot routes a live slot uses, so the page, its downloads and dfo.fetch(share) agree; a card whose read fails is fetched again by the browser and shows the retryable error state. A stored write token is dropped only when dfo-apis says the share is gone or the token is wrong. Every share call runs under a 35 s deadline, a create whose token cannot be stored deletes the new share, and switching a search or conversation share to a snapshot is refused as before. The proxy refuses path segments that decode to "." or ".." or hold a slash. * fix(proxy): refuse dot segments on every forwarding route The graph and search catch-alls and the component export route refuse a path segment that decodes to "." or ".." or holds a slash, as the datasets and presentation routes do. A test finds every forwarding catch-all under /api/v1 and checks each carries the guard. * test(share): round-trip legacy shares through export and import scripts/share-migration-round-trip.ts seeds one share per legacy case, exports them, imports them into a local dfo-apis and the tokens back, renders /s and /embed for each, checks an owner can write with the imported token and a guest's share has none, and checks a second run of both imports changes nothing. * fix(share): leave a held token's creation time alone on a re-put Putting the token a share already holds for the same owner changes nothing, so a second run of import-share-tokens leaves share_token as it was. * docs: tighten comments and share docs Plain sentences in the share, proxy and studio comments and in docs/share-export.md and docs/development.md, and one typo. * fix(proxy): encode every forwarded segment and pass slot freshness Every forwarding route builds its upstream path through forwardablePath, which refuses dot segments and encodes each segment, so a double-encoded ".." or a "?" or "#" segment stays inside the surface it was sent to; the component read route takes the same check, and a test finds every route under /api/v1 and /api/presentation that forwards. Slot data read through the site keeps dfo-apis' x-dfo-freshness and x-dfo-checked-at headers. * fix(share): take a deleted account's shares down and keep refusals honest Deleting an account takes each of the user's shares down in dfo-apis with its stored token before the user row goes; a share already gone counts as done, and any other failure keeps the account and the remaining tokens. A create refusal dfo-apis marks permanent answers share_refused with its code and no retry; a transient one keeps the retry. A create whose token cannot be stored, because another user's row holds the id, takes the new share down. Titles are cut by code point, GET /api/share/<id> answers a problem document when dfo-apis fails, and the share export reads shares and parts from one snapshot. * fix(share): draw SDK shares, bound a page's wait, and read slots directly A share made through the Presentation API with no website payload draws one card per filled slot. A share page waits at most 4 s for its slot cards and leaves the rest for the browser to fill. Card refresh and download URLs carry the slot, so their routes read that slot and never the whole share record; a card no slot draws, as on a conversation share, downloads through the component export route. A component read refused with anything but 404 is transient, so a busy dfo-apis no longer withdraws a live link. * fix(share): cite a snapshot card from the time its slot was served A snapshot card's citation dates from when its slot first served the frame it shows, read once from the share's history within the page budget, and falls back to the share's creation time. The share dialog shows a permanent refusal, or a share too large, without a retry. A lost write token no longer blocks account deletion, a token the site cannot keep answers a retryable 502, and the download-pin path encodes its segments. * test(share): compare round-trip tokens in byte order The round trip orders share ids with collate "C", as the JavaScript sort it is compared with does, so ids of mixed case compare equal in either order. * fix: read every page of share history before dating a slot readShareHistory reads one 500-entry page and returns its next cursor, and servedSince follows the cursor until every framed slot has its entry or the history ends, within the share page's 4 s deadline. A cursor that repeats the one just sent counts as an invalid reply. A snapshot citation takes its date from the slot's served-since entry alone. When that entry is not found, the citation prints "(snapshot)" with no date; the share's creation date could name newer data. * fix: fence account deletion against share creates and keep failed take-downs takeDownUserShares commits user.deleting_at before it lists the user's share tokens. putShareToken inserts through a select of the user row FOR SHARE that requires the fence unset (or older than 15 minutes, since Better Auth runs no hook after a failed user delete), so a share created during deletion either lands before the list or is refused and taken down by createOwnedShare's compensation. A failure that keeps the account clears the fence. A take-down that dfo-apis answers 204 or not_found is done. One refused with write_token_* is written to the new share_cleanup table, which has no foreign key to user, logged as an error, and deletion continues; an operator removes those shares with datasets-maintenance take-down-share. A failed compensation take-down is recorded there too. Migration 0023 adds the table and the column. * fix: hold each deletion attempt's own fence until its list is empty fenceShareTokens returns the attempt's fence value; takeDownUserShares refreshes that value before each take-down, re-lists tokens until none remain, and on failure lifts only a fence that still holds its value, so a failed attempt cannot open another attempt's fence. createOwnedShare reads the fence first and refuses a deleting account with 403 account_deleting before any dfo-apis call. A failed share_cleanup write logs share_cleanup_not_recorded and keeps token_not_kept. The share_cleanup_recorded log line is the alert for shares an operator takes down; deploy/README.md gives the drain steps. History reads 100 entries on the first page and stops on a repeated cursor. The token import reports a deleting owner as ownerDeleting.
…86) The share migration job reads the website's own share rows, places each as the export did, keeps each owner's shw_ write token in share_token, and calls dfo-apis' PUT /api/presentation/shares/{id}/import with only the token's SHA-256. It holds the owner's user row FOR SHARE across the call, so an account deletion either waits and takes the imported share down or lands first and the job skips the share. The export and token-import scripts are removed. apisFetch signs requests from a Cloud Run job as well as a service, so the job calls dfo-apis with an ID token.
…owser tests in CI without Chrome (#87) The root tsconfig included motion-kit, so a product change that broke motion-kit's types failed `next build` and with it the production image. motion-kit is now excluded there and checked as its own project, which `pnpm typecheck` runs after the app's. Its four Chrome-backed tests skipped when no browser started; under CI they now fail with the launch error, and locally they still skip with it.
…ards a canvas shows (#89) * fix(share): give gallery cards their share slots and embed only the cards a canvas shows The live gallery pages refreshed each card without `?slot=`, which the slot refresh route answers 404, so every card failed once shares moved to dfo-apis. `liveSlotsOf` derives each card's slot with `slotCanvas`, the function that gave the prod share its slots. `/embed/<id>` drew every card not removed, including older-turn cards the canvas hides. A live share holds those with no slot, so a live embed showed them as failed and a snapshot embed crashed on them. The embed now lists `canvasWidgetsOf`, as `/s/<id>` does. * test(share): drop slotCards rows that pass without the embed fix
jm-rivera
changed the title
…og (#90) * feat(data): tidy the unit picker, About this answer and download dialog The unit picker drops the standing euro preset, keeps the currency of any country the question names, and puts a "Search N currencies" field first (lib/search/unitPicker.ts, UnitPicker.tsx). About this answer shows each fact once. A fact the search filled in carries its reason and a Change control that opens the same menu the sentence uses (changeTarget in lib/search/refineMenu.ts). Breakdowns list their categories without a variable count, the publisher's own indicator code sits on the Measure row from stat_vars.source_codes, the source appears once, and the citation says "processed by" only when a conversion was applied (lib/search/view.ts, WhyPanel.tsx). The download dialog has one switch for the README, sources and citation files, and folds extra columns under plain names with examples from the answer and blank counts in words. Extra columns the long file already writes are hidden by the API's in_long_layout flag, with the long_layout_defaults comparison as the fallback until the API serves it (lib/search/downloadOptions.ts). * fix(data): route Change to Refine for an open recipient role Refine keyed an open role by the role name (`destination:`) while the About rows and the wire use `roleRef` (`where:`), so Change found no Refine entry for an open recipient. Refine now keys roles by `roleRef`.
…91) * feat(studio): credit sources on cards and redraw the More Info panel The canvas title runs to the card filters and shows in full on hover (WorkspaceToolbar.tsx, clipTitle removed). The card foot credits its sources as "Source: X +N more" with a small More Info mark, drops the indicator count, and credits a computed card's input publishers (sourceLine in lib/chat/widgets.ts). More Info shows the calculation first for quotient and share cards, lists sources publisher first, offers short and reference citations, and mints a public snapshot link for the citation; POST /api/share returns the snapshot's servedAt. How this card was made is a step rail (descriptionBlocks.ts stepRows). Cards drag from the head and frame after a 4px move, so content clicks and selects text, while touch and read-only canvases still pan (canvas/cardPress.ts). * fix(studio): keep one citation link request per card version The pending request lived in CardInfoPanel, so reopening More Info re-enabled the button and a second click minted a second public snapshot. Studio now holds the request per `citeLinkKey` (`citeLinkOnce`) and reuses it until it settles.
) * feat(studio): group answer cards in the chat and focus chart hovers Several cards from one answer collapse into one "N cards" row that expands to one-line rows, with Remove on hover or focus and a Remove all menu (WidgetChips.tsx); the approved plan pill takes one line. Chart hovers name only the nearest series with its legend name, fade the others and the other legend entries, and list every series while Shift is held (chartSetup.ts, lib/studio/chartHover.ts). Every palette keeps its first eight colours and adds two, so ten series stay apart (theme.ts). Table cards move constant columns and the text shared by variable names into a caption, shorten only the variable column, and sort by header with numbers read back from their display form (lib/studio/tableView.ts). * fix(studio): drop a revised table's stale names column `redraw` spread the held widget under the new cells, so a revision with no `stat_var` column kept the old `namesColumn` and shortened names such as South Sudan to Sudan.
* feat(studio): show each turn's work in the Activity tab The Activity tab showed failures and folded every step that worked behind "Show N completed steps", so a turn with no errors told the reader nothing. Each turn now opens with its question and an outcome line, then lists the plan and the steps that ran a tool on the step rail More Info uses, with each card a step made as a chip that brings it into view. A retry that worked folds into its step as "Worked on try N"; a failure that stands reads "Not done" with its full sentence. `turnView` in src/lib/studio/activityView.ts builds the view, and useChat stamps `approvedAt` on the turn when a plan is approved, so the plan row sits among the steps. * fix(studio): order Activity retries and the plan by arrival Browser receive times can tie when a failed try and its retry arrive in one chunk, so the retry and plan placement rules misread them. `recordStep` and `recordApproval` in activity.ts number each turn's events as they arrive (`startOrder`, `settleOrder`, `approvedOrder`), and `turnView` folds a retry or places the plan by that order, with the clock rule kept for older saves. A saved plan with no steps list counts none, so reopening it no longer throws.
A Change in "About this answer" opens the menu of a headline token or of the Refine button. Once the reader has scrolled down into the panel that trigger sits above the viewport, and Radix drew the menu clipped off the top of the page. revealThenOpen in src/lib/search/menuReveal.ts scrolls the trigger under the site header first and opens the menu once the scroll settles, unless another Change, a new answer, or the reader scrolling takes over.
A heading menu reported a ref as one it changes even when its only option was the current value, so "About this answer" offered Change on a lone source and opened a menu with nothing else to pick. The source menus now report their ref through sourceMenuRefs, which applies the slotOffersChoice rule Refine already used, and the shared entity menu through sharedEntityMenuRefs, which reports its ref only when it lists another option or a truncated list. The menus still open, so a lone source keeps its About block.
) * fix(studio): open only the latest Activity turn and drop "not done" The Activity tab flagged "N steps not done" in red on turns that answered, where the agent had recovered from the errors another way. A step that stands failed in a turn that answered now reads a grey "Hit an error" with its cause. A turn ends marked failed only when its stream failed before the reply had any text (endTurnActivity's `answered`, read by useChat's hasAnswer), and only that turn shows a red "No answer"; a turn the user stops is never failed. Only the latest turn opens by default, each outcome part keeps to one line, and the Activity scroll no longer stacks the panel padding on its fade padding above the first question. * fix(studio): count a card as an answer when a turn fails A turn whose stream broke after it made a card, before any reply text, read "No answer" though the canvas held its card. endTurnActivity marks a turn failed only when it has neither reply text nor a card.
…t Data (#97) * fix(studio,data): repair the release review findings in Studio and Get Data Number each chat message's turn from the end of the transcript so forked shares pair replies with their own cards and plans. Keep the "Computed by" credit in More Info, describe each dividing step from its own inputs, settle a failed cite link and log its cause. Let loading cards and text-card gaps drag. Read the members list while membership is unsettled, scroll a heading token clear of the header before its menu opens, and build the download dialog's in-file note from the same flags as its extra columns, with kinds per role. * fix(data): leave the Change menu scroll to #96 #96 scrolls a Change's trigger into view before its menu opens, for heading tokens and the Refine button alike, so this branch keeps SlotMenu, the .slot scroll margin and the header height as on dev. * fix(studio,data): classify in-file columns once and settle cite links in the helper The download dialog's extras list and in-file note read one classifier, longLayoutSuffix, so a role's kind columns are never both claimed in the file and offered as extras. messageTurns numbers no turn below 1. citeLinkOnce settles a throwing snapshot as a failed link and logs it.
* feat(studio): show publisher codes in a card's More Info
Card descriptions can carry the publisher's indicator codes behind
their series (about.source_codes, and source_codes on each drawn
value). More Info shows a one-series card's code under the measure
("World Bank code DT.TDS.DEGG.CD") and each drawn value's code beside
its Shows row. The grouping by publisher and the sentence move into
src/lib/sourceCodes.ts and the SourceCodes atom, which Get Data's
"About this answer" now uses too, so both surfaces word codes the same
way. The description reader drops a malformed code list and keeps the
rest of the description.
* fix(studio): keep card codes no value carries and count hidden chips
More Info dropped the card-level code line whenever any Shows value
carried codes, so a card code that no value carried vanished.
unlistedCodes now keeps the card codes no shown value carries, and the
line shows only when some remain. The per-value chips show three codes
a publisher and now add "+N more" for the rest (sourceCodeChips).
The client at sdk 82701c8 follows spec-kit 4ad7739, whose wire types carry EnrichmentOptionModel.in_long_layout, so DatasetEnrichment is the generated type itself. Built from typescript/ and re-locked with `just relock-client`.
* feat(studio): fit the table card to its rows and columns A table card was a fixed 22 by 14 cells, so a 40-row table showed five rows with no sign it scrolled and a 3-row table left half the card empty. The card now measures its height from its rows, showing ten and a "Show all N rows" control, and a new card takes the narrowest width that fits its columns, estimated from Outfit glyph advances in tableFit.ts. A series table turns on its side when that fits the default width, and sideways scroll with a sticky first column comes last. The value column carries the unit, decimals line up per column and a repeated name shows once per run. * fix(studio): cap a table's row labels and drop series on redraw A sticky row label had no width limit, so a long label could cover every value column even when scrolled fully right. Row labels now stop at ROW_LABEL_MAX (160px, at most 40% of the table frame) with an ellipsis and the full text in the cell's title, and tableWidthPx uses the same cap. A sketch table redrawn with payload cells kept its series flag and transposed its dimension columns; redraw now deletes it with namesColumn.
jm-rivera
changed the title
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releases website at
5dee9d0to production, cut onrelease/2026-10-31: fifteen pull requests./s/<id>and/embed/<id>render from dfo-apis' share record, and the proxy forwards/api/v1/datasets,/presentation/componentsand/presentation/shareswithx-dfo-callerandx-dfo-caller-principal, so dfo-apis counts allowances per visitor. Migrations 0022 and 0023 addshare_tokenand the account-deletion fence.share-migratejob and makesSHARE_MIGRATE_JOB_WEBSITEa required deploy variable. The deploy pointsdfo-prod-website-share-migrateat the new image./api/share/<id>/cards/<cardId>/…?slot=<n>, and limits/embed/<id>to the cards the canvas shows, as/s/<id>does.in_long_layout; until then thelong_layout_defaultscomparison decides.POST /api/share, which now returnsservedAt. Computed cards read "Source: DFO calculation" until dfo-apis serves credited source entries. Cards drag from their head and frame.source_codes.82701c8, typingin_long_layoutand the analysis, export, frame and share routes.Before merging: the infra release has applied,
scripts/handoff.sh dfo-prod websitehas setSHARE_MIGRATE_JOB_WEBSITE, and both apis releases have deployed: every share call goes to routes apis #170 and #195 add, and #100 reads the publisher codes apis #239 adds.From the deploy until the migration finishes, old
/s/<id>links answer 404. Straight after the deploy, run:Exit 0 means every share was imported or already there. Exit 1 logs one
share <id>: <code> <detail>line per refused share, and a rerun skips imported shares. Then runnormalise-recipesondfo-prod-datasets-maintenance, first with--dry-run, and read itsdatasets_recipes_normalisedline before the real run. Open a few old share links to check them.The migration refuses a share for good when dfo-apis cannot pin one of its frames:
placement_refusedfor a live card with nocmp_ref,not_foundfor a card revision dfo-apis does not hold, andframe_unavailablefor a frame past the 30-day working tier. Prod's first shares date from 2026-10-08, so their frames start to expire around 2026-11-07. Run the job straight after the deploy.Rollback: the previous image runs on the migrated database, since migrations 0022 and 0023 only add. It serves
/s/<id>from the localsharetable, which this release no longer writes, so after a rollback a share taken down since the deploy shows again and a share made since the deploy answers 404.Testing: CI passed on each pull request into
dev, and the dev deploy passed on5dee9d0. The production deploy waits for approval.