[nick-nlb]

Overview

#574 added the four unified-synthesis modules (workflows/charts.py, workflows/chart_defaults.py, workflows/synthesis.py, and workflows/blocks.py) without runtime callers. This PR connects them into the live agent loop, replacing the separate parallel chart-planning and data-validation Gemini calls (workflows/chart_config.py) and the {text} and {chart_config} SSE content frames with a single synthesis stream that emits interleaved TextBlock and ChartBlock items as {block} SSE content frames.

1. Unified synthesis and block streaming (workflows/chat_pipeline.py, workflows/sse_frames.py): run_synthesis_phase builds typed prompt inputs with build_synthesis_inputs and render_synthesis_message, composes the synthesis system instruction from config.prompts.synthesis, the packaged synthesis contract, and the pattern's synthesis directive, bounds both opening the stream and reading each chunk by SYNTHESIS_CHUNK_TIMEOUT_SECONDS, and streams the model output through BlockSplitter, gating inline chart blocks on discovery.has_data and discovery.tables before resolving them against TurnCache. Completed prose paragraphs and validated chart blocks stream as BlockContentFrame ({"block": ...}) events; when a data-bearing turn produces no valid model-authored chart block, build_default_charts appends at most one fallback chart; and charts_emitted and charts_rejected are recorded on TurnTelemetry in finally:. run_clarification emits its single question as a TextBlock BlockContentFrame, and collect_follow_up_topics derives follow-up topics from the digest results, emitted chart titles, and data-gap missing-data notes.
2. Synthesis prompt and packaged contract (workflows/contracts/synthesis.md, workflows/prompt_contracts.py, defaults/prompts/synthesis.md): Splits the synthesis prompt into configurable prose and grounding guidance (defaults/prompts/synthesis.md) and a code-owned contract (workflows/contracts/synthesis.md) appended by compose_system_instruction, defining citation formatting, answer structure, and the per-kind fenced chart block grammar so prompt overrides cannot drift from ChartSpecification.
3. Resolved-chart state slots and retired chart roles (workflows/state_slots.py, config_model.py, gemini/schemas.py, telemetry.py): Updates build_state_slots to accept Sequence[ResolvedChart] directly instead of untyped chart_config dictionaries, deletes workflows/chart_config.py, removes the CHART_CONFIG and CHART_VALIDATION roles, model settings, and structured-output schemas, and adds charts_emitted and charts_rejected to the chat_turn JSON log and OpenTelemetry span attributes.

The core files to review are workflows/chat_pipeline.py, workflows/contracts/synthesis.md, defaults/prompts/synthesis.md, workflows/state_slots.py, and workflows/sse_frames.py. This PR and the next PR in this sequence form a back-to-back pair: this PR switches the server's SSE content frame payload from {text} and {chart_config} to {block}, and the next PR updates the React UI consumer to render interleaved text and chart blocks, so the two PRs should be reviewed together and merged back to back.

Related Issues

This is the sixteenth step in the continued agent loop update sequence, following #547, #549, #550, #558, #559, #562, #563, #564, #565, #566, #568, #569, #570, #571, #572, and #574.

Changes Made

• Unified synthesis pipeline (workflows/chat_pipeline.py): Replaces the parallel chart task and plain-text streaming with a single synthesis stream that splits the response into interleaved text and validated chart blocks, appending a default chart when a data-bearing turn produces none. Clarification turns emit their question as a single text block, turn signing signs the prose without chart fences alongside the emitted charts' state slots, and follow-up generation draws topics from the digest results, chart titles, and data-gap notes.
• Packaged synthesis contract and default prompt (workflows/contracts/synthesis.md, workflows/prompt_contracts.py, defaults/prompts/synthesis.md): Moves citation formatting, answer structure, and the fenced chart block grammar into a code-owned synthesis contract appended to every synthesis prompt so instance overrides cannot drift from ChartSpecification. Replaces defaults/prompts/synthesis.md with the five-section input description and eight grounding rules for answering from the digest.
• Deleted chart-planning workflow (workflows/chart_config.py, workflows/chart_config_test.py): Deletes the standalone chart-planning and data-validation model calls and their unit tests now that charts are selected and validated inline during synthesis.
• Changes to previously merged code (workflows/sse_frames.py, workflows/state_slots.py, workflows/follow_up.py, workflows/messages.py, workflows/mcp_loop.py, config_model.py, gemini/schemas.py, telemetry.py, schemas/agent-config.example.json, schemas/agent-config.schema.json): Replaces the text and chart-config SSE frames with BlockContentFrame and updates build_state_slots to read scopes directly from ResolvedChart instances instead of untyped dictionaries. Retires the CHART_CONFIG and CHART_VALIDATION Gemini roles, model settings, and structured-output schemas, and records charts_emitted and charts_rejected in turn telemetry.
• Unit tests (workflows/chat_pipeline_test.py, workflows/sse_frames_test.py, workflows/state_slots_test.py, workflows/prompt_contracts_test.py, server/routes/chat_test.py, config_model_test.py, gemini/schemas_test.py, gemini/client_test.py, telemetry_test.py): Updates and expands unit coverage for block streaming, inline chart resolution and fallback defaults, ResolvedChart state-slot grounding, synthesis contract alignment with ChartSpecification, and chart telemetry.
• Documentation and deployment comments (narratives/AGENTS.md, narratives/README.md, docs/architecture.md, docs/deployment.md, docs/development.md, deploy/terraform-custom-datacommons/modules/main.tf): Updates the architecture, deployment, and developer guides to describe unified synthesis, {block} SSE frames, the retired chart roles, and the new chart telemetry fields.

Testing Done

• pnpm test:agent (uv run pytest): 1,706 passed (1,751 on main; deleting workflows/chart_config_test.py and the retired CHART_CONFIG and CHART_VALIDATION schema and role tests and replacing the pipeline, SSE-frame, and state-slot suites accounts for the difference).
• pnpm test:ui (vitest run): 28 files, 304 passed.
• pnpm lint:ui (tsc --noEmit and biome check): passed across 53 checked files.
• pnpm lint:agent (ruff format --check . ../deploy, ruff check . ../deploy, and mypy --strict): passed across all 124 source files (126 on main).
• pnpm build and node scripts/stage_static.mjs --config-only: passed.
• pnpm run lint:shell, pnpm run lint:docker, and terraform fmt -check -recursive deploy/terraform-custom-datacommons/: passed (terraform validate fails locally with a pre-existing backend configuration error on main).
• Validated schemas/agent-config.example.json and defaults/agent-config.json against schemas/agent-config.schema.json and schemas/theme.example.json and defaults/theme.json against schemas/theme.schema.json with Draft202012Validator, and confirmed chart_config_model and chart_validation_model are rejected by the updated schema.
• uv --directory agent run python -m narratives_agent.analytics.golden.regenerate_golden --check: exits 0; core_golden.json is unchanged.
• Confirmed that 29 planted mutations across sse_frames.py, state_slots.py, prompt_contracts.py, contracts/synthesis.md, defaults/prompts/synthesis.md, telemetry.py, and chat_pipeline.py each fail the unit suite before being reverted.

Checklist:

• Unit tests passed
• Integration tests passed (N/A: the agent has no integration suite)
• Manual verification
• Any updated goldens or fixtures were reviewed and are intentional (N/A: no goldens or fixtures changed)

Manual

From narratives/agent, ran run_turn, run_clarification, and build_state_slots via uv run python with Gemini and MCP stubbed and verified each scenario by hand:

• A turn whose streamed synthesis Markdown contains prose paragraphs around a valid fenced chart block emits three BlockContentFrame events (text, chart, text), signs only the prose paragraphs in turn.model_response without the chart fence, records grounded table and chart QueryScope entries in turn.state_slots, records charts_emitted=1 and charts_rejected=0 in TurnTelemetry, and passes the digest scopes and emitted chart title into follow-up generation.
• A data-bearing turn whose streamed synthesis Markdown contains an ungrounded chart block drops the invalid block, appends the deterministic line fallback chart from build_default_charts, and records charts_emitted=1 and charts_rejected=1 in TurnTelemetry.
• A clarification turn emits a single BlockContentFrame carrying TextBlock(index=0, markdown=question) and signs an empty state_slots list.
• build_state_slots across ResolvedLineChart (date_range=None), named-place and cohort ResolvedBarChart, ResolvedRankingChart, ResolvedMapChart, ResolvedScatterChart, and ResolvedHighlightChart emits grounded QueryScope entries for each chart kind and respects MAX_SCOPES_PER_TURN.

Risk & Rollback

There are no live instances of this application yet. This PR changes the SSE content event payload from {"text": ...} and {"chart_config": ...} to {"block": ...}, which the current UI ignores until the paired next PR updates use_sse_chat.ts and card_response.tsx; review and merge the two PRs back to back. Existing agent-config.json files that still set gemini.chart_config_model or gemini.chart_validation_model continue to load because parse_agent_config ignores unknown keys. Rollback is reverting the PR.

Follow-ups

• The next PR in this sequence updates the React UI to parse {block} SSE content events and render interleaved Markdown text and chart blocks.
• A subsequent test PR adds golden fixture cases for the distribution, composition, derived, and relation analytics scripts.

Checklist

• I have read AGENTS.md and followed CODING_GUIDELINES.md, plus FRONTEND.md for UI changes.
• I have run the app's lint, test, and build commands, as documented in that application's guide.
• I have commented my code, particularly in hard-to-understand areas.
• My changes generate no new warnings.

---

Note: Only Maintainers can approve and merge PRs. Expected initial review time: 3 business days.

[gemini-code-assist]

Code Review

This pull request refactors the narratives agent's synthesis and chart generation pipeline by replacing the parallel background chart configuration and validation workflow with inline, block-based synthesis. The agent now streams interleaved text and chart blocks, resolving chart specifications in place against the turn cache using a new BlockSplitter and BlockContentFrame. Additionally, a code-owned synthesis contract has been introduced to enforce citation and chart block grammar, telemetry has been updated to track emitted and rejected charts, and configuration models have been simplified by removing the obsolete chart configuration and validation roles. No reviewer comments were provided for evaluation, but the implementation is clean, robust, and well-covered by updated unit tests.

[juliawu]

Very nice!