Skip to content

fix: Add bounded distribution Excel example - #113

Closed
emmy-devop wants to merge 4 commits into
mainfrom
em/dist-excel
Closed

emmy-devop wants to merge 4 commits into
mainfrom
em/dist-excel

Conversation

@emmy-devop

@emmy-devop emmy-devop commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Requested by @drafty

🔴 Review tier: HIGH — human review required

  • Security: credential-consuming example; strict domain isolation, bounded shared retries/pacing, sanitized errors. No access changes.
  • Contract: standalone example now requires explicit --start/--stop or optional --month; workbook adds metadata and selected bounds. No SDK/API runtime contract changes.
  • Deps: existing example-only pinned openpyxl dependency; no SDK dependency/lockfile changes.
  • Release: this is a release-bearing SDK repository. Main runs version-bump automation, release tags publish PyPI; the fix: follow-up may select a patch release despite being examples-only. Human merge/release ownership remains unchanged.
  • Size: comprehensive exporter and synthetic tests exceed 400 handwritten lines.
  • Unverified: live customer extraction/authorization were expressly excluded; only offline behavior is verified.

What / why

Maintained standalone exact-domain Excel report under examples/distribution_report, revised on the same PR/branch at @drafty's request. No package runtime changes, version bumps, workflow edits, permission changes, deployments or customer data.

  • --start and --stop: explicit timezone-aware ISO8601, inclusive start/exclusive stop, normalized to UTC; no silent month expansion. --month YYYY-MM remains an optional mutually exclusive shorthand. Reject invalid/naive/nonzero-fractional inputs and cross-month UTC ranges before credentials or requests. Full month with exclusive next-month midnight works.
  • All three paginated event rankings, weighted share, optional internal aggregate/daily metrics and clipped UTC-day index queries use the selected range. The enrichment cohort is the UUID union of validated selected-range event rows, not a publication-date cohort.
  • Add actual sdk.news.get_articles(article_ids=batch, full_text=False) in batches of <=100. UUID joins, exact URL-host AND metadata-domain checks, duplicate/unrequested ID guards, metadata fields including title/pub_date/language/classification/sentiment/entities/keywords/reporting_voice/key_points. Missing retained-index IDs/metadata stay blank without losing event counts. No full-text fetch or per-article fanout.
  • Keep optional internal sections. Distinct routes take year/month only: neither is called for partial months; monthly/daily distinct cells stay blank with explicit unsupported coverage. Never relabel monthly data as weekly or sum daily unique as range unique.

Publisher guide and scoped SDK exceptions

Re-read the rendered https://docs.asknews.app/en/publisher on 2026-10-10, including all four Python recipes. Recipes 1–3 expressly use authenticated sdk.client.request; released 0.14.6 has no named ranking/share/unique helpers. Recipe 4 has the real named article method and is now implemented, correcting its earlier omission.

Internal metric helpers exist, but their DTOs drop grounded fields; raw response is retained for that narrow fidelity reason. Named index helper drops timezone offsets, so index transport preserves explicit UTC. These exceptions are documented, not a claim all SDK helpers are missing. No invented local wrapper is presented as an SDK method, and no duplicate named/raw requests merely demonstrate helper use.

Safety / interpretation

  • Six seconds before every distribution send; one second for news/auth. Reader-wide Retry-After survives final retries and section changes, including named-method sends. Invalid/over-limit waits or exhausted budgets permanently stop all future sends.
  • Defaults: 400 total HTTP sends including OAuth/retries/enrichment; 2400-second request-start budget; bounded pages and 30-second per-operation timeout. Any 401 stops before SDK refresh; 403 never retries. Denied internal group stops without endpoint probing.
  • Integer inclusive server end is stop - 1s: fractional events between that instant and stop are omitted. Index repeats that gap at every clipped UTC-day stop. One-second index clips cannot form a server bucket and are skipped/flagged. No exact coverage claim.
  • No shared snapshot; page/reconciliation checks do not prove historical completeness. Retained index is not historical inventory. Weighted share is not payout.
  • Missing versus zero remains explicit. --allow-partial is needed for actual section failures/missing IDs (exit 2); expected unsupported partial-month distincts alone are a documented caveat, not a fabricated total. Atomic no-overwrite output; literal cells/no formulas, hyperlinks or external links.

Request counts

Successful no-retry data sends:

P_surface + P_citation + P_grounded + 1 + I*(2 + 2F) + ceil(U/100) + D

P includes terminal empty ranking pages; I=1 only with internal flag; F=1 only for a full UTC month; U is unique validated IDs across rankings; D is clipped UTC-day index windows lasting >=2 seconds. Actual HTTP sends add OAuth token sends and extra retry attempts; failures/caps can stop early. With one page per metric and 1–100 IDs: week 12 public / 14 internal; September 35 public / 39 internal. All sends consume the shared budget.

README includes explicit Sep 1–Sep 8 week and Sep 1–Oct 1 month commands, credentials guidance and precise caveats.

How tested

  • Source checkout: 246 passed — 114 exporter tests plus 132 SDK tests, sockets forbidden for exporter tests.
  • Independent environment, verified import of published asknews 0.14.6 from site-packages: 118 passed — 114 exporter tests + four retained independent reviewer regressions. The four-test compatibility copy only adds the new synthetic news response and explicit month argument; assertions unchanged.
  • Actual named SDK helper invocation, batching 100/100/1, UUID joins/reordered replies, exact-domain failures, partial/missing enrichment, no full text, week/month/offset/invalid/cross-month/partial-day dates, monthly-only route omissions, auth/retries/global caps, workbook reopening/injection/no false totals/no overwrite.
  • SDK lint, explicit exporter lint/format, wheel build, whitespace and conventional commit gitlint pass. Python 3.11 tested locally.
  • GitHub CI: all 10 push/PR matrix jobs pass at b6a1d6680916cd17d8f28dfdbf685b7b921b40f1 (SDK Python 3.8–3.12). Push run needed one failed-job rerun after unchanged tests/test_security.py::test_load_save_token_disk raised InvalidToken; no unrelated runtime/test fix was made. Unchanged CI does not run optional Excel examples; those were separately tested above.

Risk / rollback

Examples/documentation only; no runtime package, dependency lockfile, version or workflow changes. Source-inspected service contracts are not evidence of deployed customer authorization/schema. Revert the follow-up to undo this correction; do not merge until independent human review. No merge, release, live extraction or production/permission change performed.

@emmy-devop

Copy link
Copy Markdown
Collaborator Author

Addressed both blocking findings in e2e3adf: Retry-After is now reader-wide and fatal wait/header/budget conditions stop every later send; UNIQUE_DAILY requires both row fields and response totals, with blanks + explicit partial exits and no daily-to-monthly distinct summation. Six-second distribution pacing remains in place from the guide follow-up.

204 source tests, 72 isolated published-SDK example tests, and all four exact independent-review regressions pass; lint/format/build pass. README and workbook methodology updated. No live extraction, permission changes or merge. Please re-review the new head; HIGH tier remains.

@emmy-devop emmy-devop changed the title docs: Add bounded distribution Excel example fix: Add bounded distribution Excel example Oct 10, 2026
@emmy-devop

Copy link
Copy Markdown
Collaborator Author

Implemented @drafty’s requested correction on the same branch/PR in b6a1d66.

  • Actual sdk.news.get_articles(..., full_text=False) batches of 100, UUID joins and exact-domain checks; useful article metadata now in Excel. Missing retained-index articles remain blank, with event counts preserved.
  • Explicit UTC-normalized inclusive --start / exclusive --stop, or optional mutually exclusive --month. Week/partial-day bounds apply across supported sections; monthly-only distinct routes are not queried for partial months. Cross-month/nonzero-fractional inputs fail before requests. Inclusive-second precision gaps are explicit.
  • Kept only documented transport exceptions (guide Recipes 1–3, internal grounded DTO loss, index timezone serialization); no package/runtime expansion or duplicate method-demonstration calls. Shared pacing, Retry-After, auth and total-send/time budgets cover enrichment too.
  • 246 source tests, 118 published-SDK checks (114 exporter + four prior reviewer assertions with compatibility-only fixture updates), lint/format/wheel/gitlint pass. All 10 CI matrix checks green; one push rerun followed an unchanged SDK token-disk InvalidToken test failure.
  • Successful no-retry data sends: Σ ranking pages + 1 + I*(2+2F) + ceil(unique IDs/100) + clipped index windows>=2s; add OAuth and extra retries, subject to global caps. README supplies week/month commands and definitions.

No live customer calls, permissions/production changes, merge or release. Ready for independent review, not independently approved.

@robcaulk robcaulk closed this Oct 10, 2026
@robcaulk
robcaulk deleted the em/dist-excel branch October 10, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants