Skip to content

Collect server-side identity provider EIDs in a single page-load write - #1487

Draft
dhruv8sh wants to merge 15 commits into
mainfrom
feat/s2s-eid-collection-core
Draft

dhruv8sh wants to merge 15 commits into
mainfrom
feat/s2s-eid-collection-core

Conversation

@dhruv8sh

@dhruv8sh dhruv8sh commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds a server-side identity provider framework so Trusted Server can collect provider EIDs (extended IDs) on publisher navigations instead of depending only on browser-side ID modules, which ad blockers and browser privacy features increasingly strip.
  • Stores provider EIDs at full fidelity (source, all UIDs, atype, ext) and serves them unchanged into OpenRTB and Prebid, so bidders see the same shape the provider returned.
  • Collapses every browser-driven Edge Cookie KV write into a single page-load write, so provider lookups, partner syncs and consent changes share one write per navigation and stay inside the KV write budget.

This is the core of the work. Follow-up PRs stack on this branch: mirror/pull sync, then the per-vendor provider implementations.

Changes

The original three units:

Area Change
ec/eid_types.rs, ec/kv_types.rs, ec/kv.rs, ec/eids.rs, openrtb.rs, ec/prebid_eids.rs Full-fidelity provider EID storage and serving
ec/write_plan.rs, ec/page_load.rs, ec/finalize.rs, publisher.rs One page-load write per browser navigation, built from a write plan
ec/provider.rs, ec/coordinator/*, ec/registry.rs, settings.rs, config.rs, new trusted-server-identity-providers crate, adapter app.rs / main.rs Provider config, validation and a coordinator that runs provider lookups concurrently on publisher navigations within the request deadline
docs/guide/configuration.md, trusted-server.example.toml ec.identity_providers configuration docs and example

Follow-up fixes on top:

  • Fix provider ownership, orphan attribution and size cap issues.
  • Skip partner token length checks in deploy-time provider validation (tokens are still secret key names at deploy time).
  • Stop warning about unregistered provider kinds where kinds are unknown.
  • Write orphan-recovery lookups to the replacement EC row.
  • Keep responses whose headers arrive, and surface arrived bodies once, in an expired deadline snapshot (platform/deadline.rs, auction/endpoints.rs).
  • Fall back to browser IDs for provider-owned sources the provider did not return.

Closes

Part of #1246 and #1245; the provider implementations that close them come in follow-up PRs.

Spec

Spec: reviewed separately; not included in this PR.

Test plan

  • cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spin
  • All eight clippy aliases (clippy-fastly, -axum, -cloudflare, -cloudflare-wasm, -spin-native, -spin-wasm, -cli, -codegen)
  • cargo fmt --all -- --check
  • Parity tests, native template build-digest test and clippy, scripts/lint-wasm-clock.sh
  • JS tests: cd crates/trusted-server-js/lib && npx vitest run (plus npm run format and npm run lint)
  • Docs format and Markdown prettier check
  • ./scripts/test-cli.sh, domain lint against origin/main, CI Node pin guard
  • Manual testing via fastly compute serve

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses log macros (not println!)
  • New code has tests
  • No secrets or credentials committed

Add EID value types, an additive KV schema for provider-written EIDs with owner and expiry, EID-level fields on the OpenRTB Eid, and a resolver that serves only live UIDs to auctions, x-ts-eids, identify and the admin auction preview. Auction EIDs merge per (source, inserter, matcher, mm, ext) with first-occurrence UID dedupe. KV writes over 16 KiB are refused.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Top-level document navigations on the publisher route receive a single-use write permit; finalize plans one row change (browser ingest, provider results, prune, size cap) and commits it once without retry. Non-navigation requests and redirects never write, integration proxies no longer mint ECs, and new ECs are created by the page-load write instead of at generation. Withdrawal tombstones only the active EC and keeps the EC cookie when the tombstone does not land; the withdrawal completion marker is the one deliberate extra write.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add the IdentityProvider/ProviderKind interface and [[ec.identity_providers]] config, response validation, a deadline-bounded LookupTransport over select_until, the IdentityCoordinator with consent, allowlist and budget checks, the trusted-server-identity-providers crate (no kinds yet), and Fastly wiring that dispatches before the origin fetch and collects before the single page-load write.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Orphan recovery minted a new EC ID and then discarded the navigation's provider results because they were looked up under the orphaned ID. Lookups depend on the browser, not the EC ID, and the orphan has no row, so retarget them to the replacement as the spec requires.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Collection that starts after its deadline takes one nonblocking snapshot. A request still in the headers state moved to the body state without reading bytes, which counted as no progress, so a response that had fully arrived was abandoned unread and reported as a timeout. Count entering the body state as progress.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A response body the host already holds can report not ready until the host services async I/O once, so an expired snapshot abandoned identity responses that arrived well before the deadline and reported them as timeouts. Give each pending body one host select capped at 1 ms before abandoning it.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…not fill

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh dhruv8sh self-assigned this Nov 2, 2026
Page-load writes only run on real navigations, which browsers mark with both sec-fetch-dest: document and sec-fetch-mode: navigate. Without the mode header the consent withdrawal scenario never got a write permit, so the EC cookie was kept.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A lookup sends the EC ID to the provider before finalize can decline the EC, so GPC, sale opt-out and unknown-jurisdiction requests must not dispatch. Matches the gate pull sync already applies.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
… read

LiveRamp envelopes are about 700 bytes, so the 512-byte limit dropped every one from KV and the auction. Rows written under a higher limit now load with the oversized IDs removed instead of failing as a whole.

Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
@dhruv8sh
dhruv8sh force-pushed the feat/s2s-eid-collection-core branch from 304f395 to 2de6d0d Compare November 3, 2026 14:29
@jwrosewell

Copy link
Copy Markdown
Collaborator

@dhruv8sh, directionally the layout here is not compatible with V2 (the 55 pull requests tracked in #777, merged in order).

This pull request creates one shared trusted-server-identity-providers crate, and #1488 adds eight vendors to it as files. In V2 each identity vendor is a module in a folder of its own at identity/<vendor>, where Lockr already is, with its own named maintainer. A module hands core the identifiers it learned for a request, and the Edge Cookie itself is a module.

The V2 spec is identifier-modules and pluggable-providers, with the folder layout in integrations-crate, all in #1342. The associated pull requests are #1382 (a module hands core its identifiers, starting with the Prebid EID cookie) and #1343 (the Edge Cookie as a module, with the HMAC module built in).

Could you evaluate the V2 spec and those pull requests before progressing this one?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants