Skip to content

withRetry retries all errors unconditionally — permanent failures waste retry budget and delay the error signal #111

Open

Description

@ooloth

Why

withRetry is used across every external API call in the build (Notion, iTunes, TMDB, Cloudinary); retrying a 401 Unauthorized or 404 Not Found burns two extra retry cycles and delays the build error by up to 12 seconds per call site.

Current state

The catch block in io/utils/retry.ts (lines 59–74) retries every thrown error unconditionally: if the attempt count is below maxAttempts, it sleeps and retries regardless of the error type. A 401 Unauthorized response from Notion (expired token), a 403 Forbidden from TMDB, a 404 Not Found from Cloudinary, and a transient 503 Service Unavailable are all treated identically — they each exhaust the full retry budget before surfacing. The onRetry log messages in call sites such as io/notion/getBlockChildren.ts and io/notion/getPosts.ts already say "timeout", implying the intent was to retry only transient failures — but the implementation retries everything.

Ideal state

  • withRetry accepts an optional isRetryable predicate that, when provided, is called with the caught error before deciding to retry.
  • When isRetryable returns false, the error is re-thrown immediately on the first failure without sleeping or consuming retry attempts.
  • Call sites that wrap HTTP responses (Notion, TMDB, iTunes, Cloudinary) pass an isRetryable predicate that returns true only for network-level errors and 5xx/429 responses.
  • A 401 or 403 from any external API surfaces as a build error on the first attempt rather than after the full retry sequence.

Out of scope

  • Adding timeouts to the underlying fetch calls (separate concern).
  • Changing the backoff or jitter behaviour.

Starting points

  • io/utils/retry.ts — withRetry implementation; the catch block where the retry decision is made (lines 59–74)
  • io/notion/getBlockChildren.ts — representative call site with a misleading "timeout" onRetry label (lines 99–111)
  • io/cloudinary/fetchCloudinaryImageMetadata.ts — call site most likely to hit a permanent 404 on a missing asset (lines 140–157)

QA plan

  1. In io/notion/getBlockChildren.ts, add isRetryable: (e) => false to the withRetry options and trigger a Notion API error (e.g. use an invalid token) — expect the error to surface on the first attempt with no retry delay.
  2. Remove the test predicate and add isRetryable: (e) => e instanceof NetworkError (or equivalent 5xx check) — trigger a genuine 503 response — expect retries to occur with backoff as before.
  3. In io/utils/retry.ts, pass maxAttempts: 3 and an isRetryable that returns false for Error('permanent') — call withRetry with a function that always throws Error('permanent') — expect the error to be thrown after 1 attempt, not 3.

Done when

withRetry accepts an optional isRetryable predicate, and call sites wrapping HTTP responses pass one so that only transient failures (5xx, 429, network errors) consume retry budget.

Activity

  1. ooloth commented on May 29, 2026

    @ooloth
    OwnerAuthor

    Related gap in io/utils/retry.ts (Should tier, separate from the isRetryable concern tracked here):

    calculateDelay at lines 33–36 computes the delay deterministically — initialDelayMs * backoffMultiplier^(attempt-1) — with no jitter. Multiple concurrent build workers retrying a shared downstream service (e.g., Notion, TMDB) after a simultaneous failure will all retry at identical intervals, creating a thundering herd that can overwhelm a recovering service. Adding ±20% random jitter to the computed delay would spread concurrent retries across a time window.


    Generated by Claude Code

  2. ooloth commented on Jun 5, 2026

    @ooloth
    OwnerAuthor

    Additional evidence: io/utils/retry.ts lines 33–36 — calculateDelay computes a deterministic exponential delay with no jitter:

    const delay = options.initialDelayMs * Math.pow(options.backoffMultiplier, attempt - 1);
    return Math.min(delay, options.maxDelayMs);

    When concurrent callers all fail at the same moment (e.g. a TMDB or Notion outage during a build), all retry at exactly the same calculated interval, producing a thundering herd against the recovering service. Adding a random jitter factor — e.g. delay * (0.5 + Math.random() * 0.5) — would spread retries across the backoff window. This is a distinct gap from the transient-vs-permanent classification issue tracked here, but lives in the same function.


    Generated by Claude Code

  3. ooloth commented on Oct 18, 2026

    @ooloth
    OwnerAuthor

    As of 2d73fa4, the Cloudinary call site rethrows a plain Error whose message carries the HTTP status as text (HTTP 404: …, via describeCloudinaryFailure). An isRetryable predicate would have to parse that text, or the rethrow would need to keep http_code as a field.

  4. ooloth commented on Nov 7, 2026

    @ooloth
    OwnerAuthor

    Since #219, io/notion/client.ts passes retry: false to the Notion SDK, whose 5.10+ releases retry 429 and 529 on their own by default. withRetry, with the shared pause from #183, is the only retry path for Notion calls. An isRetryable predicate for Notion belongs there, and turning the SDK's retries back on would bypass the pause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions