Skip to content

fix(diagnostics): preserve source failures and request ownership - #818

Merged
superuser404notfound merged 2 commits into
superuser404notfound:mainfrom
yucelokan:codex/source-failure-diagnostics
Oct 25, 2026
Merged

superuser404notfound merged 2 commits into
superuser404notfound:mainfrom
yucelokan:codex/source-failure-diagnostics

Conversation

@yucelokan

@yucelokan yucelokan commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Preserve source failure causes and keep request diagnostics attached to the load that created them. Retiring a failed ranged request also clears its stored failure so a healthy forward-only replacement can open.

What changed

  • Preserve historical HLS/Demuxer NSError domains and case codes; expose finite HLS reasons and the original FFmpeg code separately.
  • Keep failed live starts classified as liveSourceUnavailable, with the original URL domain/code. HTTP refusals and TLS trust failures retain their specific classifications.
  • Capture diagnostic ownership before detached opens and pass it through AVIO probes, reconnects, prefetch, chunk and streaming delegates. Session reopens retain the same identity; explicit nil stays unattributed.
  • Make diagnostic events immutable and use extensible, string-backed Role and Phase types without changing their raw tokens. Hosts still own URL redaction.
  • Check HLS reason literals against the finite mapping and document that historical playlistUnreachable(status:) also covers segment/key HTTP refusals.
  • Make the streaming read-loss regression cut its connection only after a successful open/read, so an initial URL error cannot race the mid-read EIO assertion. The blocking work runs on a cancellable test worker.
  • Rebase on current main (7.45.0). Drop the separate refusal-test workaround in favor of main's fix(tests): LiveStreamRefusalRetryTests no longer flakes: per-reader budget and backoff #809 fix; dependency declarations and that test now match upstream.

These changes remain together because the failure cause and request owner must survive the same open/replacement lifecycle. No retry budgets, TLS policy or host fallback policy are changed.

Test plan

  • Device / OS: MacBookPro18,1; macOS 27.0.1 (26A434); Xcode 27.1 (27A9275).
  • Source media: In-process HTTP range fixture with deterministic bytes for AVIO open/seek/reconnect; synthetic URL/HTTP/TLS failures and malformed HLS parser inputs. These validate transport and diagnostic contracts, not audio/video decode.
  • Result: At 6f1608d7, 62 scoped unit tests passed (35 XCTest + 27 Swift Testing), with all four production traits enabled. The package and ExampleSources compiled; documentation links/anchors passed.
  • New regressions cover attributed and unbound reconnects across a real thread boundary, session reopen identity, live error compatibility, future diagnostic tokens and source-literal mapping drift. Existing error-code, reload-identity and public API/documentation checks are included.
  • The CI-only streaming-drop race was reproduced by the failure ordering and repaired in ad4b342d; its targeted unit test passed against the same reader/test sources in the 7.45.0 fork integration. An unchanged upstream tail-prefetch test hit its 4 s HTTP timeout under runner load on the next CI run. The identical tree at 7d643f26 passed all four checks on GitHub Actions: macOS default/all-traits tests and both single-thread runs, iOS/tvOS/visionOS builds, runtime regressions and documentation links. No timing assertion was weakened.
  • No new physical-device playback or local full-suite run for this revision. Earlier validation described different revisions; the results above apply to this update.

Checklist

  • CHANGELOG.md updated
  • Commit messages follow Conventional Commits
  • The fix lives in the engine, not in a host-side workaround
  • Public API changes are intentional and documented

@superuser404notfound superuser404notfound left a comment •

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Posted twice by accident, the review below is the same text.)

@superuser404notfound superuser404notfound left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, the cause-preservation work is a real improvement: keeping the historical NSError domains and case codes while carrying the FFmpeg code separately, terminalError staying source compatible, zero cost while no handler is installed, and the retired ranged connection no longer poisoning its forward-only replacement are all things I want in the engine. A few points before it can go in, roughly in order of weight.

1. Rebase onto main. The branch conflicts in CHANGELOG.md, AetherEngine+ReloadWithOptions.swift and LiveStreamRefusalRetryTests.swift. The flaky give-up test was already fixed on main in #809 (2394ab2e: the refusal budget is passed per reader and the give-up case is bounded from below instead of counting requests), so b9169f38 should probably be dropped in favour of main's version rather than merged on top of it. If something in it is still needed after the rebase, a short note on what main's fix does not cover would help.

2. A failed live start should stay .liveSourceUnavailable. Before this PR, a live probe that burned its reconnect budget (AetherEngine.swift live fail-fast) and a transport failure in the live HLS ingest both published .liveSourceUnavailable. Now both go through publishLoadFailure, so a timeout or a refused connection arrives as .sourceOpenFailed. Hosts branch on that kind: Sodalite maps .liveSourceUnavailable to its localized "live channel unavailable" screen and would fall back to a generic engine-classified error for every dead channel. api.md treats the kinds as API, and a host cannot tell this move apart from a new failure class.

I'd suggest keeping the new detail without moving the kind:

  • HTTP status: .sourceRefused / .sourceRateLimited with the status (this matches what the live HLS path already did, fine as is).
  • TLS trust: .sourceCertificateRejected (also fine).
  • Everything else on a live start: .liveSourceUnavailable, with the original URL error in underlyingDomain / underlyingCode.

The test "Live initial connection failure keeps the URL error instead of generic live unavailable" would then assert the kind plus the preserved domain and code.

3. The diagnostics context does not reach AVIOReader requests. The HLS readers store their context explicitly, but AVIOReader never receives one: every delegate (Probe, ChunkFetch, PersistentRead, Streaming, TailPrefetch) reads the ambient task-local when it is constructed. The size probes run on Thread.detachNewThread (around AVIOReader.swift:4438 / 4451), and reconnects, prefetch and chunk fetches run on the FFmpeg read thread or the prefetch threads, where the task-local is nil. So on the progressive path, which is the most common one for a Jellyfin direct play, most events arrive without a context. Capturing SourceNetworkDiagnostics.context once in AVIOReader.init (or passing options.sourceDiagnosticsContext where the reader is built) and handing it to each delegate would make it hold, ideally with a test that drives a reconnect and checks the context on the second request.

4. diagnosticReason matches on message text. Rewording any playlistInvalid(reason:) or segmentDecryptFailed(reason:) string silently drops it into the fallback bucket. A test that collects the reason: literals thrown in Sources/ and checks that each maps to a non-fallback value would catch that drift. Small thing, but the whole point of the field is that it stays stable.

5. Key fetch refusals. An HTTP failure on the AES key now throws playlistUnreachable(status:) instead of segmentDecryptFailed. Publishing it as .sourceRefused with the status seems right to me, but the public terminalError then calls a key refusal a playlist one. Either a word in the doc comment for playlistUnreachable or a note in the changelog would do.

6. Public Event shape. role and phase are free Strings, and several fields are public vars on a type hosts only ever read. String-backed structs like PlaybackErrorKind (stable raw values, room to add) and let fields would make this easier to keep stable. Raw URLs in the event are fine given the documented redaction requirement.

Happy to take another look once it is rebased. Points 2 and 3 are the ones that block; 4 to 6 could also land as follow-ups if you prefer.

@yucelokan
yucelokan force-pushed the codex/source-failure-diagnostics branch 2 times, most recently from ad4b342 to 7d643f2 Compare October 21, 2026 21:27
@yucelokan

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review! I've addressed all six points and rebased onto main, keeping the existing #809 fix.

Failed live starts retain liveSourceUnavailable and the underlying cause, and request context now survives AVIO thread hops and reconnects. The diagnostic contracts, documentation, and regression coverage have also been updated.

All four CI checks are now passing. Thanks again for catching these compatibility and attribution issues!

@superuser404notfound superuser404notfound left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough follow-up, and for keeping the #809 fix intact through the rebase. Re-reviewed at 7d643f26:

Resolved

  • Live start keeps .liveSourceUnavailable (HTTP and TLS answers still type as sourceRefused / sourceRateLimited / sourceCertificateRejected), with the cause carried in underlyingCode.
  • AVIOReader captures the context at init and hands it to every delegate explicitly, so the FFmpeg and URLSession threads no longer depend on the TaskLocal. The reconnect test covers it.
  • Reason literals have a drift test, the key 403 stays distinct, and Role / Phase are open RawRepresentable structs on an immutable Event.

One new blocker: a transport failure at open no longer reaches the reconnect ladder

failIfStreamingRefused now also throws the connection's URL error when nothing arrived (AVIOReader.swift:1686-1692), and the live open calls it on the no-data path (:1407). On main a live open whose first connection ends without a status falls through to the read loop, which rides it out; the comment on refusalOfLiveOpenConnection (:1630) still says exactly that. With this change, a first connection that is reset or times out (an IPTV or Jellyfin tuner still spinning up does this) fails the load on the spot instead of reconnecting.

The same check runs on the sequential and forward-only VOD opens (:1293, :1426, :1479, :1530), which is why two existing tests had to change: SequentialOriginReaderTests.shortBodyReportsEIO now accepts URLError.networkConnectionLost from open(), and StreamingBufferBoundTests.droppedConnectionHasNoCause switched its fixture from serveThenDrop to serveThenGoSilent. Those tests pinned the reconnect behaviour, so they should stay as they were.

Suggested shape: keep the URL code as diagnostic data only. Record it, and when the reader finally gives up (the live probe fail-fast in AetherEngine.swift, or the read loop's existing give-up) attach it to the error that is thrown then, instead of throwing it from the open. That keeps "Live initial connection failure keeps liveSourceUnavailable and the original URL error" true without changing when the engine gives up. The TLS branch (-1206 ... -1200) can stay as is, since a trust failure cannot be fixed by reconnecting.

Housekeeping
CHANGELOG.md conflicts with main again after the #847 merge; everything else merges cleanly. Happy to resolve that one myself at merge time.

Failed initial live probes keep liveSourceUnavailable with the original
URL error in underlyingDomain/underlyingCode; HTTP and TLS refusals stay
typed. AVIO readers capture the diagnostics context at init and hand it
to every delegate. HLS reasons are finite and pinned by a literal drift
test, refused AES keys keep playlistUnreachable(status:), and source
network events are immutable with open Role/Phase values.

A transport failure at open is not a reason to fail it: the open and
the read loop ride resets and timeouts out on the reconnect ladder as
before, and the reader only records the last URL error. The live start
attaches that record when it finally gives up (superuser404notfound#796's typed status path
is kept). Only a TLS trust failure, which no reconnect can fix, fails
the open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yucelokan
yucelokan force-pushed the codex/source-failure-diagnostics branch from 7d643f2 to b794e04 Compare October 24, 2026 07:56
yucelokan added a commit to yucelokan/AetherEngine that referenced this pull request Oct 24, 2026
Bring the PR superuser404notfound#818 second-review fix (b794e04) into develop. A reset or
timeout at open no longer fails it: the open and the read loop ride the
failure out as upstream does, and the reader only records the last URL
error. A failed live start attaches that record to liveSourceUnavailable
when it gives up; a TLS trust failure still fails the open. The two
reconnect regressions return to their upstream form, and a live open
whose first connection resets after its header now pins the behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yucelokan

Copy link
Copy Markdown
Contributor Author

Thanks for catching that. Updated at b794e04c (rebased onto main, so the CHANGELOG conflict is resolved as well):

  • The open no longer throws a connection's URL error. A reset or timeout at open goes back to the reconnect ladder as on main; the reader only records the last URL error (cleared once data arrives), and the live start fail-fast attaches it to liveSourceUnavailable as underlyingDomain / underlyingCode when it gives up. "Live initial connection failure keeps liveSourceUnavailable and the original URL error" still holds; it now runs the full ladder (~16 s locally) before failing.
  • The TLS branch (-1206…-1200) is unchanged.
  • SequentialOriginReaderTests.shortBodyReportsEIO and StreamingBufferBoundTests.droppedConnectionHasNoCause are back to main's versions byte for byte; the follow-up commit that changed the latter is dropped.
  • New regression: a live open whose first connection resets after its header must reconnect and read (URLSession silently retries a GET that dies before one). It fails on 7d643f26 with -1005 and passes now.
  • The live fail-fast keeps fix(live): a live stream read off its URL rides out an IPTV panel's refusals #796: an HTTP answer still publishes sourceRefused / sourceRateLimited and throws the reader's httpStatus.

Scoped local runs (source failure diagnostics, sequential/forward-only reconnects, HLS diagnostics, PlaybackErrorInfo, live refusal retry, TLS classification, documentation tests): 95 passing.

A held HTTP/1 open that failed after a response head (an oversized
declared body, a malformed answer, a redirect limit) recorded its
terminal event against the original URL and without a status. It now
reports the hop that answered, so a failure behind a redirect names the
redirect target, with the status its head carried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yucelokan added a commit to yucelokan/AetherEngine that referenced this pull request Oct 24, 2026
Retire the fork-only held HTTP/1 event shape (completed + cancelled for
abandonment, drain and discarded bytes counted) in favour of PR superuser404notfound#818's
reviewed contract, plus the failing-hop fix prepared for upstream on
codex/held-http-failed-open-evidence (b70de87). Develop now carries no
engine change that is not in an upstream branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yucelokan

Copy link
Copy Markdown
Contributor Author

One small addition on top (b70de870), in the held HTTP/1 code this PR introduces: an open that failed after a response head (an oversized declared body, a malformed answer, the redirect limit) recorded its terminal event against the original URL and without a status. It now reports the hop that answered, with the status its head carried, so a failure behind a redirect names the redirect target. heldBodyLimit now also checks the terminal event's status, and a new test covers a failure after a redirect; both fail without the change. One sentence in docs/api.md and the CHANGELOG entry describe it.

The swift test (macOS) failure on the previous push is VideoPrerollTests "A decoder still producing frames holds the clock past the plain timeout", which fails the same way on main's own CI (bc471ded, 88a459e1); everything this PR touches passed.

@superuser404notfound
superuser404notfound merged commit aa0bf3b into superuser404notfound:main Oct 25, 2026
3 of 4 checks passed
superuser404notfound pushed a commit that referenced this pull request Oct 25, 2026
lastTransportFailureCode was inserted between the comment and the
property it describes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJcXkGQrMiBPixVEFTSBJS
@superuser404notfound

Copy link
Copy Markdown
Owner

Merged as aa0bf3b3, thank you! The reconnect ladder is back exactly as it was, and the recorded URL code only reaching liveSourceUnavailable when the live start finally gives up is the shape I was hoping for. The held HTTP/1 hop attribution in b70de870 is a nice catch on top.

The branch conflicted with today's #859 / #860 merges (AVIOReader.init now takes both diagnosticContext and routesDiscMediaType, your live no-data check sits after the DASH and disc-image hand-offs, both CHANGELOG entries kept); the full suite is green on the merged tree. One small follow-up on main, 370b0e4d: lastTransportFailureCode had landed between the DMX-107 doc comment and lastReadFailure, so the comment moved back to its property.

It goes out with the next engine release.

yucelokan added a commit to yucelokan/AetherEngine that referenced this pull request Oct 27, 2026
Merge published 7.48.0. It contains PR superuser404notfound#818 (merged as aa0bf3b) with
everything develop carried beyond 7.47.0, so the tree now matches 7.48.0
exactly apart from MODIFICATIONS.md and COPYING.GPL3. The engine has no
fork-only change left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants