From 61d05879820df1255b20539f21401b48b0bde5fe Mon Sep 17 00:00:00 2001 From: ThorSP Date: Tue, 8 Sep 2026 19:00:36 +0300 Subject: [PATCH] fix: ship yt_dlp_ejs, which carries the solver's missing `lib` script v1.6.2 still came down at 360p. Its own log named the problem, once the diagnostic was honest enough to see it. The challenge needs a JS engine **and two scripts** - a `lib` and a `core` - and they live in different packages: core -> vendored in yt_dlp (and in yt_dlp_ejs) lib -> ONLY in yt_dlp_ejs. yt_dlp vendors bun/deno *import shims*, and there is no vendored `yt.solver.lib.js` at all. Plain `yt-dlp` does not depend on yt_dlp_ejs, so v1.6.2 shipped with the core and no lib - and a missing script is reported by returning None, silently, so it failed identically to having no engine. requirements.txt -> yt-dlp[default] (that extra pins yt-dlp-ejs==0.8.0, and yt-dlp validates the version at runtime, so yt-dlp owns the pin rather than us) spec -> collect_data_files("yt_dlp_ejs") alongside yt_dlp's Why the last two attempts "passed": my Pi verification was reading `~/.cache/yt-dlp/challenge-solver/lib.json` - 157,708 bytes that MY OWN earlier `--remote-components ejs:github` experiment had written. That cache is a script source ranked above the vendored one, so every test after it passed for the wrong reason, source and frozen alike. I had even listed the file and moved on. Re-verified with the user cache moved aside and an isolated empty cachedir: - source, no yt_dlp_ejs -> FAILS (what ships) - source, with it -> 399+251 1920x1080 - FROZEN, both packages collected -> js_solver_status() "ok", 1920x1080 - FROZEN negative control -> never reports "ok", download fails Standalone yt-dlp.exe on the church PC with the same qjs.exe gets 1080p, which is what ruled out Windows and the engine: yt-dlp.exe bundles yt_dlp_ejs. `js_solver_status` now calls exactly what yt-dlp calls - `yt_dlp_ejs.yt.solver .lib()` / `.core()` - and names each half. Its first version checked only the core, and its second looked for files on disk when yt-dlp needs the *module* importable; both logged "ok" while the download fell back to 360p. A diagnostic that can say "ok" about a broken build is worse than none. CLAUDE.md now carries the two-scripts/two-packages shape, the cache trap, and the rule that a positive-only frozen result cannot distinguish a real fix from a contaminated environment - build the negative control too. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 3 +++ CHANGELOG_ro.md | 3 +++ CLAUDE.md | 42 +++++++++++++++++++++------------ app/backend/downloader.py | 49 ++++++++++++++++++++++++++++++--------- requirements.txt | 6 ++++- tests/test_downloader.py | 22 ++++++++++++++---- youtubeweekly.spec | 13 +++++++---- 7 files changed, 103 insertions(+), 35 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ef1a75a..1477340 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,8 @@ # Changelog +## Unreleased +- **Children's Videos Really Do Arrive in Full Quality Now**: The last release said it had fixed this and hadn't — those videos still came down at 360p instead of the quality you picked. YouTube's check needs two pieces of code to run, and the app was only carrying one. It now ships both, so they download at the full 1080p + ## v1.6.2 - **Children's Videos Download Again, in Full Quality**: Videos marked "Made for Kids" on YouTube — children's songs, Bible stories and the like — all failed with "This video is not available", even though they play perfectly in a browser. YouTube holds those back unless the app can run a piece of its code, so the app now carries the small engine needed. They download at full 1080p again, and any link that still refuses is retried a second way before the app gives up diff --git a/CHANGELOG_ro.md b/CHANGELOG_ro.md index abe102d..5b48894 100644 --- a/CHANGELOG_ro.md +++ b/CHANGELOG_ro.md @@ -1,5 +1,8 @@ # Istoric versiuni +## Unreleased +- **Videoclipurile pentru copii vin acum chiar la calitate maximă**: Versiunea anterioară spunea că a rezolvat asta, dar nu o făcuse — videoclipurile veneau tot la 360p, nu la calitatea aleasă. Verificarea YouTube are nevoie de două bucăți de cod, iar aplicația includea doar una. Acum le include pe amândouă, așa că se descarcă la 1080p complet + ## v1.6.2 - **Videoclipurile pentru copii se descarcă din nou, la calitate maximă**: Videoclipurile marcate „Made for Kids” pe YouTube — cântece pentru copii, istorioare biblice și altele — eșuau toate cu „Acest videoclip nu este disponibil”, deși se redau perfect în browser. YouTube le reține dacă aplicația nu poate rula o bucată din codul lui, așa că aplicația include acum micul motor necesar. Se descarcă din nou la 1080p complet, iar un link care tot refuză este reîncercat pe a doua cale înainte ca aplicația să renunțe diff --git a/CLAUDE.md b/CLAUDE.md index 0559f1c..8a6d88b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -181,22 +181,34 @@ with clean URLs and never needs JavaScript at all. auto-detect". `download_video` logs which engine it used instead, so a build that shipped without the binary — or lost its executable bit in packaging — is visible in the log rather than silently back to 360p-or-nothing. -- ⚠⚠ **An engine is only HALF of it.** yt-dlp runs a *vendored JavaScript file* - (`yt.solver.core.js`) through the engine, and those `.js` files are the only - non-Python assets in the whole `yt_dlp` package — so a plain PyInstaller - freeze **drops them**, and `vendor.load_script()` reports that by returning - `None` **silently**. The result is indistinguishable from having no engine at - all: the same misleading "This video is not available". - **This is what shipped in v1.6.1** — QuickJS bundled, resolved and logged, - with no script to run. `youtubeweekly.spec` now calls - `collect_data_files("yt_dlp")`; `tests/test_downloader.py` pins that, and - `download_video` logs `solver script: ok|MISSING` beside the engine path. +- ⚠⚠ **An engine is only HALF of it, and the other half is TWO scripts from + TWO packages.** yt-dlp runs a `lib` script and a `core` script through the + engine. `yt_dlp` vendors the **core** plus bun/deno *import shims* — there is + **no vendored `yt.solver.lib.js`**, so the `lib` script exists only in the + separate **`yt_dlp_ejs`** package, which plain `yt-dlp` does not depend on. + All of them are **data files**, which a freeze drops unless collected, and a + missing one is reported by returning `None` **silently**. Missing either fails + identically to having no engine: "This video is not available". + So `requirements.txt` uses **`yt-dlp[default]`** (that extra pins + `yt-dlp-ejs==0.8.0`, and yt-dlp validates the version at runtime — let yt-dlp + own that pin) and the spec collects **both** `collect_data_files("yt_dlp")` + and `collect_data_files("yt_dlp_ejs")`. + **v1.6.1 shipped with neither script; v1.6.2 shipped with the core but no + lib** — engine bundled, resolved and logged, still 360p. +- ⚠⚠ **Beware yt-dlp's solver CACHE when verifying.** `~/.cache/yt-dlp/challenge-solver/` + is a script source in its own right, ranked above the vendored one. A single + earlier run with `--remote-components ejs:github` populates it, and every + later test then passes for the wrong reason — which is exactly how v1.6.2 was + "proved" working. **Always pass an isolated, empty `cachedir`** (and move the + user cache aside) when testing the solver, or you are measuring your own + earlier experiment. - ⚠ **Verify a packaging fix in a frozen build, never from source.** From a - checkout those `.js` files are always present, so the bug is invisible — the - first fix passed every source-level test and still shipped broken. Freeze a - probe (`PyInstaller` a script that calls `vendor.load_script` and downloads a - kids video) or parse the shipped exe's PYZ; see - [[feedback_verify_which_build_operator_tested]]. + checkout those data files are always present, so the bug is invisible — two + fixes passed every source-level test and still shipped broken. Freeze a probe + (`PyInstaller` a script that calls `js_solver_status()` and downloads a kids + video, with an isolated cachedir) and build a **negative control** too: a + positive-only result cannot tell a real fix from a contaminated environment. + See [[feedback_verify_which_build_operator_tested]]. - ⚠ **`js_runtimes` must be a dict** of `{runtime: {config}}`. A list raises a bare `ValueError` from `YoutubeDL.__init__` that names no option. - ⚠ **yt-dlp auto-enables only `deno`.** Node can sit on `PATH` completely diff --git a/app/backend/downloader.py b/app/backend/downloader.py index 3f5ac6d..3e98902 100644 --- a/app/backend/downloader.py +++ b/app/backend/downloader.py @@ -18,24 +18,51 @@ def js_solver_status(): - """Whether yt-dlp can actually reach its vendored JS challenge solver. + """Which of the two challenge-solver scripts yt-dlp can actually reach. - A JS engine is only half of what solving YouTube's signature / "n" - challenge takes: yt-dlp runs a *vendored JavaScript file* through it, and - `vendor.load_script()` returns None **silently** when that file is missing. - v1.6.1 shipped exactly that way - QuickJS bundled and configured, no script - to run - and the only symptom was YouTube's misleading "This video is not - available", indistinguishable from having no engine at all. + Solving YouTube's signature / "n" challenge takes a JS engine **and two + scripts** - a `lib` and a `core` - which live in different packages: - Reaches into yt-dlp internals on purpose, and never raises: this is - diagnostics, and a yt-dlp refactor must not be able to break a download. + * `core` - vendored inside yt_dlp, and also in yt_dlp_ejs + * `lib` - **only** in the separate `yt_dlp_ejs` package. yt_dlp vendors + bun/deno *import shims*, not a usable `yt.solver.lib.js`. + + So a QuickJS build needs `yt_dlp_ejs`; `requirements.txt` pulls it via + `yt-dlp[default]` and the spec collects its data files. Scripts are data + files, a freeze drops them unless collected, and a missing one is reported + by returning None - **silently**. Missing either fails identically, with + YouTube's misleading "This video is not available". + + Calls exactly what yt-dlp calls (`yt_dlp_ejs.yt.solver.lib()` / `.core()`) + rather than probing for files. Two earlier versions of this function + checked the wrong thing and logged "ok" while the download still fell back + to 360p - once by checking only the core, once by looking for the files on + disk when yt-dlp needs the *module* to be importable. + + Never raises: this is diagnostics, and a yt-dlp refactor must not be able + to break a download. """ try: - from yt_dlp.extractor.youtube.jsc._builtin import vendor - return "ok" if vendor.load_script("yt.solver.core.js") else "MISSING" + import yt_dlp_ejs.yt.solver as solver + + def loads(fn): + try: + return bool(fn()) + except Exception: + return False + + lib, core = loads(solver.lib), loads(solver.core) + if lib and core: + return "ok" + return f"INCOMPLETE (lib={'ok' if lib else 'MISSING'}, core={'ok' if core else 'MISSING'})" + except ImportError: + # No yt_dlp_ejs at all: the lib script is unreachable, so the challenge + # cannot be solved no matter what the vendored core offers. + return "INCOMPLETE (yt_dlp_ejs not installed - lib script unreachable)" except Exception as e: return f"unknown ({type(e).__name__})" + # YouTube does not serve every video to every InnerTube client, and yt-dlp only # asks the handful it defaults to. Some channels answer UNPLAYABLE - which # surfaces as the flatly wrong "This video is not available" - on exactly those diff --git a/requirements.txt b/requirements.txt index 0323f1a..e6a9a31 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,9 @@ pytest -yt-dlp>=2026.08.19 +# [default] is what pulls yt-dlp-ejs, which carries the challenge solver's +# `lib` script. yt-dlp pins it exactly (yt-dlp-ejs==0.8.0) and validates the +# version at runtime, so letting yt-dlp own that pin is what keeps the two in +# step. Plain `yt-dlp` omits it and the app cannot solve the challenge at all. +yt-dlp[default]>=2026.08.19 plyer pyobjus; sys_platform == "darwin" screeninfo diff --git a/tests/test_downloader.py b/tests/test_downloader.py index 85e2946..5cd76b6 100644 --- a/tests/test_downloader.py +++ b/tests/test_downloader.py @@ -602,16 +602,30 @@ def boom(*a, **k): assert dl.js_solver_status().startswith(("unknown", "MISSING", "ok")) -def test_spec_bundles_yt_dlp_data_files(): - """The actual fix for the v1.6.1 packaging bug. +def test_spec_bundles_both_solver_script_packages(): + """The fix for the v1.6.1 and v1.6.2 packaging bugs. Asserted against the spec text because the failure mode is invisible from source and only appears in a frozen build - so there is nothing else here - to pin it to. + to pin it to. Both packages are needed: yt_dlp carries the `core` script, + yt_dlp_ejs the `lib` one, and either missing fails identically. """ import os spec = os.path.join(os.path.dirname(__file__), "..", "youtubeweekly.spec") with open(spec, encoding="utf-8") as f: content = f.read() - assert "collect_data_files" in content assert 'collect_data_files("yt_dlp")' in content + assert 'collect_data_files("yt_dlp_ejs")' in content + + +def test_requirements_pull_the_solver_lib_package(): + """`yt.solver.lib.js` exists ONLY in yt_dlp_ejs, and plain `yt-dlp` does + not depend on it - which is why v1.6.2 shipped unable to solve anything. + `[default]` is the extra that pulls it, and yt-dlp pins the exact version it + validates at runtime, so yt-dlp must own that pin rather than us.""" + import os + req = os.path.join(os.path.dirname(__file__), "..", "requirements.txt") + with open(req, encoding="utf-8") as f: + lines = [l for l in f if not l.strip().startswith("#")] + assert any("yt-dlp[default]" in l for l in lines), \ + "requirements.txt must use yt-dlp[default] so yt_dlp_ejs comes with it" diff --git a/youtubeweekly.spec b/youtubeweekly.spec index b33a59b..6d11c84 100644 --- a/youtubeweekly.spec +++ b/youtubeweekly.spec @@ -16,9 +16,14 @@ from PyInstaller.utils.hooks import collect_data_files # download dies with YouTube's misleading "This video is not available". # # That is exactly what shipped in v1.6.1: the bundled QuickJS was present and -# correctly configured, but had no script to run. Verified by freezing a probe -# both ways - without this call the solver is missing and the download fails; -# with it the script loads and the same video comes down at 1080p. +# correctly configured, but had no script to run. +# +# It takes TWO scripts, a `lib` and a `core`, and they come from different +# packages. yt_dlp vendors only the core plus bun/deno *import shims* - there is +# no vendored `yt.solver.lib.js`, so a QuickJS build needs `yt_dlp_ejs`, whose +# lib.min.js/core.min.js are likewise data files a freeze would drop. Missing +# either one fails identically and silently. v1.6.2 shipped with the core but no +# lib, which is why it still came down at 360p. # The .ico carries every size Windows asks for (16-256) and the .icns is the # high-resolution master both are generated from - see scripts/build_icon.py. @@ -38,7 +43,7 @@ a = Analysis( ("app/tools", "app/tools"), ("app/frontend/assets", "assets"), ("docs", "docs"), - ] + collect_data_files("yt_dlp"), + ] + collect_data_files("yt_dlp") + collect_data_files("yt_dlp_ejs"), hiddenimports=[ "yt_dlp", "plyer",