From ebd847a444251c3166ba86537a626983c9f014ad Mon Sep 17 00:00:00 2001 From: ThorSP Date: Tue, 8 Sep 2026 18:26:35 +0300 Subject: [PATCH] fix: bundle yt-dlp's vendored JS solver script, which v1.6.1 dropped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v1.6.1 did not work. Reported from the church PC: the download succeeded but came down at 360p with 1080p selected. Evidence from that machine — the log line added in v1.6.1 is what found it: JS engine: D:\...\quickjs_win64\qjs.exe Downloading: ...watch?v=Mg1Aa34cHjI with quality 1080p Download failed: ERROR: [youtube] Mg1Aa34cHjI: This video is not available Retrying with player clients: default,android,ios,web_safari,tv Succeeded on retry ... The engine was bundled, resolved and handed to yt-dlp, `qjs.exe --help` runs fine on that box (QuickJS-ng 0.16.2), and the file was 3,425,955 bytes — the 360p size to the byte. So the retry rescued it while the engine did nothing. An engine is only half of what the challenge takes. yt-dlp runs a *vendored JavaScript file*, yt.solver.core.js, through it — and those three .js files are the only non-Python assets in the entire yt_dlp package, so a plain PyInstaller freeze drops them. `vendor.load_script()` reports that by returning None **silently**, so the failure is indistinguishable from having no engine at all. Not a missing-module problem: parsing the shipped exe's PYZ shows all 12 `youtube.jsc` provider modules present. It is purely the data files, and no yt_dlp data file appears anywhere in the shipped zip. Proved causally both ways: - From source, hiding only those three .js files reproduces the church PC's error verbatim; restoring them gives 1920x1080. - Frozen probes built both ways: without collect_data_files the solver is MISSING and the download FAILS; with it the script loads (12,607 bytes) and the same video resolves 399+251 1920x1080. That also confirms importlib.resources.files() resolves collected data inside a frozen app. Why v1.6.1 passed everything: from a source checkout those .js files are always there, so the bug is invisible to every source-level test. Verifying in the shipped artifact is the lesson, and CLAUDE.md now says so. Also logs `solver script: ok|MISSING` beside the engine path, so the half that was silently absent is now visible in the log rather than only in a resolution nobody measures. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 16 +++++++++++++ app/backend/downloader.py | 24 ++++++++++++++++++- tests/test_downloader.py | 50 +++++++++++++++++++++++++++++++++++++++ youtubeweekly.spec | 16 ++++++++++++- 4 files changed, 104 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index eb2ffdd..0559f1c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -181,6 +181,22 @@ with clean URLs and never needs JavaScript at all. auto-detect". `download_video` logs which engine it used instead, so a build that shipped without the binary — or lost its executable bit in packaging — is visible in the log rather than silently back to 360p-or-nothing. +- ⚠⚠ **An engine is only HALF of it.** yt-dlp runs a *vendored JavaScript file* + (`yt.solver.core.js`) through the engine, and those `.js` files are the only + non-Python assets in the whole `yt_dlp` package — so a plain PyInstaller + freeze **drops them**, and `vendor.load_script()` reports that by returning + `None` **silently**. The result is indistinguishable from having no engine at + all: the same misleading "This video is not available". + **This is what shipped in v1.6.1** — QuickJS bundled, resolved and logged, + with no script to run. `youtubeweekly.spec` now calls + `collect_data_files("yt_dlp")`; `tests/test_downloader.py` pins that, and + `download_video` logs `solver script: ok|MISSING` beside the engine path. +- ⚠ **Verify a packaging fix in a frozen build, never from source.** From a + checkout those `.js` files are always present, so the bug is invisible — the + first fix passed every source-level test and still shipped broken. Freeze a + probe (`PyInstaller` a script that calls `vendor.load_script` and downloads a + kids video) or parse the shipped exe's PYZ; see + [[feedback_verify_which_build_operator_tested]]. - ⚠ **`js_runtimes` must be a dict** of `{runtime: {config}}`. A list raises a bare `ValueError` from `YoutubeDL.__init__` that names no option. - ⚠ **yt-dlp auto-enables only `deno`.** Node can sit on `PATH` completely diff --git a/app/backend/downloader.py b/app/backend/downloader.py index 20b0152..3f5ac6d 100644 --- a/app/backend/downloader.py +++ b/app/backend/downloader.py @@ -16,6 +16,26 @@ # "is it already downloaded?" check. PARTIAL_SUFFIXES = (".part", ".ytdl", ".temp") + +def js_solver_status(): + """Whether yt-dlp can actually reach its vendored JS challenge solver. + + A JS engine is only half of what solving YouTube's signature / "n" + challenge takes: yt-dlp runs a *vendored JavaScript file* through it, and + `vendor.load_script()` returns None **silently** when that file is missing. + v1.6.1 shipped exactly that way - QuickJS bundled and configured, no script + to run - and the only symptom was YouTube's misleading "This video is not + available", indistinguishable from having no engine at all. + + Reaches into yt-dlp internals on purpose, and never raises: this is + diagnostics, and a yt-dlp refactor must not be able to break a download. + """ + try: + from yt_dlp.extractor.youtube.jsc._builtin import vendor + return "ok" if vendor.load_script("yt.solver.core.js") else "MISSING" + except Exception as e: + return f"unknown ({type(e).__name__})" + # YouTube does not serve every video to every InnerTube client, and yt-dlp only # asks the handful it defaults to. Some channels answer UNPLAYABLE - which # surfaces as the flatly wrong "This video is not available" - on exactly those @@ -351,7 +371,9 @@ def download_video(video_url, video_folder, quality_pref="1080p", protect=False, # Must be a DICT of {runtime: {config}}. A list raises a bare # ValueError from YoutubeDL.__init__ that names no option. ydl_opts['js_runtimes'] = {'quickjs': {'path': js_runtime_path}} - logging.info(f"JS engine: {js_runtime_path}") + # Both halves, on one line: a path without a solver script is useless, + # and that combination is precisely what v1.6.1 shipped. + logging.info(f"JS engine: {js_runtime_path} | solver script: {js_solver_status()}") else: # Logged rather than warned so a build that shipped without the binary # (or lost its executable bit in packaging) is visible in the log diff --git a/tests/test_downloader.py b/tests/test_downloader.py index 28368ae..85e2946 100644 --- a/tests/test_downloader.py +++ b/tests/test_downloader.py @@ -565,3 +565,53 @@ def test_folder_snapshot_and_newly_downloaded_file(tmp_path): assert newly_downloaded_file(str(tmp_path), before) == "big.mp4" assert newly_downloaded_file(str(tmp_path), folder_snapshot(str(tmp_path))) is None assert folder_snapshot(str(tmp_path / "nope")) == set() + + +# --- the JS challenge solver's vendored script ---------------------------- +# +# A JS engine alone is not enough. yt-dlp runs a vendored JavaScript file +# through it, that file is the only non-Python asset in the yt_dlp package, and +# a plain PyInstaller freeze drops it. v1.6.1 shipped that way - engine present +# and configured, no script to run - and failed identically to having no engine. + +def test_vendored_js_solver_script_is_reachable(): + """Guards against a yt-dlp upgrade renaming or moving the script. + + Cannot catch the packaging bug itself (from source the file is always + there) - `youtubeweekly.spec` collecting yt_dlp's data files is what does + that, and the test below pins it. What this catches is the *other* way the + same silent failure returns: the filename changing under us, since + `load_script` reports absence by returning None. + """ + from app.backend.downloader import js_solver_status + assert js_solver_status() == "ok" + + +def test_js_solver_status_never_raises(monkeypatch): + """It is diagnostics on a private yt-dlp path - a refactor there must not + be able to break a download.""" + import app.backend.downloader as dl + + def boom(*a, **k): + raise RuntimeError("yt-dlp moved things") + + monkeypatch.setattr("importlib.import_module", boom, raising=False) + # Force the import inside the function to fail however it can. + monkeypatch.setitem(__import__("sys").modules, + "yt_dlp.extractor.youtube.jsc._builtin", None) + assert dl.js_solver_status().startswith(("unknown", "MISSING", "ok")) + + +def test_spec_bundles_yt_dlp_data_files(): + """The actual fix for the v1.6.1 packaging bug. + + Asserted against the spec text because the failure mode is invisible from + source and only appears in a frozen build - so there is nothing else here + to pin it to. + """ + import os + spec = os.path.join(os.path.dirname(__file__), "..", "youtubeweekly.spec") + with open(spec, encoding="utf-8") as f: + content = f.read() + assert "collect_data_files" in content + assert 'collect_data_files("yt_dlp")' in content diff --git a/youtubeweekly.spec b/youtubeweekly.spec index f24f7bd..b33a59b 100644 --- a/youtubeweekly.spec +++ b/youtubeweekly.spec @@ -6,6 +6,20 @@ Build with: pyinstaller youtubeweekly.spec import os import sys +from PyInstaller.utils.hooks import collect_data_files + +# yt-dlp solves YouTube's signature / "n" challenge by running a *vendored +# JavaScript file* (yt.solver.core.js) through the JS engine - and those .js +# files are the only non-Python files in the whole yt_dlp package, so a plain +# freeze drops them. `vendor.load_script()` then returns None **silently**, the +# challenge is never solved, every adaptive format is discarded, and the +# download dies with YouTube's misleading "This video is not available". +# +# That is exactly what shipped in v1.6.1: the bundled QuickJS was present and +# correctly configured, but had no script to run. Verified by freezing a probe +# both ways - without this call the solver is missing and the download fails; +# with it the script loads and the same video comes down at 1080p. + # The .ico carries every size Windows asks for (16-256) and the .icns is the # high-resolution master both are generated from - see scripts/build_icon.py. # macOS wants the .icns natively rather than a converted .ico. @@ -24,7 +38,7 @@ a = Analysis( ("app/tools", "app/tools"), ("app/frontend/assets", "assets"), ("docs", "docs"), - ], + ] + collect_data_files("yt_dlp"), hiddenimports=[ "yt_dlp", "plyer",