[knguyen1]

Note

This PR was created with AI. The code, tests, and this description were written by an AI coding assistant.

Problem

Every post page currently fails with:

Failed to parse page JSON data: expected value at line 1 column 1 | /r/<sub>/comments/<id>/<slug>/.json?&sort=confidence&raw_json=1

Reddit's edge now answers all comments API routes with a 403 "blocked by network security" HTML page when they are called with the anonymous Android OAuth token. This affects /comments/{id}.json, /r/{sub}/comments/..., and both oauth.reddit.com and www. Other routes (/by_id/t3_{id}.json, listings, /duplicates) still work. The GraphQL comments endpoints return NOT_AUTHENTICATED, so they are not an alternative.

Possibly related: #488.

Fix

• The post itself is fetched from /by_id/t3_{id}.json, which still returns the same JSON that parse_post expects.
• Comments are built from the HTML fragments that new Reddit's own frontend uses (www.reddit.com/svc/shreddit/comments/r/{sub}/t3_{id}[/t1_{comment}]?sort=...). The new src/shreddit.rs module:
  • parses <shreddit-comment> elements with scraper into the same JSON shape the comments endpoint used to return, so parse_comments, query_comments and the templates are unchanged;
  • expands "more comments" placeholders (/svc/shreddit/more-comments/... POSTs carrying the hidden cursor) breadth-first, keeping up to 10 requests in flight and starting the next as soon as one finishes. This is bounded by CRAWL_LIMITS (200 comments, 30 expansions, 10 in flight). Unexpanded placeholders become the usual "More replies (N)" links;
  • supports comment permalinks (/comments/{id}/{slug}/{comment}) and ?context=N by expanding only placeholders under the focused comment, then trimming the tree to the focused thread;
  • caches results for 30s, like the other cached fetchers.
• In client.rs, shreddit_fragment is the only network entry point. It accepts only /svc/shreddit/ paths and handles Reddit's throttling of these endpoints:
  • Cookieless requests receive a reCAPTCHA page with status 200 after about 10 requests. Shreddit requests therefore share their own wreq cookie jar (enabled via the cookies feature), which gets about 225 requests per session window.
  • The session is rotated before it runs out (x-ratelimit-remaining < 20), and also on a 429 or captcha response, followed by a single retry.
  • Reddit's bot check answers some clients with 302 Location: <same path>?rdt=NNNNN plus an rdt cookie. Up to three redirects are followed with the original method and cursor. Targets outside /svc/shreddit/ on www.reddit.com are refused.

Trade-offs

• Each uncached post page now costs 1 to about 30 requests instead of 1. In a release build, uncached pages took 0.7–1.8s in my testing; about 0.5s of that is the first comments fragment. Cached pages are instant.
• Author flair and awards are not present in the HTML fragments, so comments render without them. Bodies, scores, timestamps, edits, mod distinguish/stickied, collapsing and nesting all work.
• I could not test quarantined subreddits against shreddit; their existing error handling for the post fetch is unchanged.
• New dependency: scraper 0.29, with default features off. Its html5ever MSRV of 1.85 is below the effective MSRV already imposed by time.

Testing

• cargo fmt --check and cargo clippy --all-targets are clean.
• There are 20 new unit tests in shreddit.rs, covering:
  • fragment and placeholder parsing, including edge cases;
  • path building and timestamp parsing;
  • tree building (dedupe, parent resolution, cycle guard);
  • breadth-first batching, focus and context trimming, and the "more" stub shape;
  • the crawler with an injected fetcher (failure fallback, expansion and comment budgets, focus mode).
• One new test in client.rs checks that non-shreddit paths are rejected.
• The full cargo test run shows the same network-dependent failures as main. Those tests are flaky against live Reddit; none of the failures come from this change.
• Manual end-to-end checks against live Reddit:
  • The originally failing r/mildlyinfuriating post loads 272 comments, nested 5 deep.
  • sort=new and sort=top work.
  • /comments/{id} works.
  • Comment permalinks work, with and without ?context=3.
  • About 20 posts across r/AskReddit, r/worldnews, r/pics and r/rust loaded back to back with no warnings, errors or rate-limit failures.
• The Dockerfile.alpine image works when run read-only as nobody.

[arch-btw]

Great idea!

Unfortunately, I'm getting this error: ⚠️ Error checking update status: TypeError: Cannot read properties of null (reading 'remove')

and:

 ERROR redlib::utils > Error page rendered: Reddit returned 403 Forbidden 
 ERROR redlib::utils > Error page rendered: Reddit returned 403 Forbidden 

Edit: tried a different IP, now I'm getting 302 found error.

[Sparronator9999]

@arch-btw I've made a commit on my fork that should fix the 302 Found error, by following the redirect similar to somewhere else in the client.rs file.

[Sparronator9999]

Okay, I've done some brief testing, and it's obvious to me that this is much slower than the original, even with release optimisations. I guess that's web scraping for ya.

The GraphQL comments endpoints return NOT_AUTHENTICATED, so they are not an alternative.

How does the official app authenticate with the GraphQL endpoints then? Does it also not load posts now because of the account requirement? EDIT: I downloaded the Reddit app for Android to see what happens, and I can't do anything without logging in to a Reddit account (which I don't have, and don't plan on creating). So, looks like they did indeed shut down the (unauthenticated) GraphQL endpoints too. Hopefully they'll realise people scraping their website HTML when they find the JSON/GraphQL endpoints don't work will(?) cost them more in the long run.

[knguyen1]

Thanks both! I pushed 1d55e3e to address this. (Like the PR, this reply was written with an AI coding assistant.)

302 Found (@arch-btw, @Sparronator9999): this is Reddit's bot check. It answers with 302 Location: <same path>?rdt=NNNNN and sets an rdt cookie, then expects the client to retry. I took the same approach as Sparronator9999's fork commit and follow the redirect, with a few changes:

• An absolute Location (https://www.reddit.com/svc/...) resolves correctly instead of being appended to the base URL.
• Redirects to anything other than a /svc/shreddit/ path on www.reddit.com (for example a login page or another host) are refused rather than followed.
• Running out of redirects returns an error instead of the "rate limited" signal. That signal would have cleared the session and thrown away the rdt cookie we had just been given.
• The redirect is a loop with a limit of three, so async_recursion isn't needed.

I couldn't trigger the 302 from my own IP, so it would help if someone who sees it could confirm the fix.

403 Forbidden: shreddit requests go to www.reddit.com. A 403 there most likely means Reddit is blocking that IP for www, as it does for some VPN and datacenter ranges, which code can't work around. If you have a moment, could you check whether https://www.reddit.com/r/rust/comments/18t5968/ opens in a browser from that IP? The full log line should also end with | <path>, which shows which request was blocked.

Error checking update status: TypeError ...: this comes from the front-end update checker, not from this PR. It's tracked in #598.

Speed: scraping does need more round trips than the old single JSON request. The crawler now keeps 10 expansion requests in flight and starts the next one as soon as any finishes, rather than waiting for a whole batch. In a release build, uncached post pages took 0.7–1.8s in my testing:

• The first comments fragment takes about 0.5s.
• Each "more comments" expansion takes about 120–200ms.
• Parsing takes a few milliseconds per fragment.

Results are cached for 30s.

[Sparronator9999]

I've updated my fork to the latest commit of this PR (not pushed to GitHub yet, as of writing). It does seem a bit faster than last time.

However, one issue I've noticed with this PR is that comment sort via the drop-down on the post page no longer works. EDIT: this issue was probably on me. I've fixed it in my merge commit of this PR.

[Sparronator9999]

Okay, I've done some more testing, and media proxied via the /preview/external-pre Redlib endpoint does not show up. Inspecting one of them in a media-heavy subreddit (/r/shittymoviedetails was the first one I could find on the popular page that was like this), I see quite the nested HTML:

An example of a comment that uses /preview/external-pre endpoints, as returned by Redlib with the comment scraper (click to expand)

```

```

(found on this comment)

Clicking on the blank space where the video is supposed to be does still work (it just opens in a new tab). It just looks like this type of media was missed when writing the comment scraper.

[knguyen1]

@Sparronator9999 thanks for the report and the markup sample! Fixed in a12eaf6. (This reply was written with an AI coding assistant.)

Comment media is wrapped in Reddit's <shreddit-player> custom element, which only renders through Reddit's JavaScript. That's why it showed up as an empty but clickable box. The scraper now replaces each player with a native <video> of the same proxied MP4 source (/preview/external-pre/...?format=mp4). The link and sizing around it are kept. GIFs autoplay, loop and stay muted, as on Reddit. Any non-GIF player gets controls instead.

Checked on your example comment and on https://www.reddit.com/r/memes/comments/1xd9q2w/that_would_be_the_final_straw/19rj2mpfyhh5w/. A new unit test builds on your sample markup.

[knguyen1]

Fixed a regression in 26c0219: busy posts only showed about 10 comments and no replies. (This reply was written with an AI coding assistant.)

There were two causes, and both depended on which browser redlib happens to emulate:

• Reddit's edge started blocking "more comments" POSTs that have no Origin or Referer header, returning its "blocked by network security" 403. With Firefox emulation those POSTs still went through, but with Chrome emulation every expansion failed. Shreddit POSTs now send the Origin, Referer and Sec-Fetch-* headers a browser sends; either Origin or Referer alone is enough to avoid the 403.
• Shreddit sends mobile browsers 10 comments per fragment instead of 25. Shreddit requests now use their own client with a desktop platform. The OAuth API client keeps its existing emulation.

Tested with all four profile/platform combinations: every expansion succeeds and first fragments have 25 comments. For example, an AskReddit post with 7.5k comments went from 10 visible comments to about 285, and the "More replies" links load their threads.

[Sparronator9999]

Sorry for the delayed comment on the latest commits - I literally didn't have the energy to even comment on this PR over the past few days thanks to a nasty cold (or worse).

Anyways, I tested the latest commits, and the images are indeed restored, however there are still issues with a few images overflowing their container. For example, a few comments on this post, but nearly every post I saw that was media heavy in the comments had this issue.

I can't tell for sure, but I think loading times did improve somewhat since I updated my fork to the latest version of this PR, thanks to the desktop client commit.