Skip to content

Stop crafted headers from growing memory or raising - #49

Merged
czj merged 1 commit into
mainfrom
claude/upbeat-carson-1rwsu6
Nov 6, 2026
Merged

czj merged 1 commit into
mainfrom
claude/upbeat-carson-1rwsu6

Conversation

@czj

@czj czj commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Crafted request headers could grow a process for good or make it raise. Version checks passed every version read from a user agent or a client hint to Gem::Version.new, whose class-level cache keeps each distinct string for the life of the process, and platform.name runs such a check on every Mac user agent: 20,000 user agents with unique macOS versions grew a process by 200 MB. The Accept-Language cache kept 1,000 parsed headers of any length, so 1,000 headers of 2 KB listing 1,000 languages each held 107 MB. Bytes that are not valid UTF-8 (a forged header, a user agent read back from a log or a database) made Browser.new, Bot.why?, accept_language, the client hints, the Cloudflare headers and the middleware's Accept check raise ArgumentError.

Version checks now build a Gem::Version subclass, which RubyGems does not cache, with a #release that does not cache either, and remember the last short version per object so that a cached platform stays as fast as before. A header listing more than 32 languages is parsed on each request instead of cached. The user agent and the Accept-Language header are scrubbed (each invalid byte becomes U+FFFD), and such client hints and Cloudflare headers are ignored. Browser.new also checks the size before its first regexp, as browser did.

  • No answer moves: the corpus snapshot is byte-identical to main, and 180 version checks per corpus user agent (3.6 million in all, ~> and prereleases included) give the same answers.
  • Bounded memory: 60,000 crafted Mac user agents now grow a process by 5 MB, the size of the user agent cache, instead of about 600 MB and growing; 1,000 crafted Accept-Language headers grow it by 0.6 MB instead of 107 MB.
  • Oversized user agents: rejecting one takes 3 µs whatever its size, instead of 1.4 ms for 80 KB and 19 ms for 1 MB.
  • resolv advisory: bot.verified?(ip, reverse_dns: true) reads DNS answers from a server the requester picks, and resolv before 0.7.2 is exposed to CVE-2026-80212 and CVE-2025-24294; the README and the guide now say to bundle resolv 0.7.2 or later before turning it on. Adding resolv as a runtime dependency is left for a maintainer to decide.
  • Gemspec wording: the summary and description drop "nearly twice as fast", which the benchmarks contradict (five times).
Reproductions, tests and benchmarks

Reproductions (Ruby 3.3.6, RubyGems 3.5.22; RubyGems 4.0.24 still has @@all and @@release):

20_000.times { |i| Browser.new("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_#{"1_" * 900}#{i})").platform.name }
# main: Gem::Version cache +20,001 entries, RSS +200.6 MB. Branch: +1 entry (">= 10.12"), RSS +5.1 MB

999.times { |i| Browser.new("Mozilla", accept_language: ("a," * 1020) + i.to_s).accept_language }
# main: 1,019,979 languages kept, RSS +106.9 MB. Branch: none kept, RSS +0.6 MB

Browser.new("Mozilla/5.0 (Windows NT 10.0) Chrome/120 \xFF".dup.force_encoding("UTF-8")).name
# main: ArgumentError (invalid byte sequence in UTF-8). Branch: "Chrome"

ReDoS: every regexp applied to a header was run on adversarial strings (repeated (Linux; Android; , Trident/, android, mobile, Mac OS X, GSA/., spaces, non-ASCII, and more) at 1k, 10k and 100k bytes with the size limit lifted. Time grew linearly in every case (at most 2.6 µs per byte) on Ruby 3.3, the oldest supported Ruby. No change was needed.

Each new test fails on main (7 errors and 1 failure) and passes here: test/unit/hostile_headers_test.rb, plus new tests in cache_test.rb, client_hints_test.rb, cloudflare_test.rb and request_headers_test.rb. rake test: 662 runs, no failures, on both the Gemfile (Rails 8.1) and gemfiles/rails_8_0.gemfile. RuboCop is clean, and docs:highlight:check and docs:render:check pass. bundler-audit finds nothing in either lock file.

bench/request.rb, Ruby 3.3.6 without JIT, three alternating runs each, medians: 2,000 distinct user agents 91.4 µs on main and 88.6 µs here, 58 objects on both; 20 repeated user agents 10.2 µs on main and 10.2 µs here, 12 objects on both. bench/signals.rb is within noise, with the same objects on every line. No Ruby with YJIT was available on the machine, so the YJIT runs are still to do. The published numbers are unchanged.

  • Tests cover the change (a bug fix comes with a test that fails without it)
  • Output unchanged, or every difference is intended and listed above

Version checks passed each version read from a user agent or a client
hint to Gem::Version.new, whose class-level cache keeps every distinct
string for the life of the process: 20,000 user agents with unique macOS
versions grew a process by 200 MB, and platform.name runs such a check on
every Mac user agent. They now use a Gem::Version subclass, which RubyGems
does not cache, with a #release that does not cache either.

The Accept-Language cache kept every parsed header: a thousand 2 KB
headers listing a thousand languages each held about 100 MB. A header
listing more than 32 languages is now parsed on each request instead.

Bytes that are not valid UTF-8 made Browser.new, Bot.why?, the
Accept-Language parser, the client hints, the Cloudflare headers and the
middleware's Accept check raise ArgumentError. The user agent and the
Accept-Language header are now scrubbed, and such client hints and
Cloudflare headers ignored.

Browser.new also checks the user agent's size before its first regexp,
as browser did, so rejecting a 1 MB user agent takes microseconds instead
of 19 ms.

The gemspec summary and description drop a speed claim the README
contradicts ("nearly twice as fast", where the benchmarks say five).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLwjZyBMrd7ZWPbSW6dDXp
@czj
czj enabled auto-merge (squash) November 6, 2026 11:53
@czj
czj merged commit 1b927db into main Nov 6, 2026
16 checks passed
@czj
czj deleted the claude/upbeat-carson-1rwsu6 branch November 6, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants