Repository navigation
Stop crafted headers from growing memory or raising - #49
Merged
Merged
Conversation
Version checks passed each version read from a user agent or a client
hint to Gem::Version.new, whose class-level cache keeps every distinct
string for the life of the process: 20,000 user agents with unique macOS
versions grew a process by 200 MB, and platform.name runs such a check on
every Mac user agent. They now use a Gem::Version subclass, which RubyGems
does not cache, with a #release that does not cache either.
The Accept-Language cache kept every parsed header: a thousand 2 KB
headers listing a thousand languages each held about 100 MB. A header
listing more than 32 languages is now parsed on each request instead.
Bytes that are not valid UTF-8 made Browser.new, Bot.why?, the
Accept-Language parser, the client hints, the Cloudflare headers and the
middleware's Accept check raise ArgumentError. The user agent and the
Accept-Language header are now scrubbed, and such client hints and
Cloudflare headers ignored.
Browser.new also checks the user agent's size before its first regexp,
as browser did, so rejecting a 1 MB user agent takes microseconds instead
of 19 ms.
The gemspec summary and description drop a speed claim the README
contradicts ("nearly twice as fast", where the benchmarks say five).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLwjZyBMrd7ZWPbSW6dDXp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Crafted request headers could grow a process for good or make it raise. Version checks passed every version read from a user agent or a client hint to
Gem::Version.new, whose class-level cache keeps each distinct string for the life of the process, andplatform.nameruns such a check on every Mac user agent: 20,000 user agents with unique macOS versions grew a process by 200 MB. The Accept-Language cache kept 1,000 parsed headers of any length, so 1,000 headers of 2 KB listing 1,000 languages each held 107 MB. Bytes that are not valid UTF-8 (a forged header, a user agent read back from a log or a database) madeBrowser.new,Bot.why?,accept_language, the client hints, the Cloudflare headers and the middleware's Accept check raiseArgumentError.Version checks now build a
Gem::Versionsubclass, which RubyGems does not cache, with a#releasethat does not cache either, and remember the last short version per object so that a cached platform stays as fast as before. A header listing more than 32 languages is parsed on each request instead of cached. The user agent and the Accept-Language header are scrubbed (each invalid byte becomes U+FFFD), and such client hints and Cloudflare headers are ignored.Browser.newalso checks the size before its first regexp, as browser did.main, and 180 version checks per corpus user agent (3.6 million in all,~>and prereleases included) give the same answers.bot.verified?(ip, reverse_dns: true)reads DNS answers from a server the requester picks, and resolv before 0.7.2 is exposed to CVE-2026-80212 and CVE-2025-24294; the README and the guide now say to bundle resolv 0.7.2 or later before turning it on. Adding resolv as a runtime dependency is left for a maintainer to decide.Reproductions, tests and benchmarks
Reproductions (Ruby 3.3.6, RubyGems 3.5.22; RubyGems 4.0.24 still has
@@alland@@release):ReDoS: every regexp applied to a header was run on adversarial strings (repeated
(Linux; Android;,Trident/,android,mobile,Mac OS X,GSA/., spaces, non-ASCII, and more) at 1k, 10k and 100k bytes with the size limit lifted. Time grew linearly in every case (at most 2.6 µs per byte) on Ruby 3.3, the oldest supported Ruby. No change was needed.Each new test fails on
main(7 errors and 1 failure) and passes here:test/unit/hostile_headers_test.rb, plus new tests incache_test.rb,client_hints_test.rb,cloudflare_test.rbandrequest_headers_test.rb.rake test: 662 runs, no failures, on both the Gemfile (Rails 8.1) andgemfiles/rails_8_0.gemfile. RuboCop is clean, anddocs:highlight:checkanddocs:render:checkpass. bundler-audit finds nothing in either lock file.bench/request.rb, Ruby 3.3.6 without JIT, three alternating runs each, medians: 2,000 distinct user agents 91.4 µs on main and 88.6 µs here, 58 objects on both; 20 repeated user agents 10.2 µs on main and 10.2 µs here, 12 objects on both.bench/signals.rbis within noise, with the same objects on every line. No Ruby with YJIT was available on the machine, so the YJIT runs are still to do. The published numbers are unchanged.