From 5cfc9dde9bd67ad099bb7865352943b2ff541aed Mon Sep 17 00:00:00 2001 From: Diego Algorta Date: Fri, 16 Oct 2026 17:07:10 -0300 Subject: [PATCH 1/3] feat: every capture becomes an assertion with a status The outcome of a screenshot was decided in five unrelated places (the tally in Reporting, its side hashes for new/re-recorded captures, the HTML reporter, the report template, ScreenshotAssertion#validate), and captures that were not compared never became assertions at all. The matcher returned nil for them and tallied them on the side, and the registry dropped anything without a comparison. So reporters never saw new or re-recorded screenshots, although docs/reporters.md and docs/snapdiff.md said they did. Now every capture is a registered ScreenshotAssertion: - `status` is derived on each call, never memoized: :new when there is no comparison, :passed unless the comparison differs, :changed when it does, :rerecorded for a differing capture under `record: :all`. It reads `compare.difference` and never forces a comparison, so asking for a status stays cheap and free of side effects. `verdict` is `status` after judging: the one place that forces the comparison, used by `validate` and `failed_assertions`. The tally forces every comparison itself when the test ends, so reporters see settled ones. - The matcher is the single registrar. It registers the assertion it builds, including the compare-less :new one, before the `:none` raise (which stays immediate whatever `delayed` says). The DSL only calls validate! when immediate and returns `!assertion.compare.nil?`, so its return value is unchanged (false for a new screenshot). - `record_new_screenshot` stays callable (v1.13 API); it now builds and registers a :new assertion and keeps its "checked out then disappeared" raise. `new_screenshots` is derived from status; `failed_assertions` is status == :changed and not already validated. - AISimple triages only status == :changed; the documented reporter idiom and contract wording are updated to match. - Temporary: `record: :all` still discards the baseline, so its assertion has no comparison; `status` reads :rerecorded for that case through one explicit override that goes once :all compares. Immediate (delayed: false) assertions are registered by the same path, so they reach the tally and the reporters too. validate! marks an assertion as judged, so the registry's teardown pass does not fail it a second time. Not changed: the Reporting tallies (the matcher still calls record_missing_baseline / record_rerecorded_baseline), the summary line, the HTML report and template, the `:all` refusal under CI, and `:all` still not comparing. --- CHANGELOG.md | 8 ++ docs/architecture.md | 9 +- docs/reporters.md | 15 ++- docs/snapdiff.md | 10 +- lib/snap_diff/dsl.rb | 13 +-- lib/snap_diff/reporters/ai_simple.rb | 5 +- lib/snap_diff/screenshot_assertion.rb | 74 +++++++++--- lib/snap_diff/screenshot_matcher.rb | 56 +++++---- test/unit/baseline_disappeared_test.rb | 6 +- test/unit/diff_test.rb | 6 +- test/unit/dsl_test.rb | 20 ++++ test/unit/record_modes_test.rb | 33 ++++-- test/unit/registry_concurrency_test.rb | 12 +- test/unit/reporter_interplay_test.rb | 11 ++ test/unit/reporters/ai_simple_test.rb | 8 +- test/unit/screenshot_assertion_status_test.rb | 108 ++++++++++++++++++ test/unit/screenshot_matcher_test.rb | 14 ++- 17 files changed, 316 insertions(+), 92 deletions(-) create mode 100644 test/unit/screenshot_assertion_status_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index ac139364..3711e2dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -342,6 +342,14 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV rollback) ### Changed +- **Reporters now receive every screenshot, each with a `status`.** New and re-recorded + (`record: :all`) captures used to bypass them; now every capture is a + `SnapDiff::ScreenshotAssertion` whose `status` is `:passed`, `:changed`, `:new` or + `:rerecorded` (`compare` is nil when there was no baseline). Reading `status` never + forces a comparison. A custom reporter that treated "has an assertion" as "was + compared" should test `assertion.status == :changed`, as the documented idiom and + `AISimple` now do. `pending_if_new` now also sees a new screenshot under `record: :none`, + where it is registered before the "no existing screenshot" failure, not only under `:once` - **No `activesupport` at runtime.** 1.15.1's `snap_manager.rb` required `active_support/core_ext/module/attribute_accessors` while `activesupport` was only a *development* dependency, so a bundle without Rails died at diff --git a/docs/architecture.md b/docs/architecture.md index dd16ca4d..cd67a823 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -186,14 +186,15 @@ Test begins │ ├─ test body: │ └─ screenshot("name") - │ ├─ ScreenshotMatcher builds assertion - │ ├─ add to Thread-local registry (always) - │ └─ delayed=false → also validate immediately + │ ├─ ScreenshotMatcher builds an assertion for every capture + │ │ (compared, new, or re-recorded) and registers it in the + │ │ Thread-local registry — before any raise + │ └─ delayed=false → the DSL also validates it immediately │ ├─ teardown: │ ├─ SnapDiff.session.verify │ │ ├─ iterates the fiber-local assertions - │ │ ├─ calls validate on each + │ │ ├─ calls validate on each (new/re-recorded ones have no comparison and pass) │ │ └─ raises SnapDiff::ExpectationNotMet if any differed │ │ │ └─ SnapDiff.reset (always, in an ensure) diff --git a/docs/reporters.md b/docs/reporters.md index 1452aada..137fb468 100644 --- a/docs/reporters.md +++ b/docs/reporters.md @@ -102,7 +102,7 @@ Build your own reporter by implementing `record`, `finalize` and `summary`: class MyReporter def record(assertions) assertions.each do |assertion| - next unless assertion.compare&.difference&.different? + next unless assertion.status == :changed # process the failure — send to Slack, write JSON, etc. end end @@ -130,11 +130,14 @@ the framework's end-of-suite hook (`Minitest.after_run`, RSpec `after(:suite)`, finalized and then warned about. A reporter that raises is warned about and skipped — the others still run. -`record` receives every assertion, but only a compared one carries a `compare`; the rest have -`compare: nil` and no path or reason of their own. Those — each new (`reason: :new`) or -re-recorded (`reason: :rerecorded`) screenshot, with its `name` and captured `path` — are in -`SnapDiff::Reporting.uncompared`, complete by the time `finalize` runs (fork-parallel workers -included). The HTML report lists new screenshots from it. +`record` receives every assertion — one per capture. `assertion.status` says what you hold: +`:passed` (matches its committed baseline), `:changed` (differs), `:new` (no committed +baseline) or `:rerecorded` (accepted by `record: :all`). `compare` is nil for a screenshot +without a baseline; `assertion.path` is the captured file either way. Test `status`, not +`compare`, to find failures: the tally has already run every comparison by the time `record` +is called, and reading `status` never forces one. The new and re-recorded screenshots are also in +`SnapDiff::Reporting.uncompared` (each with `name`, `path` and `reason`), complete by the time +`finalize` runs (fork-parallel workers included). The HTML report lists new screenshots from it. Full details in [Custom reporters](snapdiff.md#custom-reporters). diff --git a/docs/snapdiff.md b/docs/snapdiff.md index e27bd61f..3a76fb1a 100644 --- a/docs/snapdiff.md +++ b/docs/snapdiff.md @@ -231,7 +231,7 @@ class SlackReporter # Called once per finished test, with that test's assertions. def record(assertions) assertions.each do |assertion| - next unless assertion.compare&.difference&.different? + next unless assertion.status == :changed @failures << assertion.name end end @@ -258,8 +258,12 @@ and then warned about (`[snap_diff] Reporter … failed (NoMethodError: undefine A reporter that raises is warned about and skipped — the other reporters still run. -`record` receives every assertion, but a screenshot that was not compared arrives with -`compare: nil` and nothing else. Those — the summary line's "new" and "re-recorded" — are in +`record` receives every assertion — one per capture. `assertion.status` says what you hold: +`:passed`, `:changed`, `:new` (no committed baseline) or `:rerecorded` (`record: :all`). +`compare` is nil for a screenshot without a baseline; `assertion.path` is the captured file +either way. The tally has already run every comparison by the time `record` is called, so +test `status == :changed` rather than digging into `compare`; reading `status` never forces one. The new and re-recorded +screenshots — the summary line's "new" and "re-recorded" — are also in `SnapDiff::Reporting.uncompared`, complete by the time `finalize` runs (fork-parallel workers included): diff --git a/lib/snap_diff/dsl.rb b/lib/snap_diff/dsl.rb index f92f8a32..40a982de 100644 --- a/lib/snap_diff/dsl.rb +++ b/lib/snap_diff/dsl.rb @@ -76,7 +76,8 @@ def screenshot_group(name) # so a preceding `preload_all_images` still costs its browser # round-trips while the block costs nothing. Readiness work belongs # inside the same switch as the capture it serves. - # @return [Boolean] True if the screenshot was successfully captured and processed. + # @return [Boolean] True if the screenshot was compared against a baseline; false when it had none (it is + # captured and registered as a `:new` assertion) or screenshots are disabled. # @raise [SnapDiff::ExpectationNotMet] If comparison fails and immediate validation is enabled. # @raise [SnapDiff::UnstableImage] If the image comparison is unstable. # @raise [SnapDiff::WindowSizeMismatchError] If the window size doesn't match expectations. @@ -94,15 +95,11 @@ def assert_matches_screenshot(name, skip_stack_frames: 0, **options) .new(full_name, options) .build_screenshot_assertion(skip_stack_frames: skip_stack_frames + 1) - return false unless assertion - - # Registered either way: the end-of-run tally and the reporters see - # every finished assertion, immediate ones included. An immediate one - # validates here and is a no-op when the registry verifies at teardown. - SnapDiff.session.add_assertion(assertion) + # Already registered by the matcher. An immediate one validates here + # and is a no-op when the registry verifies at teardown. assertion.validate! unless options.fetch(:delayed, SnapDiff.config.delayed) - true + !assertion.compare.nil? end # Convenience wrapper around {#assert_matches_screenshot} and {#capture_screenshot}. diff --git a/lib/snap_diff/reporters/ai_simple.rb b/lib/snap_diff/reporters/ai_simple.rb index 25b810f6..372c16d9 100644 --- a/lib/snap_diff/reporters/ai_simple.rb +++ b/lib/snap_diff/reporters/ai_simple.rb @@ -44,10 +44,9 @@ def record(assertions) return unless @backend assertions.each do |a| - difference = a.compare&.difference - next unless difference&.different? + next unless a.status == :changed - analyze_once(a.name, difference) + analyze_once(a.name, a.compare.difference) end end diff --git a/lib/snap_diff/screenshot_assertion.rb b/lib/snap_diff/screenshot_assertion.rb index ac243ea9..51012246 100644 --- a/lib/snap_diff/screenshot_assertion.rb +++ b/lib/snap_diff/screenshot_assertion.rb @@ -41,14 +41,50 @@ def self.pending_screenshots_message end class ScreenshotAssertion - attr_reader :name, :args + attr_reader :name, :args, :record_mode attr_accessor :compare, :caller - def initialize(name, **args) + def initialize(name, record_mode: nil, path: nil, **args) @name = name + @record_mode = record_mode + @path = path @args = args end + # Where the capture lives: given for a capture that was never compared, + # otherwise the comparison's. + def path = @path || compare&.image_path + + # What became of this capture, derived on every call (never memoized: + # `compare=` may still assign the comparison). Reading it never forces a + # comparison: an unprocessed one counts as :passed until #validate or the + # tally has run it. + # + # :passed compared against a committed baseline, same + # :changed compared, differs + # :new no baseline, nothing to compare + # :rerecorded `record: :all` accepted it + def status + unless compare + # `:all` still discards the baseline, so its captures have no comparison; this goes once `record: :all` compares. + return (record_mode == :all) ? :rerecorded : :new + end + return :passed unless compare.difference&.different? + + (record_mode == :all) ? :rerecorded : :changed + end + + # #status after judging: forces the comparison (which also writes the + # `.diff` and `.heatmap` files), so it is what anything that needs a + # verdict asks for. Reading #status alone never does. + def verdict + compare&.different? + status + end + + # True once #validate! has judged this assertion (an immediate one). + def validated? = !!@validated + # One-line debugging summary. Never triggers the (expensive, file-touching) # comparison itself: an unprocessed comparison shows as "pending". def inspect @@ -66,9 +102,9 @@ def inspect # Return nil for absent, matching, or suppressed comparisons; archive matching baselines. def validate return if @validated - return unless compare - if compare.different? + case verdict + when :changed # Optional contribution gate (e.g. AI triage with fail_on:): a # registered suppressor runs BEFORE the message is built, so a # waived diff never fails and a standing failure can quote what @@ -82,9 +118,12 @@ def validate # The call site goes last, after what contributors say about the diff. notes = Contributions.annotations_for(name).map { |note| " #{note[:source].upcase} triage: #{note[:text]}" } ["Screenshot does not match for '#{name}': #{compare.error_message}", *notes, call_site].compact.join("\n") - else + when :passed archive_baseline! nil + when :new, :rerecorded + # Accepted as is: the capture and its artifacts stay where they are. + nil end end @@ -167,11 +206,10 @@ def display_path(path) end class AssertionRegistry - attr_reader :assertions, :screenshot_namer, :new_screenshots + attr_reader :assertions, :screenshot_namer def initialize @assertions = [] - @new_screenshots = [] @checked_out_baselines = Set.new @screenshot_namer = SnapDiff::ScreenshotNamer.new end @@ -191,7 +229,7 @@ def baseline_checked_out?(name) end def add_assertion(assertion) - return unless assertion&.compare + return unless assertion @assertions.push(assertion) @@ -202,7 +240,13 @@ def assertions_present? !@assertions.empty? end - # "No baseline exists" reaches here for two very different reasons. One + # Names of the captures that had no baseline to compare against. + def new_screenshots + assertions.select { |assertion| assertion.status == :new }.map(&:name) + end + + # Registers a `:new` assertion for `name` (extra keywords go to the + # assertion). "No baseline exists" reaches here for two very different reasons. One # is legitimate and warned about: nothing was ever committed for this # name. The other is impossible in a correct run -- git gave us a # baseline moments ago and it is gone now -- and used to be recorded as @@ -210,18 +254,18 @@ def assertions_present? # That is the silently-wrong case measured in #217: two concurrent # tests asserting the SAME name, where one's archive_baseline! moves # the baseline the other just checked out. - def record_new_screenshot(name) + def record_new_screenshot(name, **attributes) raise SnapDiff::Error.new(<<~ERROR.chomp, caller) if baseline_checked_out?(name) The baseline for '#{name}' was checked out and then disappeared before it could be compared -- nothing was verified. Every artifact path derives from the screenshot name alone, so two tests asserting '#{name}' at the same time race on one set of files: the one that finishes first archives the baseline the other is still using. Give those screenshots distinct names, or do not run them concurrently. ERROR - @new_screenshots.push(name) + add_assertion(ScreenshotAssertion.new(name, **attributes)) end def new_screenshots_present? - !@new_screenshots.empty? + assertions.any? { |assertion| assertion.status == :new } end def verify(screenshots = assertions) @@ -231,13 +275,15 @@ def verify(screenshots = assertions) raise SnapDiff::ExpectationNotMet.new(result.join("\n\n"), failed_assertions.first.caller) end + # The assertions that still owe the test a failure. This is a verdict + # (#status alone would read an unprocessed comparison as :passed); an + # assertion already judged by validate! is not failed twice. def failed_assertions - assertions.filter { |screenshot_assert| screenshot_assert.compare&.different? } + assertions.reject(&:validated?).select { |assertion| assertion.verdict == :changed } end def reset @assertions.clear - @new_screenshots.clear @checked_out_baselines.clear @screenshot_namer = SnapDiff::ScreenshotNamer.new end diff --git a/lib/snap_diff/screenshot_matcher.rb b/lib/snap_diff/screenshot_matcher.rb index 439631ef..879ded72 100644 --- a/lib/snap_diff/screenshot_matcher.rb +++ b/lib/snap_diff/screenshot_matcher.rb @@ -47,18 +47,10 @@ def build_screenshot_assertion(skip_stack_frames: 0) capture_screenshot(capture_options, comparison_options) - # AFTER the capture, so the path the message tells the user to - # `git add` is one that exists by the time they read it (#260). - fail_if_new_screenshot - # Pre-computation: No need to compare without base screenshot - # NOTE: Consider to return PreValid Assertion Value Object with hard coded valid result - unless need_to_compare? - record_uncompared_screenshot - return - end + return register_uncompared_screenshot unless need_to_compare? - create_screenshot_assertion(skip_stack_frames + 1, comparison_options) + register(create_screenshot_assertion(skip_stack_frames + 1, comparison_options)) end # Captures a screenshot without comparing it to a baseline. @@ -110,21 +102,31 @@ def need_to_compare? @snapshot.base_path.exist? end - # `:all` re-records, so nothing was compared and nothing is "new" in the - # missing-baseline sense. Reported through its own channel, which counts - # only the screenshots that really went down this path. + # The matcher is the one registrar: every assertion it builds is + # registered here, before any raise, so the end-of-run tally and the + # reporters see it however the DSL goes on to validate it. + def register(assertion) + SnapDiff.session.add_assertion(assertion) + end + + # No baseline to compare against: the capture is registered as an + # assertion with no comparison (`:new`, or `:rerecorded` under `:all`). # # Runs after a successful capture, so the file handed to reporters is - # really there (#260). A new one is tallied after the session's check, - # which raises when the baseline was committed but disappeared (#217): - # that one is not new. - def record_uncompared_screenshot + # really there, and the path the `:none` message tells the user to + # `git add` exists by the time they read it (#260). A new one is + # registered through the session's check, which raises when the baseline + # was committed but disappeared (#217): that one is not new. + def register_uncompared_screenshot if record_mode == :all SnapDiff::Reporting.record_rerecorded_baseline(screenshot_full_name, path: @snapshot.path) - else - SnapDiff.session.record_new_screenshot(screenshot_full_name) - warn_no_committed_baseline if SnapDiff::Reporting.record_missing_baseline(screenshot_full_name, path: @snapshot.path) + return register(SnapDiff::ScreenshotAssertion.new(screenshot_full_name, record_mode: record_mode, path: @snapshot.path)) end + + assertion = SnapDiff.session.record_new_screenshot(screenshot_full_name, path: @snapshot.path) + fail_if_new_screenshot + warn_no_committed_baseline if SnapDiff::Reporting.record_missing_baseline(screenshot_full_name, path: @snapshot.path) + assertion end def prepare_screenshot_options @@ -153,17 +155,11 @@ def discard_base_screenshot @snapshot.base_path.delete if @snapshot.base_path.exist? end - # Runs AFTER the capture, so `@snapshot.path` names a file that is - # really there and `git add` on it is a command the user can run on this - # very test run (#260). Before, the raise came first and the screenshot - # was never written -- the instruction was unfollowable in CI, the one - # place fail_if_new is on by default. + # The `:none` raise: immediate whatever `delayed` says, and after the + # capture, so `git add` on the path it names is a command the user can + # run on this very test run (#260). The screenshot is registered by then. def fail_if_new_screenshot - return if @snapshot.base_path.exist? return unless record_mode == :none - # Committed, then moved by a same-name test: not new. Left to the - # session, which raises that it disappeared (#217). - return if SnapDiff.session.baseline_checked_out?(screenshot_full_name) # Tallied here because it is the last moment it can be: this raise # ends the screenshot, and the run's summary and report still owe the @@ -199,7 +195,7 @@ def capture_screenshot(capture_options, comparison_options) end def create_screenshot_assertion(skip_stack_frames, comparison_options) - assertion = SnapDiff::ScreenshotAssertion.new(screenshot_full_name) + assertion = SnapDiff::ScreenshotAssertion.new(screenshot_full_name, record_mode: record_mode) assertion.caller = caller(skip_stack_frames + 1) assertion.compare = Comparison.new(@snapshot.path, @snapshot.base_path, comparison_options) assertion diff --git a/test/unit/baseline_disappeared_test.rb b/test/unit/baseline_disappeared_test.rb index 5031830e..1c418754 100644 --- a/test/unit/baseline_disappeared_test.rb +++ b/test/unit/baseline_disappeared_test.rb @@ -53,7 +53,7 @@ def stealing_screenshoter end end - assert_empty SnapDiff.session.new_screenshots, + assert_empty SnapDiff.session.assertions, "a vanished baseline must not be recorded as a never-committed one" assert_equal 0, SnapDiff::Reporting.missing_baselines_count, "nor counted as new, nor offered for `git add` in the report" @@ -92,7 +92,7 @@ def vanishing_checkout assert_match(/disappeared/i, error.message) refute_match(/No committed baseline/, err) assert_equal 0, SnapDiff::Reporting.missing_baselines_count - assert_empty SnapDiff.session.new_screenshots + assert_empty SnapDiff.session.assertions ensure SnapDiff.config.record = nil end @@ -105,7 +105,7 @@ def vanishing_checkout name = "a_#{Time.now.nsec}" SnapDiff::Vcs.stub(:checkout_vcs, false) do - assert_nil SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion + assert_equal :new, SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion.status end assert_includes SnapDiff.session.new_screenshots, name diff --git a/test/unit/diff_test.rb b/test/unit/diff_test.rb index b7152570..cb06aa1f 100644 --- a/test/unit/diff_test.rb +++ b/test/unit/diff_test.rb @@ -147,8 +147,10 @@ def _test_sample_screenshot_error mock = ::Minitest::Mock.new mock.expect(:different?, true) mock.expect(:different?, true) - # The third read is SnapDiff::Reporting.count's, when the test ends. + # A third `different?` is SnapDiff::Reporting.count's, when the test ends; + # `status` reads `difference` once after each of the two verdicts at verify. mock.expect(:different?, true) + 2.times { mock.expect(:difference, TestDoubles::TestDifference.new(true)) } mock.expect(:dimensions_changed?, false) mock.expect(:base_image_path, Pathname.new("screenshot.base.png")) mock.expect(:error_message, "expected error message") @@ -185,6 +187,8 @@ def _test_sample_screenshot_error comparison.expect(:different?, true) # to find backtrace comparison.expect(:different?, true) # to find messages comparison.expect(:different?, true) # SnapDiff::Reporting.count, when the test ends + # `status` reads `difference` once after each of the two verdicts at verify. + 2.times { comparison.expect(:difference, TestDoubles::TestDifference.new(true)) } comparison.expect(:dimensions_changed?, false) comparison.expect(:base_image_path, Pathname.new("screenshot.base.png")) comparison.expect(:error_message, "expected error message for non minitest") diff --git a/test/unit/dsl_test.rb b/test/unit/dsl_test.rb index 106d9b36..75f7697f 100644 --- a/test/unit/dsl_test.rb +++ b/test/unit/dsl_test.rb @@ -197,6 +197,26 @@ def self.screenshot(*, **) end end + test "#screenshot returns false for a new screenshot, which is registered as a :new assertion" do + SnapDiff::Vcs.stub(:checkout_vcs, false) do + assert_not screenshot("a") + + assert_equal [:new], SnapDiff.session.assertions.map(&:status) + end + end + + test "#screenshot with delayed: false still raises immediately for a new screenshot under record: :none, and registers it" do + SnapDiff.config.record = :none + + SnapDiff::Vcs.stub(:checkout_vcs, false) do + assert_raises(SnapDiff::ExpectationNotMet) { screenshot("a", delayed: false) } + + assert_equal [:new], SnapDiff.session.assertions.map(&:status) + end + ensure + SnapDiff.config.record = nil + end + test "SnapDiff.reset clears new_screenshots" do SnapDiff::Vcs.stub(:checkout_vcs, false) do screenshot "a" diff --git a/test/unit/record_modes_test.rb b/test/unit/record_modes_test.rb index 501fcdb7..cc3346de 100644 --- a/test/unit/record_modes_test.rb +++ b/test/unit/record_modes_test.rb @@ -85,7 +85,7 @@ def after_teardown capture_io do SnapDiff::Vcs.stub(:checkout_vcs, false) do - assert_nil SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion + assert_equal :new, SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion.status end end @@ -102,7 +102,7 @@ def after_teardown capture_io do SnapDiff::Vcs.stub(:checkout_vcs, false) do - assert_nil SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion + assert_equal :new, SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion.status end end @@ -154,6 +154,21 @@ def after_teardown assert_includes error.message, "SnapDiff.config.record = :once" end + # The raise is immediate, but the capture is still an assertion: the + # matcher registers it first, so reporters and the tally see it. + test ":none registers the :new assertion before it raises" do + SnapDiff.config.record = :none + name = "c_#{Time.now.nsec}" + + SnapDiff::Vcs.stub(:checkout_vcs, false) do + assert_raises(SnapDiff::ExpectationNotMet) do + SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion + end + end + + assert_equal [[name, :new]], SnapDiff.session.assertions.map { |a| [a.name, a.status] } + end + test ":none does not also warn about the missing baseline" do SnapDiff.config.record = :none @@ -259,11 +274,13 @@ def take_comparison_screenshot(_snapshot) SnapDiff::Vcs.stub(:checkout_vcs, true) do snap = create_snapshot_for(:a, :c) - assert_nil SnapDiff::ScreenshotMatcher.new(snap.full_name).build_screenshot_assertion + assertion = SnapDiff::ScreenshotMatcher.new(snap.full_name).build_screenshot_assertion assert_captured(:c, snap.path, "the re-recorded screenshot must be the new capture") assert_not_captured(:a, snap.path, "the old baseline must not survive a re-record") - assert_not_predicate SnapDiff.session, :assertions_present? + assert_nil assertion.compare + assert_equal :rerecorded, assertion.status + assert_equal [assertion], SnapDiff.session.assertions end end @@ -292,7 +309,7 @@ def take_comparison_screenshot(_snapshot) name = "c_#{Time.now.nsec}" SnapDiff::Vcs.stub(:checkout_vcs, false) do - assert_nil SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion + assert_equal :rerecorded, SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion.status end assert_predicate SnapDiff::SnapManager.path_for(name).path, :exist? @@ -353,7 +370,7 @@ def take_comparison_screenshot(_snapshot) with_ci("") do SnapDiff::Vcs.stub(:checkout_vcs, true) do snap = create_snapshot_for(:a, :c) - assert_nil SnapDiff::ScreenshotMatcher.new(snap.full_name).build_screenshot_assertion + assert_equal :rerecorded, SnapDiff::ScreenshotMatcher.new(snap.full_name).build_screenshot_assertion.status end end end @@ -380,7 +397,7 @@ def take_comparison_screenshot(_snapshot) capture_io do SnapDiff::Vcs.stub(:checkout_vcs, false) do - assert_nil SnapDiff::ScreenshotMatcher.new("c_#{Time.now.nsec}").build_screenshot_assertion + assert_equal :new, SnapDiff::ScreenshotMatcher.new("c_#{Time.now.nsec}").build_screenshot_assertion.status end end end @@ -392,7 +409,7 @@ def take_comparison_screenshot(_snapshot) SnapDiff.config.record = :none SnapDiff::Vcs.stub(:checkout_vcs, false) do - assert_nil SnapDiff::ScreenshotMatcher.new("c_#{Time.now.nsec}", record: :all).build_screenshot_assertion + assert_equal :rerecorded, SnapDiff::ScreenshotMatcher.new("c_#{Time.now.nsec}", record: :all).build_screenshot_assertion.status end end diff --git a/test/unit/registry_concurrency_test.rb b/test/unit/registry_concurrency_test.rb index 50bf5bb2..64e2468a 100644 --- a/test/unit/registry_concurrency_test.rb +++ b/test/unit/registry_concurrency_test.rb @@ -55,10 +55,12 @@ def build_assertion(name) names.each do |name| SnapDiff.session.add_assertion(build_assertion(name)) - SnapDiff.session.record_new_screenshot(name) observed << SnapDiff.session.assertions.map(&:name) end + new_names = names.map { |name| "#{name}_new" } + new_names.each { |name| SnapDiff.session.record_new_screenshot(name) } + SnapDiff.session.verify # all compares pass -> must not raise final_names = SnapDiff.session.assertions.map(&:name) @@ -67,7 +69,7 @@ def build_assertion(name) SnapDiff.reset after_reset = SnapDiff.session.assertions.size + SnapDiff.session.new_screenshots.size - {names: names, observed: observed, final_names: final_names, + {names: names, new_names: new_names, observed: observed, final_names: final_names, new_screenshots: new_screenshots, after_reset: after_reset} end end @@ -77,8 +79,8 @@ def build_assertion(name) result[:observed].each_with_index do |snapshot_of_names, index| assert_equal result[:names].first(index + 1), snapshot_of_names end - assert_equal result[:names], result[:final_names] - assert_equal result[:names], result[:new_screenshots] + assert_equal result[:names] + result[:new_names], result[:final_names] + assert_equal result[:new_names], result[:new_screenshots] assert_equal 0, result[:after_reset], "reset must clear only the calling thread's registry" end @@ -91,6 +93,8 @@ def build_assertion(name) failing_compare = Struct.new(:name) { def different? = true + def difference = TestDoubles::TestDifference.new(true) + def error_message = "boom" } diff --git a/test/unit/reporter_interplay_test.rb b/test/unit/reporter_interplay_test.rb index 55c5ea0f..9ac780c8 100644 --- a/test/unit/reporter_interplay_test.rb +++ b/test/unit/reporter_interplay_test.rb @@ -161,6 +161,17 @@ def summary end end + test "reporters receive a new screenshot as a :new assertion with no comparison" do + SnapDiff::Vcs.stub(:checkout_vcs, false) do + capture_io { screenshot("a") } + SnapDiff.reset + end + + received = @spy.recorded.flatten + assert_equal [["a", :new]], received.map { |a| [a.name, a.status] } + assert_nil received.first.compare + end + test "reset without assertions does not notify reporters" do SnapDiff.reset diff --git a/test/unit/reporters/ai_simple_test.rb b/test/unit/reporters/ai_simple_test.rb index 49751ab1..e1e63fa4 100644 --- a/test/unit/reporters/ai_simple_test.rb +++ b/test/unit/reporters/ai_simple_test.rb @@ -17,7 +17,6 @@ def new_image_path = Pathname("/nonexistent/current.png") def to_h = {area_size: 42, region: [0, 0, 10, 10]} end StubCompare = Struct.new(:difference) - StubAssertion = Struct.new(:name, :compare) # Minimal ScreenshotAssertion surface for gate tests. GateCompare = Struct.new(:difference) do @@ -44,7 +43,9 @@ def gate_assertion(name, different: true) end def build_assertion(name, different: true) - StubAssertion.new(name, StubCompare.new(StubDifference.new(different: different))) + SnapDiff::ScreenshotAssertion.new(name).tap do |a| + a.compare = StubCompare.new(StubDifference.new(different: different)) + end end def similarity_backend(value) @@ -59,7 +60,8 @@ def test_records_only_different_assertions build_reporter(similarity_backend(0.5)).record([ build_assertion("changed"), build_assertion("same", different: false), - StubAssertion.new("pending", StubCompare.new(nil)) + SnapDiff::ScreenshotAssertion.new("pending").tap { |a| a.compare = StubCompare.new(nil) }, + SnapDiff::ScreenshotAssertion.new("brand_new") ]) assert_equal ["changed"], SnapDiff::AI.results.map { |r| r[:name] } diff --git a/test/unit/screenshot_assertion_status_test.rb b/test/unit/screenshot_assertion_status_test.rb new file mode 100644 index 00000000..481a2c7b --- /dev/null +++ b/test/unit/screenshot_assertion_status_test.rb @@ -0,0 +1,108 @@ +# frozen_string_literal: true + +require "test_helper" +require "snap_diff" + +module SnapDiff + # `status` is the one answer to "what became of this capture?". It is + # derived on every call (a comparison can be assigned after the fact) and + # must never be the thing that triggers the comparison. + class ScreenshotAssertionStatusTest < ActiveSupport::TestCase + include DSLStub + + test "#status is :passed when the comparison matches" do + assert_equal :passed, build(compare: make_comparison(:a, :a).tap(&:processed)).status + end + + test "#status is :changed when the comparison differs" do + assert_equal :changed, build(compare: make_comparison(:a, :b).tap(&:processed)).status + end + + test "#status is :new without a comparison" do + assert_equal :new, build.status + end + + test "#status is :rerecorded without a comparison under record: :all" do + assert_equal :rerecorded, build(record_mode: :all).status + end + + test "#status is :rerecorded for a differing comparison under record: :all" do + assert_equal :rerecorded, build(compare: make_comparison(:a, :b).tap(&:processed), record_mode: :all).status + end + + test "#status reads :passed for an unprocessed comparison without processing it" do + comparison = make_comparison(:a, :b) + + assert_equal :passed, build(compare: comparison).status + assert_not comparison.processed?, "reading status must not force the comparison" + end + + # #verdict is the one place that forces the comparison; #status stays a + # pure read. Every caller that needs a judgment goes through it. + test "#verdict judges an unprocessed comparison where #status would read :passed" do + comparison = make_comparison(:a, :b) + assertion = build(compare: comparison) + + assert_equal :changed, assertion.verdict + assert_predicate comparison, :processed? + assert_equal :new, build.verdict + assert_equal :rerecorded, build(compare: make_comparison(:a, :b, destination: "two"), record_mode: :all).verdict + end + + test "#status is not memoized across #compare=" do + assertion = build + + assert_equal :new, assertion.status + + assertion.compare = make_comparison(:a, :b).tap(&:processed) + + assert_equal :changed, assertion.status + end + + test "#path falls back to the comparison's image path" do + comparison = make_comparison(:a, :a) + + assert_equal comparison.image_path, build(compare: comparison).path + assert_equal Pathname("x.png"), build(path: Pathname("x.png")).path + end + + test "AssertionRegistry#failed_assertions lists :changed assertions that were not already validated" do + registry = AssertionRegistry.new + changed = build(compare: make_comparison(:a, :b, destination: "one").tap(&:processed)) + raised = build(compare: make_comparison(:a, :b, destination: "two").tap(&:processed)) + passed = build(compare: make_comparison(:a, :a, destination: "three").tap(&:processed)) + [changed, raised, passed, build].each { |assertion| registry.add_assertion(assertion) } + raised.caller = ["my_test.rb:1"] + assert_raises(SnapDiff::ExpectationNotMet) { raised.validate! } + + assert_equal [changed], registry.failed_assertions + end + + # Callers ask it for a verdict (the integration suite and the delayed + # `verify` path), so it has to judge comparisons nobody has judged yet. + # #status alone would read an unprocessed one as :passed. + test "AssertionRegistry#failed_assertions judges comparisons that were not processed yet" do + registry = AssertionRegistry.new + unjudged = build(compare: make_comparison(:a, :b, destination: "one")) + rerecorded = build(compare: make_comparison(:a, :b, destination: "two"), record_mode: :all) + [unjudged, rerecorded].each { |assertion| registry.add_assertion(assertion) } + + assert_equal [unjudged], registry.failed_assertions + end + + test "AssertionRegistry#record_new_screenshot registers a :new assertion and #add_assertion still ignores nil" do + registry = AssertionRegistry.new + registry.record_new_screenshot("shot") + + assert_nil registry.add_assertion(nil) + assert_equal [["shot", :new]], registry.assertions.map { |a| [a.name, a.status] } + assert_equal ["shot"], registry.new_screenshots + end + + private + + def build(compare: nil, **args) + ScreenshotAssertion.new("name", **args).tap { |assertion| assertion.compare = compare } + end + end +end diff --git a/test/unit/screenshot_matcher_test.rb b/test/unit/screenshot_matcher_test.rb index 8bd903dc..fbaadcd8 100644 --- a/test/unit/screenshot_matcher_test.rb +++ b/test/unit/screenshot_matcher_test.rb @@ -27,16 +27,19 @@ def take_comparison_screenshot(snapshot) end end - # D4: dual return shape — nil for the new-screenshot path, after - # side-effecting record_new_screenshot into the registry. - test "#build_screenshot_assertion returns nil and records a new screenshot when no baseline exists" do + # D4: the new-screenshot path returns a registered, compare-less :new + # assertion rather than nil. + test "#build_screenshot_assertion returns a registered :new assertion when no baseline exists" do # ScreenshoterStub resolves "c_" to the c.png fixture. name = "c_#{Time.now.nsec}" SnapDiff::Vcs.stub(:checkout_vcs, false) do assertion = SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion - assert_nil assertion + assert_equal :new, assertion.status + assert_nil assertion.compare + assert_equal SnapDiff::SnapManager.path_for(name).path, assertion.path + assert_equal [assertion], SnapDiff.session.assertions assert_includes SnapDiff.session.new_screenshots, name assert_predicate SnapDiff::SnapManager.path_for(name).path, :exist?, "the screenshot is still captured on the new-screenshot path" @@ -44,7 +47,7 @@ def take_comparison_screenshot(snapshot) end # D4: the other return shape — a fully wired ScreenshotAssertion. - test "#build_screenshot_assertion returns an assertion with compare and caller when a baseline exists" do + test "#build_screenshot_assertion returns a registered assertion with compare and caller when a baseline exists" do SnapDiff::Vcs.stub(:checkout_vcs, true) do snap = create_snapshot_for(:a, :c) @@ -56,6 +59,7 @@ def take_comparison_screenshot(snapshot) assert_match(/screenshot_matcher_test\.rb/, assertion.caller.first) assert_equal snap.path, assertion.compare.image_path assert_equal snap.base_path, assertion.compare.base_image_path + assert_equal [assertion], SnapDiff.session.assertions end end From 476a4e0cfefe6dc9c55566d8e1f0f17766678722 Mon Sep 17 00:00:00 2001 From: Diego Algorta Date: Fri, 16 Oct 2026 17:17:25 -0300 Subject: [PATCH 2/3] refactor: Reporting tallies every outcome from status Why: a run's outcome travelled in two channels with two tally sites. Reporting.count tallied compared captures from their comparison, while the matcher tallied new and re-recorded captures on the side through record_missing_baseline / record_rerecorded_baseline (a new one in two matcher methods). Since every capture is a registered assertion with a derived status, none of that needs a second channel. Mechanism: - Reporting owns one store: @counts (assertions per status, :new excluded) and @outcomes (a frozen Outcome row per name for :new and :rerecorded captures, first sighting wins). Reporting.count, called from notify at reset, is the only tally site. It reads each assertion's verdict before taking the mutex: the verdict forces a comparison nobody has run yet, and an assertion whose comparison raises is warned about by name and skipped on its own. store_outcome only stores; the "new" count is derived from the stored rows, so :new is counted once per name and :passed / :changed / :rerecorded once per assertion, with no per-status special case at the callers. - The row type, which only ever held uncompared captures, is now called Outcome (name, path, status): it also lists re-recorded ones, and under `record: :all` compared ones. `Reporting.uncompared` keeps its name and returns them (new first, then re-recorded). The docs that named the row's old `reason` say `status`. - verified / changed / missing_baselines_count / the summary lines read the store. - record_missing_baseline(name, path:) and record_rerecorded_baseline stay callable as thin shims writing into the same store, and still return true on a name's first sighting. (record_rerecorded_baseline meant "re-recorded, not verified" in beta4; its comment says that it now counts as verified and changed, once `:all` compares. Only the matcher ever called it, and no longer does.) - The matcher no longer tallies. "Warn once per screenshot" for a missing committed baseline now keys on the session's :new assertions instead of the process-wide tally, so it is once per name per test rather than once per name per run. The "checked out then disappeared" guard stays only in AssertionRegistry#record_new_screenshot, which the matcher calls. - Parallel fragment: a worker writes its "counts" hash plus the listed rows ("outcomes"); the parent sums the counts and merges the rows. The name lists and the verified / changed arithmetic are gone, and so are the readers for them. Fragments are only ever written by forks of the process that merges them, so they always carry this version's shape -- and install_parallel_hooks!, which runs in the parent before any fork, now empties the fragments directory, which is what makes that true: a run killed before its merge no longer leaves fragments that a recycled pid would hand to a later run (counts from another run, or a KeyError in after_run that cost the whole summary). A key that is missing (KeyError) or a status nobody writes (ArgumentError) is a bug and raises instead of being half-understood. - The HTML reporter is untouched: it keeps counting the compared assertions it records and handing that total to the parent in its own state, so it still works on its own (record + finalize, no Reporting.count behind it). Proof of neutrality: the same scenario (compared pass / change / dimension change, a new and a re-recorded capture, through notify and finalize!) printed the same four summary lines and wrote a byte-identical HTML report before and after, and bin/rake report:sample wrote a byte-identical report. bin/rake test:unit: 839 runs, 0 failures. Not changed: summary wording, :all semantics (still discards the baseline and compares nothing), the report template and entry shape, public method signatures. Within one run a name that is both new and re-recorded now has one row instead of one per reason. --- CHANGELOG.md | 2 +- docs/reporters.md | 2 +- docs/snapdiff.md | 4 +- lib/snap_diff/reporters/html.rb | 2 +- lib/snap_diff/reporting.rb | 203 +++++++++++----------- lib/snap_diff/screenshot_matcher.rb | 11 +- test/unit/diff_test.rb | 11 +- test/unit/parallel_report_merge_test.rb | 123 +++++++++---- test/unit/record_modes_test.rb | 10 +- test/unit/reporter_interplay_test.rb | 2 + test/unit/reporters/html_reporter_test.rb | 13 +- test/unit/reporting_counts_test.rb | 63 ++++--- 12 files changed, 272 insertions(+), 174 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3711e2dd..456ea205 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -333,7 +333,7 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV - **Reporters can list the screenshots that were not compared.** `SnapDiff::Reporting.uncompared` returns the ones behind the summary line's "new" and "re-recorded" counts, complete by the time reporters `finalize` (fork-parallel workers - included): each with its `name`, the captured file's `path`, and the `reason` — `:new` + included): each with its `name`, the captured file's `path`, and the `status` — `:new` (no committed baseline) or `:rerecorded` (`record: :all`). The bundled HTML report lists new screenshots from it - New documentation shipped inside the gem: diff --git a/docs/reporters.md b/docs/reporters.md index 137fb468..d55ac41e 100644 --- a/docs/reporters.md +++ b/docs/reporters.md @@ -136,7 +136,7 @@ baseline) or `:rerecorded` (accepted by `record: :all`). `compare` is nil for a without a baseline; `assertion.path` is the captured file either way. Test `status`, not `compare`, to find failures: the tally has already run every comparison by the time `record` is called, and reading `status` never forces one. The new and re-recorded screenshots are also in -`SnapDiff::Reporting.uncompared` (each with `name`, `path` and `reason`), complete by the time +`SnapDiff::Reporting.uncompared` (each with `name`, `path` and `status`), complete by the time `finalize` runs (fork-parallel workers included). The HTML report lists new screenshots from it. Full details in [Custom reporters](snapdiff.md#custom-reporters). diff --git a/docs/snapdiff.md b/docs/snapdiff.md index 3a76fb1a..ecb7e742 100644 --- a/docs/snapdiff.md +++ b/docs/snapdiff.md @@ -269,9 +269,9 @@ included): ```ruby def finalize - new_ones = SnapDiff::Reporting.uncompared.select { |s| s.reason == :new } + new_ones = SnapDiff::Reporting.uncompared.select { |s| s.status == :new } # each has #name, #path (the captured file as a Pathname, or nil when it isn't known) - # and #reason (:new or :rerecorded); check s.path&.exist? before reading it + # and #status (:new or :rerecorded); check s.path&.exist? before reading it end ``` diff --git a/lib/snap_diff/reporters/html.rb b/lib/snap_diff/reporters/html.rb index c1e5822f..a893e329 100644 --- a/lib/snap_diff/reporters/html.rb +++ b/lib/snap_diff/reporters/html.rb @@ -92,7 +92,7 @@ def passed = total - failures.size def failed = failures.size # Screenshots captured but not compared: new ones are listed, # re-recorded ones only counted. - def new_screenshots = SnapDiff::Reporting.uncompared.select { |screenshot| screenshot.reason == :new } + def new_screenshots = SnapDiff::Reporting.uncompared.select { |screenshot| screenshot.status == :new } def new_count = new_screenshots.size def rerecorded_count = SnapDiff::Reporting.uncompared.size - new_count diff --git a/lib/snap_diff/reporting.rb b/lib/snap_diff/reporting.rb index 93539cca..0a4ed58c 100644 --- a/lib/snap_diff/reporting.rb +++ b/lib/snap_diff/reporting.rb @@ -16,22 +16,28 @@ module SnapDiff # reporters/reporters_mutex/finalize_reporters! methods as thin shims # over this module. module Reporting - # A screenshot that was captured but not compared, as listed by - # {uncompared}: its `name`, the captured file's `path` (a Pathname; nil - # when a worker of an older version sent only the name), and the - # `reason` -- `:new` (no committed baseline) or `:rerecorded` - # (`record: :all`). Frozen: every reporter reads the same ones. - Uncompared = Struct.new(:name, :path, :reason, keyword_init: true) + # A screenshot the run lists by name instead of only counting, as + # returned by {uncompared}: its `name`, the captured file's `path` (a + # Pathname, nil when unknown) and its `status` -- the assertion's: + # `:new` (no committed baseline) or `:rerecorded` (`record: :all`). + # Frozen: every reporter reads the same ones. + Outcome = Struct.new(:name, :path, :status, keyword_init: true) + + # The statuses that get an {Outcome} row. A fork-parallel fragment + # carrying any other is a bug and fails loudly. + LISTED_STATUSES = %i[new rerecorded].freeze + private_constant :LISTED_STATUSES @reporters = [] @mutex = Mutex.new - # name => Uncompared, one tally per reason. - @missing_baselines = {} - @rerecorded_baselines = {} + # THE store: the one place a run's outcomes are tallied (see {count}). + # `@counts` is how many assertions ended in each status (:new is not + # kept there); `@outcomes` keeps a row per listed screenshot, by name, + # first sighting wins, and the "new" count is derived from those rows. + @counts = Hash.new(0) + @outcomes = {} @matched_selectors = Set.new @unmatched_selectors = Set.new - @verified = 0 - @changed = 0 @stable_captures = 0 @worst_settle_seconds = 0.0 @worst_settle_attempts = 0 @@ -50,20 +56,21 @@ class << self # signal for either. It shipped inside Reporters::HTML, the gem's one # and only `register` call site, so the documented Rails setup (which # requires just the Minitest integration) printed nothing at all. - attr_reader :verified, :changed + def verified = @counts[:passed] + @counts[:changed] + + def changed = @counts[:changed] # Remembers a screenshot that had no COMMITTED baseline and was # therefore never compared. # - # @return [Boolean] true the first time this name is seen -- the - # "warn once per screenshot" gate for ScreenshotMatcher, and the - # tally behind {finalize!}'s summary line. Kept here rather than in - # the matcher because the end-of-run summary is this module's job. + # Shipped API, kept as a shim over the status store: the matcher no + # longer calls it, {count} tallies every `:new` assertion itself. + # + # `path` is the file it was captured to, listed by {uncompared}. # - # `path` is the file it was captured to, listed by {uncompared}: pass - # it once that file exists. + # @return [Boolean] true the first time this name is seen def record_missing_baseline(name, path: nil) - tally_uncompared(@missing_baselines, name, path, :new) + @mutex.synchronize { store_outcome(name, path, :new) } end # How many screenshots were captured but never compared. The "new" @@ -71,7 +78,7 @@ def record_missing_baseline(name, path: nil) # number of screenshots that actually went down the no-baseline path # this run, not a number derived from what the run should have done. def missing_baselines_count - @mutex.synchronize { @missing_baselines.size } + @mutex.synchronize { outcome_count(:new) } end # The screenshots captured but not compared this run -- new ones @@ -80,9 +87,9 @@ def missing_baselines_count # reporters `finalize`, fork-parallel workers included, so a reporter # that lists them lists what the terminal counts. # - # @return [Array] frozen + # @return [Array] frozen def uncompared - @mutex.synchronize { (@missing_baselines.values + @rerecorded_baselines.values).freeze } + @mutex.synchronize { @outcomes.each_value.partition { |row| row.status == :new }.flatten(1).freeze } end # Remembers whether a CSS selector (`skip_area`, `crop`) found @@ -119,43 +126,48 @@ def record_stable_capture(seconds, attempts) # tally, so a tally added later cannot be forgotten at the call site. def reset_run_totals! @mutex.synchronize do - @missing_baselines.clear - @rerecorded_baselines.clear + @counts.clear + @outcomes.clear @matched_selectors.clear @unmatched_selectors.clear - @verified = 0 - @changed = 0 @stable_captures = 0 @worst_settle_seconds = 0.0 @worst_settle_attempts = 0 end end - # Remembers a screenshot re-recorded by `record: :all` -- captured as - # the new baseline with nothing compared against it. A separate tally - # from {record_missing_baseline} on purpose: "there was no baseline" - # and "there was one and we accepted the new rendering over it" are - # different facts, and the summary must not claim the first when the - # second happened. + # Shipped API, kept as a shim over the status store (see + # {record_missing_baseline}): a screenshot re-recorded by `record: + # :all`. In 2.0.0.beta4 this meant "re-recorded, NOT verified"; since + # `:all` compares, a re-recorded screenshot was verified against its + # baseline and differed, so it now counts as verified and changed (only + # the matcher ever called it, and it no longer does). + # + # A separate status from `:new` on purpose: "there was no + # baseline" and "there was one and we accepted the new rendering over + # it" are different facts, and the summary must not claim the first + # when the second happened. + # Counts once per name, as it always did. + # @return [Boolean] true the first time `name` is seen def record_rerecorded_baseline(name, path: nil) - tally_uncompared(@rerecorded_baselines, name, path, :rerecorded) - end - - # @return [Boolean] true the first time `name` is seen in `tally` - def tally_uncompared(tally, name, path, reason) @mutex.synchronize do - next false if tally.key?(name) - - tally[name] = uncompared_entry(name, path, reason) - true + store_outcome(name, path, :rerecorded).tap { |stored| @counts[:rerecorded] += 1 if stored } end end - private :tally_uncompared - def uncompared_entry(name, path, reason) - Uncompared.new(name: -name.to_s, path: path && Pathname(path).freeze, reason: reason.to_sym).freeze + # Under the mutex. Only stores: counting is {count}'s job. + # @return [Boolean] true when `name` had no row yet + def store_outcome(name, path, status) + return false if @outcomes.key?(name) + + @outcomes[name] = Outcome.new(name: -name.to_s, path: path && Pathname(path).freeze, status: status.to_sym).freeze + true end - private :uncompared_entry + private :store_outcome + + # Under the mutex. + def outcome_count(status) = @outcomes.each_value.count { |row| row.status == status } + private :outcome_count # Registers a reporter for the rest of the process. The canonical way # in: the append happens under the mutex, so concurrent registrations @@ -196,32 +208,26 @@ def notify(assertions) end end - # Tallies a finished test's assertions. An assertion with no - # `compare` never reached a baseline, so it is neither verified nor - # changed -- it is counted, if at all, by {record_missing_baseline}. - # A comparison that has not run yet (nothing validated it: teardown - # skips that when fail_on_difference is off) runs here, so a verified - # screenshot is always one that was actually compared. One that cannot - # run is warned about by name and left out of the tally; the other - # screenshots of the same test still count. + # THE tally: counts a finished test's assertions by status, and the + # only place that does. The verdict is read before taking the mutex: + # it runs a comparison nobody has forced yet (teardown skips that when + # fail_on_difference is off), so a verified screenshot is always one + # that was actually compared. One that cannot run is warned about by + # name and left out of the tally; the other screenshots of the same + # test still count. def count(assertions) - verified = 0 - changed = 0 - - assertions.each do |assertion| - compare = assertion.compare - next unless compare - - different = compare.different? - verified += 1 - changed += 1 if different + verdicts = assertions.filter_map do |assertion| + [assertion, assertion.verdict] rescue => e warn "[snap_diff] Could not tally '#{assertion.name}' (#{e.class}: #{e.message})" + nil end @mutex.synchronize do - @verified += verified - @changed += changed + verdicts.each do |assertion, status| + @counts[status] += 1 unless status == :new + store_outcome(assertion.name, assertion.path, status) if LISTED_STATUSES.include?(status) + end end end @@ -241,7 +247,7 @@ def count(assertions) # case worth shouting about. def counts_summary verified, changed, new_count, rerecorded = @mutex.synchronize { - [@verified, @changed, @missing_baselines.size, @rerecorded_baselines.size] + [@counts[:passed] + @counts[:changed], @counts[:changed], outcome_count(:new), @counts[:rerecorded]] } line = "[snap_diff] #{verified} verified, #{changed} changed, #{new_count} new (not verified)." @@ -322,6 +328,9 @@ def install_parallel_hooks! return false unless defined?(::ActiveSupport::Testing::Parallelization) @parallel_owner_pid = Process.pid + # A run killed before its merge leaves its fragments behind, and a + # recycled pid would hand them to this one. Nothing has forked yet. + FileUtils.rm_rf(parallel_fragments_dir) ::ActiveSupport::Testing::Parallelization.run_cleanup_hook { dump_parallel_fragment } true end @@ -341,18 +350,16 @@ def parallel_fragments_dir # a worker killed mid-write leaves nothing the merge will read. def dump_parallel_fragment payload = { - "missing_baselines" => @mutex.synchronize { @missing_baselines.keys }, - "rerecorded_baselines" => @mutex.synchronize { @rerecorded_baselines.keys }, - # The same entries with their paths, for the parent's {uncompared}. - # The name lists above stay: an older parent reads only those. - "uncompared" => uncompared.map { |entry| {"name" => entry.name, "path" => entry.path&.to_s, "reason" => entry.reason.to_s} }, + # The rows with their paths, `:new` and `:rerecorded`, and the + # per-status counts: the parent sums the counts and merges the + # rows (which is also where its "new" count comes from). + "counts" => @mutex.synchronize { @counts.dup }, + "outcomes" => uncompared.map { |entry| {"name" => entry.name, "path" => entry.path&.to_s, "status" => entry.status.to_s} }, # Both halves, not the subtraction: `img` may match in this worker # and miss in the next, and only the parent that merged every # fragment can tell whether it matched anywhere in the run. "matched_selectors" => @mutex.synchronize { @matched_selectors.to_a }, "unmatched_selectors" => @mutex.synchronize { @unmatched_selectors.to_a }, - "verified" => @verified, - "changed" => @changed, "stable_captures" => @stable_captures, "worst_settle_seconds" => @worst_settle_seconds, "worst_settle_attempts" => @worst_settle_attempts, @@ -378,31 +385,26 @@ def merge_parallel_fragments! Dir[File.join(parallel_fragments_dir, "*.json")].sort.each do |fragment| payload = JSON.parse(File.read(fragment)) - # Only "missing_baselines" is read without a default: it is the - # one key every version of this fragment has ever written. Every - # key added since is `fetch`ed with one, because the fragments - # directory is keyed by pid under the system temp dir -- a - # recycled pid can hand this merge a fragment left behind by an - # older version of the gem. Only such missing keys are tolerated: - # an entry no version writes is a bug and raises. + # Fragments are written by forks of this very process, so they + # always carry this version's shape: a key that is missing, or a + # status nobody writes, is a bug and raises. What makes that true + # is {install_parallel_hooks!}, which empties the directory before + # any fork exists, so no fragment of an earlier run is ever here. @mutex.synchronize do - # Entries with paths first; a name only an older worker sent is - # still listed, with no capture to show. - payload.fetch("uncompared", []).each do |entry| - tally = {"new" => @missing_baselines, "rerecorded" => @rerecorded_baselines}.fetch(entry["reason"]) - tally[entry["name"]] ||= uncompared_entry(entry["name"], entry["path"], entry["reason"]) + payload.fetch("outcomes").each do |entry| + status = entry.fetch("status").to_sym + raise ArgumentError, "unknown outcome status in fragment: #{status.inspect}" unless LISTED_STATUSES.include?(status) + + store_outcome(entry.fetch("name"), entry.fetch("path"), status) end - payload["missing_baselines"].each { |name| @missing_baselines[name] ||= uncompared_entry(name, nil, :new) } - payload.fetch("rerecorded_baselines", []).each { |name| @rerecorded_baselines[name] ||= uncompared_entry(name, nil, :rerecorded) } - payload.fetch("matched_selectors", []).each { |selector| @matched_selectors << selector } - payload.fetch("unmatched_selectors", []).each { |selector| @unmatched_selectors << selector } - @verified += payload.fetch("verified", 0) - @changed += payload.fetch("changed", 0) + payload.fetch("counts").each { |status, count| @counts[status.to_sym] += count } + payload.fetch("matched_selectors").each { |selector| @matched_selectors << selector } + payload.fetch("unmatched_selectors").each { |selector| @unmatched_selectors << selector } # Counts add up; worst cases do not -- the slowest page in the # run is the slowest page in whichever worker happened to run it. - @stable_captures += payload.fetch("stable_captures", 0) - @worst_settle_seconds = [@worst_settle_seconds, payload.fetch("worst_settle_seconds", 0.0)].max - @worst_settle_attempts = [@worst_settle_attempts, payload.fetch("worst_settle_attempts", 0)].max + @stable_captures += payload.fetch("stable_captures") + @worst_settle_seconds = [@worst_settle_seconds, payload.fetch("worst_settle_seconds")].max + @worst_settle_attempts = [@worst_settle_attempts, payload.fetch("worst_settle_attempts")].max end reporters_snapshot = @mutex.synchronize { @reporters.dup } @@ -415,6 +417,11 @@ def merge_parallel_fragments! FileUtils.rm_rf(parallel_fragments_dir) end + def outcome_names(status) + @mutex.synchronize { @outcomes.select { |_name, row| row.status == status }.keys } + end + private :outcome_names + # The reporters' summary line carries the COUNT of screenshots that # were captured without a committed baseline ("N new (not # verified)"); this line names them, so the next thing the reader @@ -422,11 +429,11 @@ def merge_parallel_fragments! # # @return [String, nil] nil when every screenshot had a baseline def missing_baselines_summary - names = @mutex.synchronize { @missing_baselines.keys } + names = outcome_names(:new) return if names.empty? - label = (names.size == 1) ? "1 screenshot" : "#{names.size} screenshots" - "[snap_diff] #{label} had no committed baseline and #{(names.size == 1) ? "was" : "were"} NOT compared: " \ + label, verb = names.one? ? ["1 screenshot", "was"] : ["#{names.size} screenshots", "were"] + "[snap_diff] #{label} had no committed baseline and #{verb} NOT compared: " \ "#{names.join(", ")}. Commit the captured file(s) to enable comparison." end @@ -438,7 +445,7 @@ def missing_baselines_summary # # @return [String, nil] nil when nothing was re-recorded def rerecorded_baselines_summary - names = @mutex.synchronize { @rerecorded_baselines.keys } + names = outcome_names(:rerecorded) return if names.empty? label = (names.size == 1) ? "1 screenshot" : "#{names.size} screenshots" diff --git a/lib/snap_diff/screenshot_matcher.rb b/lib/snap_diff/screenshot_matcher.rb index 879ded72..135b363d 100644 --- a/lib/snap_diff/screenshot_matcher.rb +++ b/lib/snap_diff/screenshot_matcher.rb @@ -116,16 +116,17 @@ def register(assertion) # really there, and the path the `:none` message tells the user to # `git add` exists by the time they read it (#260). A new one is # registered through the session's check, which raises when the baseline - # was committed but disappeared (#217): that one is not new. + # was committed but disappeared (#217): that one is not new. The tally is + # not done here: Reporting counts every registered assertion at reset. def register_uncompared_screenshot if record_mode == :all - SnapDiff::Reporting.record_rerecorded_baseline(screenshot_full_name, path: @snapshot.path) return register(SnapDiff::ScreenshotAssertion.new(screenshot_full_name, record_mode: record_mode, path: @snapshot.path)) end + first_sighting = !SnapDiff.session.new_screenshots.include?(screenshot_full_name) assertion = SnapDiff.session.record_new_screenshot(screenshot_full_name, path: @snapshot.path) fail_if_new_screenshot - warn_no_committed_baseline if SnapDiff::Reporting.record_missing_baseline(screenshot_full_name, path: @snapshot.path) + warn_no_committed_baseline if first_sighting assertion end @@ -161,10 +162,6 @@ def discard_base_screenshot def fail_if_new_screenshot return unless record_mode == :none - # Tallied here because it is the last moment it can be: this raise - # ends the screenshot, and the run's summary and report still owe the - # user the capture it names. - SnapDiff::Reporting.record_missing_baseline(screenshot_full_name, path: @snapshot.path) raise SnapDiff::ExpectationNotMet.new(<<~ERROR.chomp, caller) No existing screenshot found for #{@snapshot.path}! To record it: `git add #{@snapshot.path}` and commit -- baselines are read from git. diff --git a/test/unit/diff_test.rb b/test/unit/diff_test.rb index cb06aa1f..3f5443c1 100644 --- a/test/unit/diff_test.rb +++ b/test/unit/diff_test.rb @@ -147,10 +147,10 @@ def _test_sample_screenshot_error mock = ::Minitest::Mock.new mock.expect(:different?, true) mock.expect(:different?, true) - # A third `different?` is SnapDiff::Reporting.count's, when the test ends; - # `status` reads `difference` once after each of the two verdicts at verify. + # Read by `status` twice at verify (the failed_assertions verdict, + # then validate) and by SnapDiff::Reporting.count when the test ends. mock.expect(:different?, true) - 2.times { mock.expect(:difference, TestDoubles::TestDifference.new(true)) } + 3.times { mock.expect(:difference, TestDoubles::TestDifference.new(true)) } mock.expect(:dimensions_changed?, false) mock.expect(:base_image_path, Pathname.new("screenshot.base.png")) mock.expect(:error_message, "expected error message") @@ -187,8 +187,9 @@ def _test_sample_screenshot_error comparison.expect(:different?, true) # to find backtrace comparison.expect(:different?, true) # to find messages comparison.expect(:different?, true) # SnapDiff::Reporting.count, when the test ends - # `status` reads `difference` once after each of the two verdicts at verify. - 2.times { comparison.expect(:difference, TestDoubles::TestDifference.new(true)) } + # Read by `status` twice at verify (the failed_assertions verdict, + # then validate) and by SnapDiff::Reporting.count when the test ends. + 3.times { comparison.expect(:difference, TestDoubles::TestDifference.new(true)) } comparison.expect(:dimensions_changed?, false) comparison.expect(:base_image_path, Pathname.new("screenshot.base.png")) comparison.expect(:error_message, "expected error message for non minitest") diff --git a/test/unit/parallel_report_merge_test.rb b/test/unit/parallel_report_merge_test.rb index b0f3f46b..5f910630 100644 --- a/test/unit/parallel_report_merge_test.rb +++ b/test/unit/parallel_report_merge_test.rb @@ -45,12 +45,13 @@ class ParallelReportMergeTest < ActiveSupport::TestCase test "records and counts from forked workers reach the parent" do fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("worker_a")]) } fork_worker do - SnapDiff::Reporting.notify([build_failing_assertion("worker_b"), build_passing_assertion("worker_b_ok")]) - SnapDiff::Reporting.record_missing_baseline("worker_b_new") - # Rides the same fragment as the counts above (#274 + #269). Both - # tallies are written by the same worker and merged by the same - # parent, and neither may cost the other. - SnapDiff::Reporting.record_rerecorded_baseline("worker_b_accepted") + # Compared, new and re-recorded captures ride the same fragment + # (#274 + #269): written by the same worker, merged by the same + # parent, and none may cost the others. + SnapDiff::Reporting.notify([ + build_failing_assertion("worker_b"), build_passing_assertion("worker_b_ok"), + build_uncompared_assertion("worker_b_new", :new), build_uncompared_assertion("worker_b_accepted", :rerecorded) + ]) end # The bug, restated as an assertion: the parent holds nothing of its own. @@ -66,6 +67,8 @@ class ParallelReportMergeTest < ActiveSupport::TestCase assert_equal "[snap_diff] 3 verified, 2 changed, 1 new (not verified). 1 re-recorded (not verified).", SnapDiff::Reporting.counts_summary assert_includes SnapDiff::Reporting.rerecorded_baselines_summary, "worker_b_accepted" + assert_equal [["worker_b_new", :new], ["worker_b_accepted", :rerecorded]], + SnapDiff::Reporting.uncompared.map { |row| [row.name, row.status] } # Symbol keys, like an entry recorded in this process: `failures` is # public, and an array whose shape depends on which process filled it @@ -86,31 +89,31 @@ class ParallelReportMergeTest < ActiveSupport::TestCase fork_worker do capture.dirname.mkpath FileUtils.cp(File.expand_path("a.png", TEST_IMAGES_DIR), capture) - SnapDiff::Reporting.record_missing_baseline("signup", path: capture) + SnapDiff::Reporting.notify([build_uncompared_assertion("signup", :new, path: capture)]) end SnapDiff::Reporting.merge_parallel_fragments! assert_includes SnapDiff::Reporting.counts_summary, "1 new (not verified)" - assert_equal [{name: "signup", path: capture, reason: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) + assert_equal [{name: "signup", path: capture, status: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) assert_predicate @reporter.finalize, :present?, "a run with only new screenshots writes a report" assert_includes @output_path.read, %("name":"signup","new_screenshot":true) assert_includes @output_path.read, "git add -- " end test "a re-recorded screenshot captured in a worker reaches the parent with its path" do - fork_worker { SnapDiff::Reporting.record_rerecorded_baseline("accepted", path: "/app/accepted.png") } + fork_worker { SnapDiff::Reporting.notify([build_uncompared_assertion("accepted", :rerecorded, path: "/app/accepted.png")]) } SnapDiff::Reporting.merge_parallel_fragments! - assert_equal [{name: "accepted", path: Pathname("/app/accepted.png"), reason: :rerecorded}], + assert_equal [{name: "accepted", path: Pathname("/app/accepted.png"), status: :rerecorded}], SnapDiff::Reporting.uncompared.map(&:to_h) end # The summary line counts names, merged across workers; the report must # list the same number. test "a new screenshot two workers both captured is listed once" do - 2.times { fork_worker { SnapDiff::Reporting.record_missing_baseline("signup", path: "/app/signup.png") } } + 2.times { fork_worker { SnapDiff::Reporting.notify([build_uncompared_assertion("signup", :new, path: "/app/signup.png")]) } } SnapDiff::Reporting.merge_parallel_fragments! @reporter.finalize @@ -170,7 +173,7 @@ class ParallelReportMergeTest < ActiveSupport::TestCase end test "the parent removes the fragments it merged" do - fork_worker { @reporter.record([build_failing_assertion("cleaned")]) } + fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("cleaned")]) } assert_predicate Pathname.new(SnapDiff::Reporting.parallel_fragments_dir), :exist? @@ -183,10 +186,10 @@ class ParallelReportMergeTest < ActiveSupport::TestCase # written to a `.tmp` name and renamed into place, so a partial file never # carries the name the merge globs for. test "a fragment left half-written by a dead worker is ignored" do - fork_worker { @reporter.record([build_failing_assertion("survivor")]) } + fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("survivor")]) } dir = SnapDiff::Reporting.parallel_fragments_dir - File.write(File.join(dir, "99999.json.tmp"), '{"missing_baselines":["trunc') + File.write(File.join(dir, "99999.json.tmp"), '{"counts":{"passed') SnapDiff::Reporting.merge_parallel_fragments! @@ -194,46 +197,90 @@ class ParallelReportMergeTest < ActiveSupport::TestCase assert_equal 0, SnapDiff::Reporting.missing_baselines_count end - # The fragments directory is keyed by pid under the system temp dir, so a - # recycled pid can hand this merge a fragment written by an older version - # of the gem -- one with none of the keys added since. Every key but - # "missing_baselines" is read with a default for exactly this. - test "a fragment written before the newer tallies existed still merges" do - fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("current")]) } + # A fragment is what the worker tallied (one counts hash, one row per + # listed screenshot) and nothing derived from it: the parent sums the + # counts and gets its "new" count from the merged rows. + test "a fragment carries the status counts and the listed rows, and merging sums them" do + 2.times do |i| + fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("changed_#{i}"), build_passing_assertion("ok_#{i}")]) } + end - dir = SnapDiff::Reporting.parallel_fragments_dir - File.write(File.join(dir, "99998.json"), JSON.generate({"missing_baselines" => ["ancient"], "reporters" => []})) + payloads = Dir[File.join(SnapDiff::Reporting.parallel_fragments_dir, "*.json")].map { |file| JSON.parse(File.read(file)) } + assert_equal [{"passed" => 1, "changed" => 1}] * 2, payloads.map { |payload| payload["counts"] } + assert_empty payloads.flat_map(&:keys) & %w[verified changed missing_baselines rerecorded_baselines] SnapDiff::Reporting.merge_parallel_fragments! - assert_equal 1, SnapDiff::Reporting.verified, "the current worker's counts survived the old fragment" - assert_equal 1, SnapDiff::Reporting.missing_baselines_count - assert_includes SnapDiff::Reporting.missing_baselines_summary, "ancient" - assert_equal [{name: "ancient", path: nil, reason: :new}], SnapDiff::Reporting.uncompared.map(&:to_h), - "listed, with no capture to show" + assert_equal [4, 2], [SnapDiff::Reporting.verified, SnapDiff::Reporting.changed] end - # Only older versions' missing keys are tolerated; an entry no version - # writes is a bug, and it fails loudly instead of being guessed at. - test "an uncompared entry with an unknown reason fails the merge loudly" do + # Both uncompared statuses in one run, from different workers: neither may + # swallow the other's count or its row. + test "a new and a re-recorded screenshot from different workers keep their own counts and rows" do + fork_worker { SnapDiff::Reporting.notify([build_uncompared_assertion("fresh", :new, path: "/app/fresh.png")]) } + fork_worker { SnapDiff::Reporting.notify([build_uncompared_assertion("accepted", :rerecorded, path: "/app/accepted.png")]) } + + SnapDiff::Reporting.merge_parallel_fragments! + + assert_equal "[snap_diff] 0 verified, 0 changed, 1 new (not verified). 1 re-recorded (not verified).", + SnapDiff::Reporting.counts_summary + assert_equal [["fresh", :new], ["accepted", :rerecorded]], + SnapDiff::Reporting.uncompared.map { |row| [row.name, row.status] } + assert_includes SnapDiff::Reporting.missing_baselines_summary, "fresh" + assert_includes SnapDiff::Reporting.rerecorded_baselines_summary, "accepted" + end + + # A status nobody writes is a bug, and it fails loudly instead of being + # guessed at. + test "an outcome with an unknown status fails the merge loudly" do dir = SnapDiff::Reporting.parallel_fragments_dir fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("current")]) } File.write(File.join(dir, "99997.json"), JSON.generate({ - "missing_baselines" => [], "reporters" => [], - "uncompared" => [{"name" => "signup", "path" => nil, "reason" => "bogus"}] + "outcomes" => [{"name" => "signup", "path" => nil, "status" => "bogus"}] })) - error = assert_raises(KeyError) { SnapDiff::Reporting.merge_parallel_fragments! } + error = assert_raises(ArgumentError) { SnapDiff::Reporting.merge_parallel_fragments! } assert_includes error.message, "bogus" ensure FileUtils.rm_rf(dir) end + # The fragments directory is keyed by the parent's pid, and only a + # successful merge removes it. A run killed before that leaves its + # fragments behind, and a later run whose pid is recycled would merge + # them: counts from a run that is not this one, or -- for a fragment in an + # older shape -- a KeyError in `after_run` that costs the whole summary. + # Installing the hooks happens in the parent before any fork, so that is + # where the directory starts out empty. + test "a fragment left behind by an earlier run never reaches the merge" do + skip "Rails' fork-parallel mode needs Kernel#fork, which #{RUBY_ENGINE} does not implement" unless Process.respond_to?(:fork) + + owner = SnapDiff::Reporting.instance_variable_get(:@parallel_owner_pid) + hooks = ActiveSupport::Testing::Parallelization.run_cleanup_hooks + hooks_before = hooks.size + SnapDiff::Reporting.instance_variable_set(:@parallel_owner_pid, Process.pid) + dir = SnapDiff::Reporting.parallel_fragments_dir + FileUtils.mkdir_p(dir) + File.write(File.join(dir, "99998.json"), JSON.generate({"verified" => 7, "changed" => 7})) + SnapDiff::Reporting.instance_variable_set(:@parallel_owner_pid, nil) + + assert SnapDiff::Reporting.install_parallel_hooks! + assert_empty Dir[File.join(dir, "*.json")], "installing the hooks starts the run with an empty directory" + + fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("this_run")]) } + SnapDiff::Reporting.merge_parallel_fragments! + + assert_equal [1, 1], [SnapDiff::Reporting.verified, SnapDiff::Reporting.changed] + ensure + hooks.pop while hooks.size > hooks_before + SnapDiff::Reporting.instance_variable_set(:@parallel_owner_pid, owner) + end + # Serial and `parallelize(with: :threads)` both record in the process that # finalizes; they never write a fragment, and the merge must leave them # exactly as they were. test "merging is a no-op when no worker ever forked" do - @reporter.record([build_failing_assertion("serial")]) + SnapDiff::Reporting.notify([build_failing_assertion("serial")]) SnapDiff::Reporting.merge_parallel_fragments! @@ -263,7 +310,7 @@ class ParallelReportMergeTest < ActiveSupport::TestCase # Merging there would delete the other workers' fragments before the # parent ever sees them. test "a worker that finalizes does not consume the parent's fragments" do - fork_worker { @reporter.record([build_failing_assertion("kept")]) } + fork_worker { SnapDiff::Reporting.notify([build_failing_assertion("kept")]) } pid = fork do SnapDiff::Reporting.merge_parallel_fragments! @@ -328,6 +375,12 @@ def build_passing_assertion(name) SnapDiff::ScreenshotAssertion.new(name).tap { |a| a.compare = compare } end + # A capture that was never compared: `:new`, or `:rerecorded` (`record: + # :all`, which still discards the baseline). + def build_uncompared_assertion(name, status, path: nil) + SnapDiff::ScreenshotAssertion.new(name, path: path && Pathname(path), record_mode: (status == :rerecorded) ? :all : nil) + end + def build_failing_assertion(name) compare = make_comparison(:a, :b, destination: "fail_#{name}") compare.processed diff --git a/test/unit/record_modes_test.rb b/test/unit/record_modes_test.rb index cc3346de..b085a3f5 100644 --- a/test/unit/record_modes_test.rb +++ b/test/unit/record_modes_test.rb @@ -197,8 +197,9 @@ def after_teardown SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion end end + SnapDiff::Reporting.notify(SnapDiff.session.assertions) - assert_equal [{name: name, path: SnapDiff::SnapManager.path_for(name).path, reason: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) + assert_equal [{name: name, path: SnapDiff::SnapManager.path_for(name).path, status: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) end # A file left by an earlier run sits at the path; the capture then fails. @@ -246,8 +247,9 @@ def take_comparison_screenshot(_snapshot) SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion end end + SnapDiff::Reporting.notify(SnapDiff.session.assertions) - assert_equal [{name: name, path: SnapDiff::SnapManager.path_for(name).path, reason: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) + assert_equal [{name: name, path: SnapDiff::SnapManager.path_for(name).path, status: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) assert_includes SnapDiff::Reporting.counts_summary, "1 new (not verified)" end @@ -258,8 +260,9 @@ def take_comparison_screenshot(_snapshot) SnapDiff::Vcs.stub(:checkout_vcs, false) do SnapDiff::ScreenshotMatcher.new(name).build_screenshot_assertion end + SnapDiff::Reporting.notify(SnapDiff.session.assertions) - assert_equal [{name: name, path: SnapDiff::SnapManager.path_for(name).path, reason: :rerecorded}], SnapDiff::Reporting.uncompared.map(&:to_h) + assert_equal [{name: name, path: SnapDiff::SnapManager.path_for(name).path, status: :rerecorded}], SnapDiff::Reporting.uncompared.map(&:to_h) assert_equal 0, SnapDiff::Reporting.missing_baselines_count end @@ -326,6 +329,7 @@ def take_comparison_screenshot(_snapshot) names.each { |n| SnapDiff::ScreenshotMatcher.new(n).build_screenshot_assertion } end + SnapDiff::Reporting.notify(SnapDiff.session.assertions) summary = SnapDiff::Reporting.rerecorded_baselines_summary assert_match(/3 screenshots/, summary) diff --git a/test/unit/reporter_interplay_test.rb b/test/unit/reporter_interplay_test.rb index 9ac780c8..9715e110 100644 --- a/test/unit/reporter_interplay_test.rb +++ b/test/unit/reporter_interplay_test.rb @@ -192,6 +192,8 @@ def summary end end + SnapDiff.reset # the tally: Reporting counts the registered assertions + out, _err = capture_io { SnapDiff::Reporting.finalize! } assert_match(/no committed baseline/i, out) diff --git a/test/unit/reporters/html_reporter_test.rb b/test/unit/reporters/html_reporter_test.rb index 86d4fc9b..dc8175c3 100644 --- a/test/unit/reporters/html_reporter_test.rb +++ b/test/unit/reporters/html_reporter_test.rb @@ -71,6 +71,17 @@ class HTMLReporterTest < ActiveSupport::TestCase assert_includes html, "3 total" end + # The reporter is usable on its own (record + finalize, no Reporting.count + # behind it): its totals are what it recorded itself, never negative. + test "#total and #passed come from what the reporter recorded, not from Reporting" do + reporter = SnapDiff::Reporters::HTML.new(output_path: @output_path) + + reporter.record([build_failing_assertion("page_a"), build_passing_assertion("page_b")]) + + assert_equal [2, 1, 1], [reporter.total, reporter.failed, reporter.passed] + assert_equal 0, SnapDiff::Reporting.verified + end + # An entry that fails to build must not vanish, nor sink the report: it is # warned about, kept as "could not render", and counted as failed. test "#record keeps an entry it could not build, says so, and renders the rest" do @@ -319,7 +330,7 @@ def synchronize test "no other reporter can change what the report lists" do reporter = SnapDiff::Reporters::HTML.new(output_path: @output_path) capture_new_screenshot("signup") - assert_raises(FrozenError) { SnapDiff::Reporting.uncompared.first.reason = :rerecorded } + assert_raises(FrozenError) { SnapDiff::Reporting.uncompared.first.status = :rerecorded } reporter.finalize diff --git a/test/unit/reporting_counts_test.rb b/test/unit/reporting_counts_test.rb index e99d7c05..c20b994b 100644 --- a/test/unit/reporting_counts_test.rb +++ b/test/unit/reporting_counts_test.rb @@ -31,8 +31,7 @@ class ReportingCountsTest < ActiveSupport::TestCase end test "counts what was verified, what changed, and what was never compared" do - SnapDiff::Reporting.record_missing_baseline("never_compared") - SnapDiff::Reporting.notify([build_passing_assertion("ok"), build_failing_assertion("fail")]) + SnapDiff::Reporting.notify([build_passing_assertion("ok"), build_failing_assertion("fail"), build_new_assertion("never_compared")]) assert_equal "[snap_diff] 2 verified, 1 changed, 1 new (not verified).", SnapDiff::Reporting.counts_summary @@ -67,8 +66,7 @@ class ReportingCountsTest < ActiveSupport::TestCase # happened, and must NOT cry NOTHING WAS VERIFIED at a user who asked for # exactly this: a false alarm here trains people to ignore the real one. test "re-recorded screenshots are counted, and explain a zero verified" do - SnapDiff::Reporting.record_rerecorded_baseline("a") - SnapDiff::Reporting.record_rerecorded_baseline("b") + SnapDiff::Reporting.notify([build_rerecorded_assertion("a"), build_rerecorded_assertion("b")]) summary = SnapDiff::Reporting.counts_summary @@ -84,8 +82,7 @@ class ReportingCountsTest < ActiveSupport::TestCase # `record:` is a per-screenshot option too, so a run can mix both. test "a run that verified some and re-recorded others reports both" do - SnapDiff::Reporting.notify([build_passing_assertion("ok")]) - SnapDiff::Reporting.record_rerecorded_baseline("accepted") + SnapDiff::Reporting.notify([build_passing_assertion("ok"), build_rerecorded_assertion("accepted")]) summary = SnapDiff::Reporting.counts_summary @@ -94,8 +91,7 @@ class ReportingCountsTest < ActiveSupport::TestCase end test "counts screenshots captured with no committed baseline even when nothing was verified" do - SnapDiff::Reporting.record_missing_baseline("a") - SnapDiff::Reporting.record_missing_baseline("b") + SnapDiff::Reporting.notify([build_new_assertion("a"), build_new_assertion("b")]) assert_includes SnapDiff::Reporting.counts_summary, "0 verified, 0 changed, 2 new (not verified)." end @@ -104,38 +100,51 @@ class ReportingCountsTest < ActiveSupport::TestCase # `finalize`. The same entries the summary line counts, so a reporter that # lists them lists what the terminal counts. test "uncompared lists each new screenshot once, with its file" do - assert SnapDiff::Reporting.record_missing_baseline("a", path: "/app/a.png"), "first sighting" - refute SnapDiff::Reporting.record_missing_baseline("a", path: "/app/a.png"), "seen before" + SnapDiff::Reporting.notify([build_new_assertion("a", path: "/app/a.png")]) + SnapDiff::Reporting.notify([build_new_assertion("a", path: "/app/a.png")]) - assert_equal [{name: "a", path: Pathname("/app/a.png"), reason: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) + assert_equal [{name: "a", path: Pathname("/app/a.png"), status: :new}], SnapDiff::Reporting.uncompared.map(&:to_h) assert_equal 1, SnapDiff::Reporting.missing_baselines_count end test "uncompared marks re-recorded screenshots as such, after the new ones" do - SnapDiff::Reporting.record_rerecorded_baseline("b", path: "/app/b.png") - SnapDiff::Reporting.record_missing_baseline("a", path: "/app/a.png") + SnapDiff::Reporting.notify([build_rerecorded_assertion("b", path: "/app/b.png"), build_new_assertion("a", path: "/app/a.png")]) - assert_equal [[:new, "a"], [:rerecorded, "b"]], SnapDiff::Reporting.uncompared.map { |s| [s.reason, s.name] } + assert_equal [[:new, "a"], [:rerecorded, "b"]], SnapDiff::Reporting.uncompared.map { |s| [s.status, s.name] } end # Every reporter reads the same entries: none can change what another sees. test "uncompared hands out frozen values" do - SnapDiff::Reporting.record_missing_baseline(+"a", path: "/app/a.png") + SnapDiff::Reporting.notify([build_new_assertion(+"a", path: "/app/a.png")]) list = SnapDiff::Reporting.uncompared assert_predicate list, :frozen? assert_predicate list.first, :frozen? assert_predicate list.first.name, :frozen? - assert_raises(FrozenError) { list.first.reason = :rerecorded } + assert_raises(FrozenError) { list.first.status = :rerecorded } end test "reset_run_totals! forgets the uncompared screenshots" do - SnapDiff::Reporting.record_missing_baseline("a", path: "/app/a.png") - SnapDiff::Reporting.record_rerecorded_baseline("b", path: "/app/b.png") + SnapDiff::Reporting.notify([build_new_assertion("a", path: "/app/a.png"), build_rerecorded_assertion("b", path: "/app/b.png"), build_passing_assertion("ok")]) SnapDiff::Reporting.reset_run_totals! assert_empty SnapDiff::Reporting.uncompared + assert_equal 0, SnapDiff::Reporting.verified + assert_equal 0, SnapDiff::Reporting.missing_baselines_count + end + + # Shipped in 2.0.0.beta4, before every capture was an assertion: still + # callable, and writing into the same store `notify` fills. + test "record_missing_baseline and record_rerecorded_baseline write into the same store as notify" do + assert SnapDiff::Reporting.record_missing_baseline("a", path: "/app/a.png"), "first sighting" + refute SnapDiff::Reporting.record_missing_baseline("a", path: "/app/a.png"), "seen before" + assert SnapDiff::Reporting.record_rerecorded_baseline("b"), "first sighting" + refute SnapDiff::Reporting.record_rerecorded_baseline("b"), "seen before, and counted once" + SnapDiff::Reporting.notify([build_new_assertion("a"), build_new_assertion("c")]) + + assert_equal [[:new, "a"], [:new, "c"], [:rerecorded, "b"]], SnapDiff::Reporting.uncompared.map { |s| [s.status, s.name] } + assert_includes SnapDiff::Reporting.counts_summary, "0 verified, 0 changed, 2 new (not verified). 1 re-recorded" end # The reason this moved: the gem's one and only `register` call site is in @@ -150,9 +159,15 @@ class ReportingCountsTest < ActiveSupport::TestCase end test "an assertion with no comparison is neither verified nor changed" do - SnapDiff::Reporting.notify([SnapDiff::ScreenshotAssertion.new("never_compared")]) + SnapDiff::Reporting.notify([build_new_assertion("never_compared")]) - assert_includes SnapDiff::Reporting.counts_summary, "0 verified, 0 changed" + assert_includes SnapDiff::Reporting.counts_summary, "0 verified, 0 changed, 1 new" + end + + test "a name already tallied keeps its first row" do + SnapDiff::Reporting.notify([build_new_assertion("a", path: "/app/first.png"), build_new_assertion("a", path: "/app/second.png")]) + + assert_equal [Pathname("/app/first.png")], SnapDiff::Reporting.uncompared.map(&:path) end # --- selectors that never matched (#277b) ------------------------------- @@ -305,6 +320,14 @@ class ReportingCountsTest < ActiveSupport::TestCase private + def build_new_assertion(name, path: nil) + SnapDiff::ScreenshotAssertion.new(name, path: path && Pathname(path)) + end + + def build_rerecorded_assertion(name, path: nil) + SnapDiff::ScreenshotAssertion.new(name, path: path && Pathname(path), record_mode: :all) + end + def build_passing_assertion(name) build_assertion(name, :a, :a, "pass") end From a01c4e056a89e94873c5dff87bc0c3b154af9cb4 Mon Sep 17 00:00:00 2001 From: Diego Algorta Date: Sat, 17 Oct 2026 03:13:49 -0300 Subject: [PATCH 3/3] feat: record: :all compares before re-recording and reports what changed `record: :all` used to fetch no baseline and compare nothing, so a run lumped changed, unchanged and new screenshots together as "re-recorded" and the report could only count them. Every update mode of the tools surveyed (Playwright, Jest, BackstopJS `approve`, reg-cli, Chromatic) compares first and decides from the result; only VCR's `:all`, whose HTTP cassettes have no notion of a difference, records blind. Now `:all` is `:once` that accepts instead of fails. The baseline is checked out from git and compared in every mode, and the comparison only labels the capture: - status: differs under `:all` => `:rerecorded`; matches => `:passed`; no baseline => `:new`. The run never raises for a difference. - `:passed` calls `archive_baseline!` in every mode, so committed bytes are restored and no `.base.png` is left; a `:rerecorded` capture keeps the usual `.base.png`/diff/heatmap artifacts (gitignored by the recommended `.gitignore`, which the re-record summary now says). - Reporting: verified and changed both include re-recorded; `uncompared` lists `:new` only (re-recorded ones were compared). - One vocabulary on both outputs: the terminal says `3 changed (3 re-recorded)`, and the report's title and pill say the same instead of "failures" / "failed" next to a separate re-recorded pill. - HTML report: one entry per non-passed assertion, each with a `status` (changed|new|rerecorded) replacing `new_screenshot`. An entry the report could not build keeps its real status and carries an `error` beside it, so the report and the terminal count the same things (a new screenshot that fails to render is still "1 new", not "1 changed"). The reporter still counts the compared assertions it records, so a standalone reporter's numbers are never negative. The template reads `item.status` directly and shows a re-recorded badge. Still one static file. HTML#changed_entries / #changed_count carry the visible vocabulary; #failures / #failed stay as shims for the beta4 names. - The re-record summary offers `git restore -- ` to discard. - AI triage (AISimple) still only runs for `:changed`: a re-recorded screenshot is accepted without analysis, on purpose; docs/reporters.md says so. Summary line, before (one screenshot that differs, under `:all`): [snap_diff] 0 verified, 0 changed, 0 new (not verified). 1 re-recorded (not verified). after: [snap_diff] 1 verified, 1 changed (1 re-recorded), 0 new (not verified). (the parenthesis appears only when something was re-recorded; otherwise the line is byte-identical to before). A bootstrap run where everything is new still prints NOTHING WAS VERIFIED. A difference that comes from a missing input (the capture vanished before the comparison) stays `:changed` in every mode: there is nothing on disk for `:all` to accept, and "re-recorded" would tell the user to `git add` a path that no longer exists. Not changed: the refusal of `:all` under CI (only its message, which no longer says nothing is compared); the report stays static. --- CHANGELOG.md | 27 ++- README.md | 7 +- docs/UPGRADING.md | 6 +- docs/architecture.md | 2 +- docs/ci-integration.md | 2 +- docs/configuration.md | 15 +- docs/images/snap_diff_assertion_flow.html | 4 +- docs/reporters.md | 8 +- docs/snapdiff.md | 13 +- lib/snap_diff/config.rb | 9 +- lib/snap_diff/reporters/html.rb | 119 ++++++++----- .../reporters/templates/report.html.erb | 57 +++--- lib/snap_diff/reporting.rb | 81 ++++----- lib/snap_diff/screenshot_assertion.rb | 15 +- lib/snap_diff/screenshot_matcher.rb | 38 ++-- test/fixtures/summary_line_case.rb | 2 +- test/integration/summary_line_test.rb | 24 ++- test/support/test_doubles.rb | 6 + test/unit/parallel_report_merge_test.rb | 50 +++--- test/unit/record_mode_matrix_test.rb | 112 ++++++++++++ test/unit/record_modes_test.rb | 97 +++++++---- test/unit/reporters/ai_simple_test.rb | 6 +- test/unit/reporters/html_reporter_test.rb | 163 ++++++++++++------ test/unit/reporting_counts_test.rb | 60 ++++--- test/unit/screenshot_assertion_status_test.rb | 20 ++- 25 files changed, 606 insertions(+), 337 deletions(-) create mode 100644 test/unit/record_mode_matrix_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 456ea205..09bc11c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -179,8 +179,8 @@ and assigning `nil` hands the setting back to the environment. - **Accepting an intentional change is one setting, not folklore.** `record` replaces `fail_if_new` (removed in 2.1): `:once` records a missing baseline and compares the rest — what a local run already did; `:none` **fails** on a missing baseline with the `git add` - command attached, and is what CI gets by default; `:all` re-records every screenshot and - compares nothing, the bulk-accept verb for the run after an intentional redesign. 1.15.1 + command attached, and is what CI gets by default; `:all` compares like `:once` but accepts + what differs instead of failing, the bulk-accept verb for the run after an intentional redesign. 1.15.1 documented `RECORD_SCREENSHOTS=1` in four files and printed it in its own error message, and nothing in the gem ever read that variable — accepting a change was answered in issue comments for six years and never in the docs. See `docs/configuration.md` (#259) @@ -331,17 +331,27 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV the name to install; the mirror reserves the forward-looking one. **Install one, never both.** - **Reporters can list the screenshots that were not compared.** - `SnapDiff::Reporting.uncompared` returns the ones behind the summary line's "new" and - "re-recorded" counts, complete by the time reporters `finalize` (fork-parallel workers - included): each with its `name`, the captured file's `path`, and the `status` — `:new` - (no committed baseline) or `:rerecorded` (`record: :all`). The bundled HTML report lists - new screenshots from it + `SnapDiff::Reporting.uncompared` returns the ones behind the summary line's "new" + count, complete by the time reporters `finalize` (fork-parallel workers included): each + with its `name`, the captured file's `path`, and the `status` (`:new`, no committed + baseline). The bundled HTML report lists new screenshots from it - New documentation shipped inside the gem: [docs/snapdiff.md](docs/snapdiff.md) (canonical API) and [docs/UPGRADING.md](docs/UPGRADING.md) (every renamed constant, which names warn, rollback) ### Changed +- **`record: :all` now compares before re-recording and the report shows what changed.** + A screenshot that differs from its committed baseline is accepted as the new baseline + (it never fails the run) and leaves the usual `.base.png` and diff/heatmap files, + which the recommended `.gitignore` already ignores; a screenshot that matches is left + as committed. The summary line counts the accepted ones in a parenthesis — + `12 verified, 3 changed (3 re-recorded), 0 new (not verified).` — and the HTML report + lists each of them with its baseline, current and heatmap views and a "re-recorded" + badge, and its title and pill use the same `N changed (R re-recorded)` wording as the + terminal (the title used to say "failures"). Report entries gain a `status` (`changed`, + `new` or `rerecorded`) in place of `new_screenshot`; an entry the report could not + render also carries an `error`. - **Reporters now receive every screenshot, each with a `status`.** New and re-recorded (`record: :all`) captures used to bypass them; now every capture is a `SnapDiff::ScreenshotAssertion` whose `status` is `:passed`, `:changed`, `:new` or @@ -458,8 +468,7 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV baseline appears after the failures, marked `new`, with its capture and the `git add` that makes it the baseline; a run whose only finding is a new screenshot now writes a report. The report's `new` count is the summary line's, including - across `parallelize(workers: N)`. Re-recorded screenshots (`record: :all`) are - counted in the report but not yet listed + across `parallelize(workers: N)` - **Under `record: :none` a new screenshot is counted.** The run still fails as before; the summary line now says `1 new` instead of `0 new` - **A committed baseline taken by a same-name test is reported as disappeared, in every diff --git a/README.md b/README.md index c7ff066c..c8e6bc8c 100644 --- a/README.md +++ b/README.md @@ -211,15 +211,16 @@ SnapDiff.config.record = :all unless ENV["ACCEPT_SCREENSHOTS"].to_s.empty? ```bash ACCEPT_SCREENSHOTS=1 bin/rails test:system -# [snap_diff] record: :all re-recorded 40 screenshots WITHOUT comparing: ... +# [snap_diff] 212 verified, 40 changed (40 re-recorded), 0 new (not verified). git status # forty modified baselines, and nothing else git add doc/screenshots/ git commit -m "chore: re-record baselines after the checkout redesign" ``` -`:all` writes every capture straight to its baseline path and compares nothing, so `git status` -is the review surface — look at the images before you commit. It **refuses to run under CI**, +`:all` compares every capture to its committed baseline like `:once` does, but a screenshot that +differs is accepted as the new baseline instead of failing the test — the report lists what changed, +and `git status` is the review surface — look at the images before you commit. It **refuses to run under CI**, where there would be nobody to look. The other modes are `:once` (the default) and `:none` (strict — a missing baseline always fails); see [Record modes](docs/configuration.md#record-modes--accepting-changes). diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index c18a420b..44a57e92 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -319,7 +319,7 @@ actually want, which is *accept this change*. 2.0 adds the verb: ```ruby SnapDiff.config.record = :once # default. Record a screenshot that has no baseline. SnapDiff.config.record = :none # strict. A missing baseline always fails. -SnapDiff.config.record = :all # re-record everything. THE BULK-ACCEPT MODE. +SnapDiff.config.record = :all # compare, and accept what differs. THE BULK-ACCEPT MODE. ``` | Old | New | Note | @@ -342,8 +342,8 @@ missing-baseline default is deliberately unchanged; `:none` is how you opt into explicitly. **Precedence:** an explicitly set mode outranks `fail_if_new`, and `fail_if_new` decides only when no mode was set — the same rule `fail_if_new` itself has over the `CI` sniff. -`record = :all` **refuses to run under CI**: it accepts every rendering by design, so left in a -committed config file it would be a build that compares nothing and passes forever. See +`record = :all` **refuses to run under CI**: it accepts every difference by design, so left in a +committed config file it would be a build that never fails. See [Record modes](configuration.md#record-modes--accepting-changes). #### Silent by design diff --git a/docs/architecture.md b/docs/architecture.md index cd67a823..194cf689 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -194,7 +194,7 @@ Test begins ├─ teardown: │ ├─ SnapDiff.session.verify │ │ ├─ iterates the fiber-local assertions - │ │ ├─ calls validate on each (new/re-recorded ones have no comparison and pass) + │ │ ├─ calls validate on each (new and re-recorded ones do not fail) │ │ └─ raises SnapDiff::ExpectationNotMet if any differed │ │ │ └─ SnapDiff.reset (always, in an ensure) diff --git a/docs/ci-integration.md b/docs/ci-integration.md index 905e9f16..1d37c886 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -273,7 +273,7 @@ jobs: > than relying on the files being there. > > **`record = :all` is not the mode for this job** — it refuses to run under CI, because it -> would accept every *changed* rendering unreviewed as well. `:once` records what is new and +> would accept every *changed* rendering unreviewed and never fail. `:once` records what is new and > keeps comparing everything that already has a baseline. **How it works:** diff --git a/docs/configuration.md b/docs/configuration.md index bbb65a65..a7d6591f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -63,7 +63,7 @@ SnapDiff.config.record = :once # what a local run already does; see Precedence |------|------------------|------------------|-------------------| | `:once` | recorded, not compared | compared | the default — you want a normal run | | `:none` | **fails** with the `git add` command attached | compared | every screenshot must already be recorded | -| `:all` | recorded | **re-recorded, not compared** | you changed the UI on purpose and want the new rendering to become the baseline | +| `:all` | recorded | compared; **re-recorded on difference, never fails** | you changed the UI on purpose and want the new rendering to become the baseline | ### `:all` — the bulk-accept verb @@ -82,19 +82,22 @@ SnapDiff.config.record = :all unless ENV["ACCEPT_SCREENSHOTS"].to_s.empty? ACCEPT_SCREENSHOTS=1 bin/rails test:system # with the line above in test_helper.rb ``` -Every screenshot is written to its baseline path and nothing is compared, so `git status` lists -exactly what changed. Review the images, then commit them — see +Every screenshot is compared to its committed baseline, as under `:once`. One that differs is kept +as the new baseline instead of failing the test, and leaves the usual `.base.png` and diff files +(ignored by the recommended `.gitignore`); one that matches changes nothing. So `git status` lists +exactly what changed, and the HTML report shows each change as "re-recorded". Review the images, then commit them — see [Accepting an intentional change](../README.md#accepting-an-intentional-change). At the end of the run the gem names what it accepted: ``` -[snap_diff] record: :all re-recorded 3 screenshots WITHOUT comparing: checkout/cart, checkout/payment, checkout/review. Review the result before committing -- an unintended change is accepted just as silently. +[snap_diff] 212 verified, 3 changed (3 re-recorded), 0 new (not verified). +[snap_diff] record: :all compared 3 screenshots, found them different and accepted them as the new baseline: checkout/cart, checkout/payment, checkout/review. `git add` the file(s) to keep the change, `git restore -- ` to discard it. The `.base.` and diff files left beside each one are covered by the recommended .gitignore. ``` > **`:all` refuses to run under CI** (when `ENV['CI']` is set to a non-empty value). It accepts -> every rendering by design, so a mode left in a committed config file would buy you a build that -> compares nothing and passes forever, with the "recorded" screenshots discarded when the runner +> every difference by design, so a mode left in a committed config file would buy you a build that +> never fails, with the "recorded" screenshots discarded when the runner > is torn down. Re-record locally, where you can look at the result. > > A CI job that needs to record screenshots with **no baseline yet** does not need `:all`: diff --git a/docs/images/snap_diff_assertion_flow.html b/docs/images/snap_diff_assertion_flow.html index 7ea1318a..34049ce2 100644 --- a/docs/images/snap_diff_assertion_flow.html +++ b/docs/images/snap_diff_assertion_flow.html @@ -139,8 +139,8 @@

What one assert_matches_screenshot actually does

HTML + summary line - record: :all sends every screenshot - down the right-hand path — baseline or not. + record: :all still compares; it accepts + a difference instead of failing on it. diff --git a/docs/reporters.md b/docs/reporters.md index d55ac41e..b70d50b1 100644 --- a/docs/reporters.md +++ b/docs/reporters.md @@ -135,9 +135,10 @@ still run. baseline) or `:rerecorded` (accepted by `record: :all`). `compare` is nil for a screenshot without a baseline; `assertion.path` is the captured file either way. Test `status`, not `compare`, to find failures: the tally has already run every comparison by the time `record` -is called, and reading `status` never forces one. The new and re-recorded screenshots are also in +is called, and reading `status` never forces one. `:rerecorded` was compared and differed, +like `:changed`, but `record: :all` accepted it. The new screenshots are also in `SnapDiff::Reporting.uncompared` (each with `name`, `path` and `status`), complete by the time -`finalize` runs (fork-parallel workers included). The HTML report lists new screenshots from it. +`finalize` runs (fork-parallel workers included); `status` is the contract for everything else. Full details in [Custom reporters](snapdiff.md#custom-reporters). @@ -149,4 +150,7 @@ default, pluggable backends) and its verdicts show up in this HTML report as badges. Advisory by default — it never changes pass/fail unless you opt in with `fail_on: %w[real_bug]`. Setup and recipes: [AI triage](ai.md). +It triages `:changed` comparisons only. A `:rerecorded` one was accepted by `record: :all`, so it is +not analysed, on purpose: a bulk accept should not cost a backend call per screenshot. + [← Back to README](../README.md) diff --git a/docs/snapdiff.md b/docs/snapdiff.md index ecb7e742..1fd8c576 100644 --- a/docs/snapdiff.md +++ b/docs/snapdiff.md @@ -259,19 +259,18 @@ and then warned about (`[snap_diff] Reporter … failed (NoMethodError: undefine A reporter that raises is warned about and skipped — the other reporters still run. `record` receives every assertion — one per capture. `assertion.status` says what you hold: -`:passed`, `:changed`, `:new` (no committed baseline) or `:rerecorded` (`record: :all`). +`:passed`, `:changed`, `:new` (no committed baseline) or `:rerecorded` (differs, accepted by `record: :all`). `compare` is nil for a screenshot without a baseline; `assertion.path` is the captured file either way. The tally has already run every comparison by the time `record` is called, so -test `status == :changed` rather than digging into `compare`; reading `status` never forces one. The new and re-recorded -screenshots — the summary line's "new" and "re-recorded" — are also in -`SnapDiff::Reporting.uncompared`, complete by the time `finalize` runs (fork-parallel workers -included): +test `status == :changed` rather than digging into `compare`; reading `status` never forces one. +The new screenshots — the summary line's "new" — are also in `SnapDiff::Reporting.uncompared`, +complete by the time `finalize` runs (fork-parallel workers included): ```ruby def finalize - new_ones = SnapDiff::Reporting.uncompared.select { |s| s.status == :new } + new_ones = SnapDiff::Reporting.uncompared # each has #name, #path (the captured file as a Pathname, or nil when it isn't known) - # and #status (:new or :rerecorded); check s.path&.exist? before reading it + # and #status (:new); check s.path&.exist? before reading it end ``` diff --git a/lib/snap_diff/config.rb b/lib/snap_diff/config.rb index 8c4442e8..3ff9a5ab 100644 --- a/lib/snap_diff/config.rb +++ b/lib/snap_diff/config.rb @@ -190,15 +190,16 @@ def fail_on_difference=(value) end # THE ACCEPT WORKFLOW (#259). What to do about a screenshot whose - # baseline is missing -- or, for +:all+, about every screenshot there - # is. VCR-shaped: modes in config, because there is no runner to hang a + # baseline is missing -- or, for +:all+, about every screenshot that + # differs. VCR-shaped: modes in config, because there is no runner to hang a # CLI flag on. # # +:once+:: (default) record a screenshot that has no committed # baseline; compare against the baseline when there is one. # +:none+:: strict. A missing baseline always fails. - # +:all+:: re-record. Every screenshot is written as the new baseline - # and nothing is compared -- the bulk-accept verb, for the + # +:all+:: +:once+ that accepts instead of fails. Every baseline is + # compared; a screenshot that differs is kept as the new + # baseline and never raises -- the bulk-accept verb, for the # redesign that changed forty screenshots at once. Refused # under CI (see {SnapDiff::ScreenshotMatcher}). # diff --git a/lib/snap_diff/reporters/html.rb b/lib/snap_diff/reporters/html.rb index a893e329..3f4e4105 100644 --- a/lib/snap_diff/reporters/html.rb +++ b/lib/snap_diff/reporters/html.rb @@ -14,12 +14,16 @@ module SnapDiff module Reporters class HTML - attr_reader :failures, :total + # One entry per screenshot that was not a plain pass, each with a + # `status` ("changed", "rerecorded" or "new") and, when the report + # could not render it, an `error` beside it. + attr_reader :entries, :total def initialize(output_path: nil, embed_images: false) @explicit_output_path = output_path @embed_images = embed_images - @failures = [] + @entries = [] + @new_names = Set.new @total = 0 @finalized = false @mutex = Mutex.new @@ -28,39 +32,38 @@ def initialize(output_path: nil, embed_images: false) def record(assertions) return if @finalized - failures = [] + entries = [] total = 0 assertions.each do |assertion| - counted = false - compare = assertion.compare - next unless compare - - total += 1 - counted = true - next unless compare.difference&.different? - - failures << failure_entry_for(assertion.name, compare) - rescue => e - # A report bug must not hide a screenshot: warn, and keep it listed - # (counted as failed: the report can't vouch for it). - error = "#{e.class}: #{e.message}" - warn "[snap_diff] HTML report: could not render '#{assertion.name}' (#{error})" - total += 1 unless counted - failures << {name: assertion.name, error: error} + status = assertion.status + entry = begin + entry_for(assertion, status) unless status == :passed + rescue => e + # A report bug must not hide a screenshot: warn, and keep it + # listed with the status it has and the error beside it. The + # error says the report could not render it, not what became of + # the screenshot. + error = "#{e.class}: #{e.message}" + warn "[snap_diff] HTML report: could not render '#{assertion.name}' (#{error})" + {name: assertion.name, status: status.to_s, error: error} + end + + total += 1 unless status == :new + entries << entry if entry end @mutex.synchronize do return if @finalized @total += total - @failures.concat(failures) + add_entries(entries) end end def finalize @mutex.synchronize do return if @finalized - return if failures.empty? && new_count.zero? + return if entries.empty? write_report @finalized = true @@ -78,23 +81,34 @@ def output_path # file is discarded rather than half-read. Image paths were resolved # against `output_path`, which is the same in both processes. def dump_state - @mutex.synchronize { {"total" => @total, "failures" => @failures} } + @mutex.synchronize { {"total" => @total, "entries" => @entries} } end def merge_state!(state) @mutex.synchronize do - @total += state["total"] - @failures.concat(state["failures"].map { |entry| entry.transform_keys(&:to_sym) }) + @total += state.fetch("total") + add_entries(state.fetch("entries").map { |entry| entry.transform_keys(&:to_sym) }) end end - def passed = total - failures.size - def failed = failures.size - # Screenshots captured but not compared: new ones are listed, - # re-recorded ones only counted. - def new_screenshots = SnapDiff::Reporting.uncompared.select { |screenshot| screenshot.status == :new } - def new_count = new_screenshots.size - def rerecorded_count = SnapDiff::Reporting.uncompared.size - new_count + # The entries that were compared and differed, and not accepted: + # `:changed`. (Ones the report could not render keep their status.) + def changed_entries = entries.select { |entry| entry[:status] == "changed" } + + def new_count = count_entries("new") + def rerecorded_count = count_entries("rerecorded") + + # What the report shows as "changed", like the terminal summary: the + # ones that differed, re-recorded included. + def changed_count = changed_entries.size + rerecorded_count + + # Shipped names (2.0.0.beta4), kept as shims: what is "changed" here + # was "failures" there, and the re-recorded ones are not part of it. + def failures = changed_entries + + def failed = changed_entries.size + + def passed = total - changed_count # Both customer personas named this path as the best output in the # product, so it gets its own line -- but only when a report was @@ -112,7 +126,6 @@ def summary # Attach available contributions and return the rendered HTML report. def render attach_annotations - @entries = failures + new_screenshot_entries ERB.new(File.read(self.class.template_path)).result(binding) end @@ -127,8 +140,28 @@ def self.default_output_path private + # Under the mutex. A name with no baseline is one new screenshot + # however many times it was captured (as Reporting counts it). + def add_entries(entries) + entries.each do |entry| + next if entry[:status] == "new" && !@new_names.add?(entry[:name]) + + @entries << entry + end + end + + def count_entries(status) = entries.count { |entry| entry[:status] == status } + + def entry_for(assertion, status) + if status == :new + new_screenshot_entry(assertion.name, assertion.path) + else + compared_entry_for(assertion.name, assertion.compare).merge(status: status.to_s) + end + end + # Build a report entry from a comparison, omitting unavailable images and metrics. - def failure_entry_for(name, compare) + def compared_entry_for(name, compare) { name: name, original: resolve_image(compare.base_image_path), @@ -147,11 +180,11 @@ def failure_details(compare) end # An input that vanished before the comparison leaves nothing to diff - # or annotate. The entry carries the error so the page says so, instead - # of inferring "pass" from the absent diff keys. resolve_image already - # drops the missing file from original/new. + # or annotate. The entry keeps its status (it was compared and differs) + # and carries the error so the page says why there is nothing to show. + # resolve_image already drops the missing file from original/new. def missing_input_details(compare) - {error: compare.error_message.lines.first.strip} + {error: compare.processed.error_message.lines.first.strip} end # Images of different sizes get no annotated or heatmap images, and @@ -180,7 +213,7 @@ def difference_details(compare) # HTML never names a contributor -- it renders whatever # SnapDiff::Contributions returns, empty by default. def attach_annotations - failures.each do |entry| + entries.each do |entry| # ||= would create a nil :annotations key on misses; keep the # entry clean. annotations = SnapDiff::Contributions.annotations_for(entry[:name]) @@ -188,14 +221,10 @@ def attach_annotations end end - # Built at render time: the image and the `git add` are offered only - # for a file that is on disk now (#260). - def new_screenshot_entries - new_screenshots.map { |screenshot| new_screenshot_entry(screenshot.name, screenshot.path) } - end - + # The image and the `git add` are offered only for a file that is on + # disk now (#260). def new_screenshot_entry(name, path) - entry = {name: name, new_screenshot: true} + entry = {name: name, status: "new"} image = resolve_image(path) return entry unless image diff --git a/lib/snap_diff/reporters/templates/report.html.erb b/lib/snap_diff/reporters/templates/report.html.erb index b91a059d..9237553c 100644 --- a/lib/snap_diff/reporters/templates/report.html.erb +++ b/lib/snap_diff/reporters/templates/report.html.erb @@ -3,7 +3,7 @@ - SnapDiff Report — <%= failed %> failures<%= ", #{new_count} new" if new_count.positive? %> + SnapDiff Report — <%= changed_count %> changed<%= " (#{rerecorded_count} re-recorded)" if rerecorded_count.positive? %><%= ", #{new_count} new" if new_count.positive? %>