Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,12 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV
content and version, matching this repository. `capybara-screenshot-diff` remains
the name to install; the mirror reserves the forward-looking one. **Install one,
never both.**
- **Reporters can list the screenshots that were not compared.**
`SnapDiff::Reporting.uncompared` returns the ones behind the summary line's "new" and
"re-recorded" counts, complete by the time reporters `finalize` (fork-parallel workers
included): each with its `name`, the captured file's `path`, and the `reason` — `:new`
(no committed baseline) or `:rerecorded` (`record: :all`). The bundled HTML report lists
new screenshots from it
- New documentation shipped inside the gem:
[docs/snapdiff.md](docs/snapdiff.md) (canonical API) and
[docs/UPGRADING.md](docs/UPGRADING.md) (every renamed constant, which names warn,
Expand Down Expand Up @@ -416,10 +422,23 @@ One removal 2.0 cannot warn about, written down instead: the legacy `LOADED_DRIV
- **A size change labels its images like every other failure.** `Dimensions have
changed` now lists `baseline:` and `actual:`, relative to the project root, instead of
two bare absolute paths, and never names annotations left over from an earlier run
- **The HTML report lists new screenshots.** A screenshot captured with no committed
baseline appears after the failures, marked `new`, with its capture and the
`git add` that makes it the baseline; a run whose only finding is a new screenshot
now writes a report. The report's `new` count is the summary line's, including
across `parallelize(workers: N)`. Re-recorded screenshots (`record: :all`) are
counted in the report but not yet listed
- **Under `record: :none` a new screenshot is counted.** The run still fails as before;
the summary line now says `1 new` instead of `0 new`
- **A committed baseline taken by a same-name test is reported as disappeared, in every
mode.** Before, if it vanished right after the checkout, the run also warned there was
no committed baseline and counted it as new; under `record: :none` it failed with
"No existing screenshot found … `git add` it" instead of saying it disappeared

### Known limitations
- **One process per worker** (`parallel_tests`, RSpec, CI sharding) still writes one
report per process to the same path, so only the last to finish survives. Rails'
report per process to the same path, so only the last to finish survives,
now including a process whose only finding is a new screenshot. Rails'
`parallelize(workers: N)` and `parallelize(with: :threads)` are both complete —
see [docs/reporters.md](docs/reporters.md#parallel-test-runs)

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ Add one line to get an interactive dashboard for reviewing all screenshot differ

```ruby
# test/test_helper.rb
require 'capybara_screenshot_diff/reporters/html'
require 'snap_diff/reporters/html'
```

After tests run, open `doc/screenshots/snap_diff_report.html`:
Expand Down
5 changes: 5 additions & 0 deletions docs/ci-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,11 @@ That's it. On failure, this will:
- Post a PR comment with links to the inline report and full artifact download
- Add a job summary with report links (visible in the Actions UI)

The report also lists *new* screenshots — captured with no committed baseline — each with the
`git add` that makes it the baseline. On CI a new screenshot fails its test, so `if: failure()`
uploads it. If you set `SnapDiff.config.record = :once` on CI, a run whose only news is a new
screenshot passes: upload with `if: always()` to still get the report.

#### Inputs

| Input | Default | Description |
Expand Down
22 changes: 16 additions & 6 deletions docs/reporters.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,19 @@ require 'snap_diff/reporters/html' # canonical
# require 'capybara_screenshot_diff/reporters/html' # legacy, same thing
```

After running tests, open the report (generated only when there are failures):
After running tests, open the report (written when a screenshot changed or is new):

```bash
open doc/screenshots/snap_diff_report.html
```

The report includes a sidebar with thumbnails, side-by-side comparison with diff toggle, search, and summary stats. No configuration needed — just require it.

**Note:** The report is not generated when all screenshots match.
New screenshots — captured with no committed baseline — are listed after the failures, marked
`new`: the capture, and the `git add` that makes it the baseline (baselines are read from git, so
committing it is what accepts it). Their count in the report is the summary line's `new`.

**Note:** The report is not written when every screenshot matched and none is new.

## The end-of-run summary

Expand All @@ -30,8 +34,9 @@ Every run ends with what it actually did, whether or not you require a reporter:

- **verified** — a committed baseline existed and was compared
- **changed** — of those, the ones that differed
- **new** — captured but *not* compared, for want of a committed baseline: neither a pass nor a
failure. Commit the files to turn them into baselines.
- **new** — captured but *not* compared, for want of a committed baseline: not a pass. Under
`record: :none` (the default on CI) it also fails its test. Commit the files to turn them into
baselines.

`0 verified` is printed as `NOTHING WAS VERIFIED`, because it is the only signal for the failures
no per-assertion rule can see: a `rake test` that ran zero system tests, or an inherited `GIT_DIR`
Expand All @@ -52,8 +57,8 @@ that hook fires in the process holding the results depends on how your runner pa
| How the suite runs | Report |
| --- | --- |
| Serial | Written, complete. |
| `parallelize(with: :threads)` (also the default on JRuby) | Written, complete — same failures and counts as a serial run; only the order of the entries differs. Verified over repeated runs, provided [screenshot names are unique across tests](thread_safety.md#screenshot-names-must-be-unique-across-tests). |
| `parallelize(workers: N)` (Rails' default, forks) | Written, complete — one report at the usual path, merged from every worker. No configuration needed. |
| `parallelize(with: :threads)` (also the default on JRuby) | Written, complete — same failures, new screenshots and counts as a serial run; only the order of the entries differs. Verified over repeated runs, provided [screenshot names are unique across tests](thread_safety.md#screenshot-names-must-be-unique-across-tests). |
| `parallelize(workers: N)` (Rails' default, forks) | Written, complete — one report at the usual path, merged from every worker, new screenshots included. No configuration needed. |
| One process per worker (`parallel_tests`, RSpec, CI sharding) | Written, but only the **last** process to finish is in it; the others are overwritten. |

Under Rails' forking parallelism the workers hold the results but never finalize: Minitest skips its
Expand Down Expand Up @@ -125,6 +130,11 @@ the framework's end-of-suite hook (`Minitest.after_run`, RSpec `after(:suite)`,
finalized and then warned about. A reporter that raises is warned about and skipped — the others
still run.

`record` only sees screenshots that were compared. The others — each new (`reason: :new`) or
re-recorded (`reason: :rerecorded`) screenshot, with its `name` and captured `path` — are in
`SnapDiff::Reporting.uncompared`, complete by the time `finalize` runs (fork-parallel workers
included). The HTML report lists new screenshots from it.

Full details in [Custom reporters](snapdiff.md#custom-reporters).

## AI triage reporter
Expand Down
12 changes: 12 additions & 0 deletions docs/snapdiff.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,18 @@ and then warned about (`[snap_diff] Reporter … failed (NoMethodError: undefine

A reporter that raises is warned about and skipped — the other reporters still run.

`record` only receives screenshots that were compared. The ones that were not — the summary
line's "new" and "re-recorded" — are in `SnapDiff::Reporting.uncompared`, complete by the time
`finalize` runs (fork-parallel workers included):

```ruby
def finalize
new_ones = SnapDiff::Reporting.uncompared.select { |s| s.reason == :new }
# each has #name, #path (the captured file as a Pathname, or nil when it isn't known)
# and #reason (:new or :rerecorded); check s.path&.exist? before reading it
end
```

The bundled HTML report is just a pre-registered reporter of this kind:

```ruby
Expand Down
31 changes: 30 additions & 1 deletion lib/snap_diff/reporters/html.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
require "fileutils"
require "pathname"
require "json"
require "shellwords"
# The auto-registration block at the bottom of this file registers with
# SnapDiff::Reporting.
require "snap_diff/reporting"
Expand Down Expand Up @@ -59,7 +60,7 @@ def record(assertions)
def finalize
@mutex.synchronize do
return if @finalized
return if failures.empty?
return if failures.empty? && new_count.zero?

write_report
@finalized = true
Expand Down Expand Up @@ -89,6 +90,11 @@ def merge_state!(state)

def passed = total - failures.size
def failed = failures.size
# Screenshots captured but not compared: new ones are listed,
# re-recorded ones only counted.
def new_screenshots = SnapDiff::Reporting.uncompared.select { |screenshot| screenshot.reason == :new }
def new_count = new_screenshots.size
def rerecorded_count = SnapDiff::Reporting.uncompared.size - new_count

# Both customer personas named this path as the best output in the
# product, so it gets its own line -- but only when a report was
Expand All @@ -106,6 +112,7 @@ def summary
# Attach available contributions and return the rendered HTML report.
def render
attach_annotations
@entries = failures + new_screenshot_entries
ERB.new(File.read(self.class.template_path)).result(binding)
end

Expand Down Expand Up @@ -164,6 +171,28 @@ def attach_annotations
end
end

# Built at render time: the image and the `git add` are offered only
# for a file that is on disk now (#260).
def new_screenshot_entries
new_screenshots.map { |screenshot| new_screenshot_entry(screenshot.name, screenshot.path) }
end

def new_screenshot_entry(name, path)
entry = {name: name, new_screenshot: true}
image = resolve_image(path)
Comment thread
sourcery-ai[bot] marked this conversation as resolved.
return entry unless image

entry.merge(new: image, git_add: "git add -- #{Shellwords.escape(display_path(path))}")
end
Comment thread
sourcery-ai[bot] marked this conversation as resolved.

# Relative to the project root, where the user runs `git add` -- the
# same form as the failure messages. Absolute when the file is outside
# the root, where a relative path would mislead.
def display_path(path)
relative = Pathname(path).relative_path_from(SnapDiff.config.root).to_path
relative.start_with?("..") ? path.to_s : relative
end

Comment thread
coderabbitai[bot] marked this conversation as resolved.
def resolve_image(path)
return unless path

Expand Down
Loading
Loading