[mkarlesky]

Summary

In whole-run fallback (no -fdirectives-only, or :test_build ↳ :preprocess_force_fallback), stage 4 used to keep every #include the text scan found. That scan ignores #if and can't read an #include whose name is a macro. So:

• a mock in a branch that is off stopped the build when its header was missing (Problems with including files depending on use_test_preprocessor value #1083 Case 1);
• a mock in a branch that is off replaced the real module, so a correct test failed;
• a mock named by a macro was never generated.

Stage 4 now runs the full preprocessor tool (plain -E) over each test file in fallback, with stand-ins on the search path. It then reconciles against the files the expansion's line markers name, the same way directives-only output is used.

Fallback stays a best effort with no guarantees, and code, comments, logs and docs say so. A test file whose expansion fails takes the existing per-file path: its text-scan #includes, checked against GCC's bare list.

Analysis behind this (prototype results on GCC 14 and Apple Clang 15): https://claude.ai/artifact/LqCm4jSTjkN77DnYdxdHCV

Changes

• Shared runner. Preprocessinator has one private run_preprocessor for its directives-only, fallback and Partial full-expansion runs. Two long methods now fit RuboCop's limits, and misplaced and stale comments are fixed.
• Rename. TestDirectivesOnlyPreprocessinator is now TestIncludesPreprocessinator. A frozen MODES table holds what differs in fallback: tool, output path, generator, testable key, and extra outputs. Caching and line-marker dependency tracking are shared.
• Output path. The fallback expansion is written to preprocess/files/<test>/fallback_expansion/ under the test file's own name, which the line-marker reader needs. Its path is recorded in testable.preprocess[:fallback_expansion]. Stage 11 and its [:directives_only] contract are untouched.
• Reconciliation. It runs in both modes. The first pass sets mock search paths in both modes, so the expansion finds the stand-ins.
• Messages. Startup notices and the new stage 4 warning call fallback best effort. Stage 4's sequencer drops an unused configurator collaborator.
• Docs. PIPELINE.md and preprocess/README.md are updated. They also correct pass 1's flags (-M -MG) and a stale file reference.

Tests (7-step test-first)

• Pinned first: the preprocessor command shapes, plus the fallback behavior that survives (stand-ins placed, a strict mis-cased mock in a branch that is on).
• Red, then green, at each step:
  • unit specs for the builder, the generator, the stage 1 key, the fallback second pass, and fallback reconciliation;
  • two of the new unit examples ("registers no further output" and "does not mark fresh on failure") passed vacuously before the change, since nothing ran in fallback.
• Integration (the main proof). spec/integration/test_includes_extractinator_spec.rb runs every scenario under both modes with shared expectations. That covers Case 1, the mis-cased mock, the macro-named mock, all three Case 2 variants, the computed #include, narrowing, stand-ins and the per-file failure. It adds the mixed real-and-mocked module and a check of where the fallback expansion lives. The ten fallback scenarios failed before reconciliation was enabled.
• System, kept minimal.
  • conditional_mock_includes: the two fallback examples now build instead of stopping. They were red on the prior commit.
  • delta_builds: expects the new warning.
• Runs:
  • units (4692) and integration (468) pass on Apple Clang 15, and integration passes on GCC 14 in Docker;
  • system specs pass in Docker (GCC 14): conditional_mock_includes, delta_builds, plugin_delta_builds, preprocessing_fallback_conditionals, preprocessing_locale, preprocessing_encoding, encoding_stress, partials_module_disambiguation, test_source_file_directive.
• RuboCop: backlog 840 → 838, with no stage 4 file in it. lint:changed is clean.

Notes for review

• One-time rebuild. Fallback projects rebuild once after upgrading: their search paths now carry reconciled mock directories, which changes later stages' meta.
• Cost. One extra -E per stale test file in fallback, matching pass 2's count in normal mode.
• Marker-stripping tools. A full preprocessor tool configured without line markers (-P) now affects fallback reconciliation as well as stage 11. That is the empty-list case tracked separately (report: https://claude.ai/artifact/TR6uCuBLbfdWz9oUNRPcVv).
• Release docs. Changelog, KnownIssues and mkdocs edits will follow as uncommitted diffs after CI.

🤖 Generated with Claude Code