Fix non-sequential _id gaps on mixed upsert batches

[luccasmmg]

Summary

• Client-reported issue: after upserting a batch that mixes already-present (unchanged) rows with new rows under a composite primary key, _id values for the newly inserted rows were non-sequential, with permanent gaps.
• Root cause: merge_sql()'s USING subquery numbered every incoming row with a single ROW_NUMBER() OVER () spanning the whole batch, but _id = MAX(_id) + _rn is only ever applied on WHEN NOT MATCHED. Unchanged matched rows (no-op'd via the IS DISTINCT FROM diff guard) still consumed _rn slots that nothing ever inserted, so subsequent new rows jumped ahead by however many unchanged rows preceded them in that chunk.
• Fix: partition the ROW_NUMBER() window on whether each row's primary key is already present in the target table (via a LEFT JOIN self-probe on the PK), so _rn restarts at 1 within the "genuinely new" partition — independent of how many unchanged rows are interleaved in the same batch. _id assignment for WHEN NOT MATCHED rows stays MAX(_id) + _rn, now gap-free.
• WHEN MATCHED semantics (diff-gated update, _updated_at only bumps on real changes) are unchanged.

Test plan

• Updated test_merge_sql_renders_typed_extractors_on_match_update_no_match_insert to assert the new partitioned ROW_NUMBER() + self-join shape.
• Added test_merge_sql_numbers_unmatched_rows_within_their_own_partition as a regression test documenting the gap-avoidance intent.
• python3 -m pytest tests/engines/bigquery/test_tables.py — 88 passed.
• Full suite python3 -m pytest — 496 passed.

Note: adds one extra self-join (target table probed on the PK) per upsert chunk on top of the existing MERGE's own join — worth checking PK-column clustering/partitioning on large resource tables if this shows up in BigQuery cost/latency.

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2f2009a4-d578-4abb-8f4a-33d2f1b3f6c7

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sagargg]

LGTM

[sagargg]

@luccasmmg i merged this but i guess we also need to fix on the aircan DAG isn't it ?