Skip to content

[4.2-dev] fix(plan): admit stable IN predicates for ZoneMap filtering - #32790

Draft
ck89119 wants to merge 3 commits into
matrixorigin:4.2-devfrom
ck89119:issue-21333-in-zonemap-4.2-dev
Draft

ck89119 wants to merge 3 commits into
matrixorigin:4.2-devfrom
ck89119:issue-21333-in-zonemap-4.2-dev

Conversation

@ck89119

@ck89119 ck89119 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

Related to matrixorigin/matrixflow#21333.

What this PR does / why we need it:

Stable non-primary-column IN predicates can remain row filters without becoming block filters when statistics are unavailable or estimate no pruning benefit. The two incident CTAS queries use such a period predicate.

Admit direct-column IN with a runtime-constant List/Vec RHS through the existing simple ZoneMap filter path on 4.2-dev. Preserve this branch's argument validation, constant-expression and zonemappability checks, residual row filtering, disable hint, and existing-filter preservation/deduplication. CTAS transaction visibility and execution topology are unchanged. This is the 4.2-dev counterpart of #32786, adapted and validated independently.

Also fix sorted-column IN seek skipping blocks with uninitialized summaries. A binary search cannot safely treat unknown summaries as nonmatches (or assume a monotone predicate with holes). Seek now skips only a contiguous prefix proven below the IN lower bound and stops at the first unknown or potentially matching block. Unsorted/float/empty/nullable-list seek eligibility is unchanged; other operators and storage formats are untouched.

Close the review-reported binary-value regression exposed by the new admission: a value longer than 30 bytes with an all-FF 30-byte prefix has a persisted MaxTruncated upper bound. Byte/string AnyIn must not reject a candidate using that incomplete bound, and SubVecIn must not shorten the BloomFilter candidate window with it. Both now retain the existing lower-bound pruning while treating that upper bound conservatively. No persisted-format change or IN-admission rollback is involved.

Validation with Go 1.26.4 on darwin/arm64:

  • Planner and seek regressions fail before their fixes and pass afterward, including PK/cluster-by, unknown first/middle/last summaries and actual selected-block IDs.
  • The binary regression uses real parse/bind, row evaluation, object writing and metadata reload: the previous PR head evaluates the row as a match but retains 0 blocks instead of the unfiltered control's 1; after the fix it retains 1. Short binary and long ASCII controls pass. ZoneMap tests cover all eight byte/string types, 29/30/31/32-byte boundaries, serialization, NULL/empty/nullable lists, and multi-item BloomFilter windows.
  • Full pkg/sql/plan, pkg/vm/engine/readutil and pkg/vm/engine/tae/index tests pass. Build, focused vet and golangci-lint 2.6.2 pass. Added seek and truncated-bound statements are 100% covered; the admission helper has 93.8% statement coverage.
  • The mo-tester-generated/reviewed BVT case passes twice on the final binary: 164/164 each time, with successful teardown. Coverage includes persisted binary IN, flush-before/after results, NULLs/duplicates, disable controls, prepared-statement reuse, aggregation/CTAS, and scan counters. Unknown-summary seek is injected in reader UT rather than corrupted through SQL.
  • Both complete incident CTAS queries were rerun locally with the original schema and 24 synthetic rows. Enabled/disabled block-filter controls produce identical business results and column metadata; each source scan decreases from 3 blocks/24 rows to 2 blocks/16 rows.

Cost/limits: sorted IN seek changes from O(log B) to O(B) metadata checks in the worst case, with no additional data I/O or allocations. A local microbenchmark (three 100ms runs) measured late/no-match seek at approximately 1.7 us for 128 blocks and 51–52 us for 4096 blocks, versus approximately 0.1/0.16 us previously; early matches take approximately 14 ns. This is an explicit correctness-first CPU tradeoff, not a claim of unchanged seek performance. Small SQL fixtures establish correctness and effective pruning, not production-scale latency improvement; production acceptance remains pending.

This branch is being deployed

1 in progress deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working size/L Denotes a PR that changes [500,999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants