Repository navigation
Prepare v4.6.6: query improvements and WAL concurrency fixes - #73
Merged
Merged
Conversation
Coalesce WAL header validation and admit large writers between checkpoint slices after covering their commit copy budget. Preserve the latest committed header and size each checkpoint for its captured database image. Validation: 3,125 CSharpDB.Tests passed and all 24 durable reopen checks passed. The local four-pair comparison improved 10,000-row throughput including drain by 16.9% and P99 by 63.6%. Single-row and 1,000-row P99 regressions remain unresolved.
Require captured provenance, complete comparison coverage, and nonzero execution even in report-only mode. Compare base and head in balanced order with a common harness, retain both artifact roots, and replace stale reports when evidence is invalid. Validation: all 42 guardrail contract tests passed. The reviewed benchmark Release build completed with zero warnings and errors.
Add bounded template plans with independent execution values for integer lookups and ranges. Preserve lookup shortcuts, allocate small-result batches lazily, and bound physical primary-key range traversal. Keep ReaderSession primary-key ranges on their existing path pending snapshot catalog qualification. Add regression coverage and reusable benchmarks; validate 3,197 engine and 313 ADO tests.
Capture committed roots, schema and row counts for reader sessions, with bounded metadata-change tracking and snapshot-backed fallback during publication. Protect base-file pages still visible to readers and retain commits appended after checkpoint planning. Retain regression coverage for catalog changes, concurrent writers and checkpoint repair. Reader and pooled prepared primary-key ranges remain disabled. Verified exact source hashes against the passing Release receipts: 3,247 engine tests and 315 ADO tests. Document diagnostic performance results and unresolved tail-latency qualification.
Use the reader's frozen catalog view to enable the existing prepared range planner for ReaderSession and pooled ADO queries. Cover snapshot root growth, commit/rollback visibility, and plan reuse across changing parameter bounds. Record the isolated 240,000-query local comparison and its qualification limits. The 65/2,048-parameter cases improved median latency by 73.38% geometrically; median, allocation and P99 checks passed. Verified Release results: 3,249 engine and 315 ADO tests passed, with current source hashes matching tested inputs.
Pass the requested upper bound to BTreeCursor as a prefetch hint. Skip the next-leaf read once the current leaf reaches that bound, while preserving cursor traversal, reseeking, and read-ahead across needed leaves. Add eight regression cases and document the paired local benchmark. Four-row pooled queries allocate 21.73% less and improve median latency by 23.15%; larger ranges pass the declared regression checks across 249,600 measured queries. Validation: 3,257 engine and 315 ADO Release tests passed. Verified the committed source matches the tested inputs and checked staged scope and whitespace.
Keep resource diagnostics on writer pagers and share a stateless read-only allocator across snapshots. Preserve faulted allocation/free operations, rejection messages, cancellation precedence, and writer diagnostics behavior. Add snapshot diagnostics, write-rejection, and retained-page regression tests. Document the 1,544-byte pager allocation saving and local pooled comparison: 7.75% fewer allocated bytes across four-row queries, with declared regression checks passing over 249,600 measured queries under normal background load. Validation: 3,260 engine and 315 ADO Release tests passed; source/runtime hashes, staged scope, and whitespace verified. Independent code and results review passed.
Initialize planner caches and diagnostics on demand, guard fast inserts against stale trigger metadata, and use bounded physical integer primary-key scans for ordinary SELECT and join predicates. Validated with 3,607 Release tests. Record paired performance results, including two original prepared-control flags and their separate passing follow-up.
Defer batch storage until batch access while preserving row reads, decode widths, reopen behavior, and retained-buffer ownership. Validated with 3,615 Release tests. Small expression/join allocation fell 12.89%; retain failed latency comparisons and the unresolved prepared-range slowdown without claiming latency qualification.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
11 of 15 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepare CSharpDB 4.6.6 with prepared-query reuse, bounded primary-key reads, lower query/write allocations, and stronger reader snapshot and storage ownership guarantees.
Type of Change
Related Issues
None linked.
Testing
dotnet build CSharpDB.slnxDbExceptionpaths)Local validation used Windows x64 and .NET SDK 10.0.401. No application/UI manual verification was performed for this PR preparation.
CheckpointConflictReadTestsafter the dependency updatesCSharpDB.Api.Testsafter the dependency updatesCSharpDB.Daemon.Testswith a corrected test-process PATHCSharpDB.Testsrun with the final production WAL fix, before the dependency updatesCSharpDB.Data.Testsrun, before the dependency updatesCSharpDB.Cli.Testsrun, before the dependency updatesCSHARPDB_ACCESS_LIVE_FIXTUREwas not configuredscripts/Test-Documentation.ps1after updating the release notesgit diff --checkThe earlier full core run included the first 19 WAL regression cases; four structural cases were added afterward and passed separately. All 23 were subsequently rerun together after the dependency updates. The full core/Data/CLI suites have not been rerun against the updated package graph.
Two deterministic retained-tail regressions failed against unchanged production code before the fix: one reproduced a short WAL read at offset 45352, and one read incorrect bytes from a stale offset that remained within file bounds. Both pass with the fix. A mutex-only candidate also exposed a pending legacy-flush failure; the final pending-commit guard addresses that case.
The initial daemon run had 33 failures because the inherited 10,623-character PATH prevented command-script helpers from finding PowerShell. A shorter PATH was applied only to the test process. The subsequent suite's remaining timeout was
Wrapper_ExecutesInterleavedSuitesAndRejectsDuplicateMedians; its isolated rerun passed without changing code or increasing the timeout. This is not a claim that the full daemon run was failure-free.Checklist
Notes for Reviewers
version.4.6.6branch relative tomain, including earlier query/write improvements and performance research, not only the latest WAL and dependency commits.CSharpDB.WalPrototype.*projects remain outside the product solution. They do not change the production database or WAL format and are not a shipping storage implementation.