Skip to content

filesystem: diagnose ACK accounting mismatches - #1343

Merged
bootjp merged 11 commits into
mainfrom
investigate/fuse-ack-accounting
Oct 27, 2026
Merged

bootjp merged 11 commits into
mainfrom
investigate/fuse-ack-accounting

Conversation

@bootjp

@bootjp bootjp commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Preserve record and session validation causes and include numeric ACK accounting state in invariant errors.
  • Add retained accounting tests over 1,280 operations, forced concurrent ACK reads, and a generation fence versus ACK race on memory, Pebble, and single-node Raft/Pebble.

Reproduction

The strict SQLite Jepsen workload reproduced the ACK accounting violation in run 40458354386 with 512 keys, rate 0.75, concurrency 8, and a five-minute workload. The job concluded success, but its uploaded n3 server log contains 3,109 ACK accounting warnings. The first is at 06:03:33 UTC, sequence 209: retained bytes 5,386, reservation 5,514, generation 5. The deficit is exactly 128 bytes, equal to one generation fence reservation. Later warnings retain the same 128-byte deficit, including operations at generation 1.

Run 40459386710 with 128 keys also concluded success; its server log showed quota warnings but no ACK accounting violation. Longer runs remain in progress.

Investigation status

The failure is reproduced, but the first write that loses 128 bytes and the root cause are not yet identified. A controlled fence/ACK race passed 30 runs on all three backends; that race alone does not explain the observed mismatch. This PR adds diagnostics and coverage but does not repair the accounting invariant. A green Jepsen checker cannot currently establish that the server-side ACK invariant held.

Verification

  • Targeted filesystem journal tests and the concurrent ACK race-detector test passed.
  • The controlled fence/ACK test passed 30 times, including single-node Raft/Pebble.
  • Changed-code lint: 0 issues.

Risk

Diagnostic errors include numeric session counters and sequence numbers. The observed accounting violation still requires a root-cause fix.

Author: bootjp

Summary by CodeRabbit

  • 改善
    • 操作ジャーナルや操作レシートの検証に失敗した際、原因や関連情報がエラーメッセージに表示されるようになりました。問題の特定に役立ちます。

Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Walkthrough

操作ジャーナルの破損エラーに、失敗条件や関連する操作・会計値を追加しました。メモリ、Pebble、Raft-Pebble 構成で保持会計と並行 ACK を検証するテストを追加しました。

Changes: 操作ジャーナルの検証と会計

Layer / File(s) Summary
レコードとセッションの検証
internal/filesystem/operation_record.go, internal/filesystem/operation_session.go, internal/filesystem/operation_journal.go
デコード失敗、ID 不一致、状態や会計値の検証失敗を区別し、関連情報を含む破損エラーを返すようにしました。
ACK 処理の破損エラー情報
internal/filesystem/operation_journal_exec.go, internal/filesystem/operation_receipt.go
保留スナップショット、ACK 保持量の予約、未コミットの操作 receipt に関する破損エラーに、操作や会計の状態を示す値を加えました。
保持会計と並行 ACK のテスト
internal/filesystem/operation_journal_accounting_test.go
メモリ、Pebble、Raft-Pebble 構成で、並行 ACK、世代フェンスとの競合、シーケンス境界をまたぐ処理後の保持会計を検証します。

Priority: Not assessed

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Possibly related PRs

  • bootjp/elastickv#1335: 操作ジャーナル本体を追加した PR です。この変更は同じレコード、セッション、ACK 処理に診断情報を加えています。
  • bootjp/elastickv#1341: 操作レコード形式と世代フェンスの予約量を変更した PR です。この変更も同じ予約量と検証処理を扱っています。
Merge Risk: Minimal · up to 08a96

No actionable risk introduced by these diagnostic and test changes is established. The underlying ACK accounting mismatch still needs separate investigation.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
Signed-off-by: bootjp <contact@bootjp.me>
@bootjp
bootjp marked this pull request as ready for review October 27, 2026 11:06
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T11:07:13.731992Z 08a9618 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-27T11:10:54.910414Z 08a9618 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the agent-review-approved Current HEAD has a completed code review with zero findings label Oct 27, 2026
@bootjp
bootjp merged commit 64fc9e6 into main Oct 27, 2026
13 checks passed
@bootjp
bootjp deleted the investigate/fuse-ack-accounting branch October 27, 2026 11:11
@github-actions github-actions Bot removed the agent-review-approved Current HEAD has a completed code review with zero findings label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant