Skip to content

fix: preserve text delta chunks and UTF-8 insertion order - #1230

Merged
lodystage[bot] merged 3 commits into
mainfrom
fix-text-delta-truncation
Oct 23, 2026
Merged

lodystage[bot] merged 3 commits into
mainfrom
fix-text-delta-truncation

Conversation

@zxch3n

@zxch3n zxch3n commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Long strings inserted through loro_delta::text_delta::TextDelta silently lost their final chunk: 129, 256, and 257 ASCII bytes became 128, 128, and 256 bytes. Middle insertion could also panic when a UTF-8 character crossed chunk capacity. These chunking defects have existed since DeltaRope was introduced in #327.

Share byte-capacity, UTF-8-aware chunking between long appends and middle insertion, including the final slice. Preserve the short-string fast path and attributes on every chunk.

Partial UTF-8 chunks expose a B-tree batch ordering defect: inserting 127 x characters plus é between the b and c in ordinary a followed by bold bc can move the x characters before b. Fix DeltaRope::insert_values in the published crate itself: buffer multiple items and insert back to front using the returned start cursor from single-item insertion. Empty and single-item inputs avoid batch allocation. This works with registry generic-btree 0.10.7, without requiring the workspace patch or a new B-tree release.

Also fix the in-tree B-tree's adjacent-left-leaf selection and end cursor after merging a batch prefix. The vendored fork is excluded from publishing, so these changes are tested separately and are not the basis for downstream correctness.

Add scripts/test-delta-published-deps.sh to CI. It creates the actual .crate archive, extracts it outside the workspace, pins registry generic-btree to the minimum supported 0.10.7, and runs all packaged tests independently of [patch.crates-io].

Regression coverage includes length boundaries; 2/3/4-byte UTF-8 characters; append and middle insertion; preservation of surrounding attributes; 1,000 sequential UTF-8 edits compared against String; B-tree leaf/parent boundaries, caches, and subsequent insertion. Include a Rust-only changeset and document the insertion and publishing invariants in context/text-delta-chunking.md.

Validation:

  • Confirmed the attributed middle-insertion regression fails before the fix.
  • Confirmed the B-tree leaf-end regression detects a stale cursor if only the adjacent-leaf fix is applied.
  • bash scripts/test-delta-published-deps.sh --offline: all 44 packaged tests passed with registry generic-btree 0.10.7, including all 22 text integration tests.
  • cargo test --offline -p generic-btree -p loro-delta: 70 passed, 3 existing ignored tests; doctests passed.
  • cargo clippy --offline -p generic-btree -p loro-delta --all-targets -- -D warnings: passed.
  • Rustfmt, Bash syntax, and git diff --check: passed.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

WASM Size Report

  • Original size: 3502.46 KB
  • Gzipped size: 1168.69 KB
  • Brotli size: 819.70 KB

See context/text-delta-chunking.md for the adjacent-leaf and end-cursor invariants.
@zxch3n zxch3n changed the title fix: preserve all text delta chunks at UTF-8 boundaries fix: preserve text delta chunks and UTF-8 insertion order Oct 23, 2026
Test the packaged crate against registry generic-btree 0.10.7. See context/text-delta-chunking.md for the workspace patch boundary.
@lodystage
lodystage Bot merged commit 75e4bf2 into main Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant