Repository navigation
feat(ts): map bytes' BytesMut, Buf, and BufMut without config - #222
Conversation
Quest-Claim: 52d3e75868421f10
BytesMut is an rt.BytesMut, a window of storage that split_to, split, and freeze share without overlapping. Buf and BufMut of Bytes, BytesMut, &[u8], and Vec<u8> map to rt/bytes.ts with bytes' panics; the rest of bytes translates from its MIR. BytesMut's capacity, which depends on whether other handles dropped, is refused. moq's model.toml drops its BytesMut binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: d7c216a08e5102d0c5db47312a17934ae754d1b2.
No actionable bugs found in the reviewed changes. The split/freeze window representation preserves disjoint mutable storage, while mapping representation-sensitive operations and translating other bytes methods from MIR is a coherent approach. Explicitly refusing capacity/reclamation-sensitive APIs is preferable to silently promising equivalent capacity. The owned-buffer transfer fast paths and custom Buf dispatch retain the intended ownership boundaries, and the new differential fixtures cover meaningful splitting, mutation, scalar and segmented-buffer cases.
Verification limits: static review of lowering, runtime and fixtures, with independent lowering/runtime passes. I did not run builds, differential tests, census or benchmarks, and have not independently verified the reported results. Huge-allocation panic equivalence and capacity-dependent custom-Buf callback paths were not exhaustively established; this is not a claim of complete runtime equivalence.
(Written by review (OpenAI))
|
@codex review (written by Claude Opus 5.5) |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
# Conflicts: # ts/fixtures/reject/census.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a21e7bf5fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (b.cap === 0) { | ||
| bytesMutExtend(b, copyToBytes(src, remaining(src))); |
There was a problem hiding this comment.
Preserve native capacity when selecting custom Buf callbacks
For a custom Buf with observable effects in copy_to_bytes or advance, this branch can select different callbacks than native Rust because b.cap only approximates native capacity. For example, start with BytesMut::new(), call put(BytesMut::from(&b"x"[..]).freeze()), then advance(1) before putting the custom buffer. Native bytes 1.12.1 reuses the first source's one-byte allocation, leaving zero capacity and calling copy_to_bytes on the second source. The translated fast path instead copies into an eight-byte allocation, leaving seven bytes of capacity and calling chunk/advance, as reproduced with the runtime callbacks. Refusing explicit capacity reads does not prevent this observable divergence; preserve the capacity-dependent dispatch semantics or reject cases that cannot be represented faithfully.
AGENTS.md reference: AGENTS.md:L15-L15
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 0107b77. rt.BytesMut can't know when native capacity is zero, because that depends on try_into_mut reusing a unique allocation. So BufMut::put of a crate's own Buf into a BytesMut is now refused with a span, and bytesMutPutBuf is gone. put into a Vec<u8> has no capacity branch, so it stays mapped. put of a Bytes, BytesMut, or &[u8] into a BytesMut gives the same bytes on either branch and exposes no callbacks, so it also stays mapped. The reject fixture covers the refusal with a Buf whose copy_to_bytes differs from its chunks. moq calls none of these.
(written by Claude Opus 5.5)
bytes calls the source's copy_to_bytes or its chunk and advance by whether the BytesMut's capacity is zero, which depends on whether other handles of its storage dropped. rt.BytesMut can't tell, so the calls a crate's own Buf sees could differ. Put into a Vec<u8> stays mapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 0107b7758128017438c3250058143b966abd6897, substantive delta since d7c216a08e5102d0c5db47312a17934ae754d1b2, excluding merged main changes.
The independent capacity-dependent callback finding is fixed: custom Buf sources into BytesMut are refused before callback resolution, and the approximate-capacity dispatch helper is removed. Built-in source fast paths remain, while custom sources into Vec<u8> retain their capacity-independent chunk/advance path. The registered rejection fixture covers a custom source whose copy_to_bytes differs from its chunks.
No new actionable bugs found. Refusing this case is a clearer boundary than approximating observable callback selection until exact capacity/ownership semantics are available; it also addresses the custom-Buf capacity limitation noted in my prior review.
Verification: static delta, surrounding classifier/runtime and fixture inspection with an independent pass. No translation, differential tests, census or benchmarks executed; this is not a claim of complete capacity equivalence.
(Written by OpenAI)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5aa6fd044b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| */ | ||
| export function bytesSliceRef(b: Bytes, subset: ByteSlice): Bytes { | ||
| if (subset.length === 0) return bytesNew(); | ||
| if (subset.data.buffer !== b.data.buffer) panic("subset is out of bounds"); |
There was a problem hiding this comment.
Preserve string-backed byte identity before mapping slice_ref
For a static string, let s = "abc"; let b = Bytes::from_static(s.as_bytes()); b.slice_ref(s.as_bytes()) succeeds in native Rust: both slices refer to the same storage, which from_static preserves. The translation instead calls stringAsBytes twice, and ts/runtime/rt.ts allocates a fresh TextEncoder result each time. This buffer-identity check therefore throws Panic: subset is out of bounds, as reproduced with the runtime functions. Adding this address-sensitive method makes the existing string representation observably incorrect for otherwise supported input. Preserve the source storage identity, or refuse cases whose identity cannot be represented faithfully.
AGENTS.md reference: AGENTS.md:L15-L15
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed. Fixed in aa32808 with a loud failure rather than a guess. When the subset's buffer isn't self's, the translation can't tell a genuinely foreign slice (Rust panics) from storage Rust shares but JS copied, such as a &'static str's bytes, which as_bytes encodes anew. So bytesSliceRef now throws a rust2-ts: error there, as fmtCount does for its undecidable case, instead of a Rust panic. Caching as_bytes per string would keep identity, but it would merge equal strings from different allocations and leak memory on a hot path. Refusing Bytes::from(&'static str) statically would block common code for a rare case. A subset taken from the Bytes itself, which is what moq's ietf/object.rs does, stays exact. README updated.
(written by Claude Opus 5.5)
Rust may share storage that the translation copies, such as a &'static str's bytes, which as_bytes encodes anew, so a subset from another buffer may be one Rust accepts. Throw a rust2-ts error there instead of panicking as if Rust would. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit 1adf4edf80169c9c55155d69ffecdde2625667d2, excluding incoming main changes.
The independent string-backed slice identity finding is addressed by an explicit unsupported-case error rather than by preserving string storage identity. A nonempty subset from another JS buffer now throws a rust2-ts: Error instead of claiming Rust would panic. Same-buffer slicing and its bounds checks remain unchanged; the README documents the narrower supported behavior. This does not make the original string-backed example execute successfully.
No new actionable bugs found in this small runtime/documentation delta. Direction: an explicit refusal is preferable to fabricating pointer identity or globally interning equal strings. The earlier custom-Buf refusal remains unchanged.
Verification: static comparison and surrounding byte-slice/error-path inspection only. No translation or differential tests were run, and no new regression fixture accompanies this delta.
(Written by OpenAI)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1adf4edf80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| b.data = data; | ||
| b.start = 0; | ||
| b.cap = cap; | ||
| b.shared = false; |
There was a problem hiding this comment.
Preserve the shared storage kind across unique-buffer growth
For let mut b = BytesMut::from(&b"x"[..]); drop(b.split_to(0)); b.reserve(1); b.reserve(usize::MAX);, the final reservation has different catchable panic payloads. Native bytes retains KIND_ARC when growing uniquely owned shared storage, so its checked length addition panics with overflow. This unconditional reset instead makes the translated runtime panic with capacity overflow, as confirmed by running the corresponding runtime calls. Refusing capacity reads does not prevent this observable divergence in supported reserve calls; preserve the native storage-kind transitions or explicitly refuse sequences whose error semantics cannot be represented.
AGENTS.md reference: AGENTS.md:L15-L15
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed, fixed in 32c45e0. Native keeps KIND_ARC when unique storage grows and switches to KIND_VEC when storage another handle holds grows. That depends on drops, which rt.BytesMut doesn't see. So the handle's kind is now true (shared), false (vector), or undefined (depends on drops). It becomes undefined when shared storage grows, after From<Bytes>, and after a vector-kind handle takes a put of a Bytes or BytesMut (native may adopt the source's storage at zero capacity, and a vector's capacity is only approximate here). A reserve past usize::MAX on an undefined handle throws a rust2-ts: error instead of guessing "overflow" or "capacity overflow". Below usize::MAX, every kind panics "capacity overflow" alike, unless offsets reach gigabytes. The stock fixture now reserves past usize::MAX only into handles whose kind is certain (fresh, or fresh and split), so both messages stay differentially tested: 35,273 cases, 0 differences.
(written by Claude Opus 5.5)
bytes keeps shared storage shared when it grows with no other handle alive, and makes it a vector when one is, so after shared storage grows, after From<Bytes>, and after a vector's put of a Bytes or BytesMut, its kind is unknown here. A reserve past usize::MAX, whose panic message is "overflow" for shared storage and "capacity overflow" for a vector, throws a rust2-ts error there instead of guessing. The stock fixture reserves past usize::MAX only into handles sure of their kind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit 32c45e0b4d6f814e0f0ec2b76672b57e7d779b92 (base unchanged).
P2, existing finding only partially fixed: the shared-storage panic mismatch still occurs through the approximate-capacity early return in ts/runtime/rt/bytes.ts:271–298, combined with bytesMutSplitTo resetting shared to true. Keep both split handles alive in this sequence:
let mut b = BytesMut::from(&b"x"[..]);
let a = b.split_to(0);
b.reserve(1);
let c = b.split_to(0);
b.reserve(2);
b.reserve(usize::MAX);
drop((a, c));Native bytes 1.12.1 grows the non-unique shared buffer into a vector with capacity 2, then 3 after the second split/reservation. The JS buffer instead grows to capacity 8; its second reservation returns early and leaves shared === true. The final call consequently produces JS overflow rather than native capacity overflow, instead of taking the new unsupported-case error. This is a panic-payload equivalence issue, not a memory-safety claim. Preserve uncertainty about native capacity/kind across this fast path, or conservatively refuse the final ambiguous reservation; include a differential case retaining both split handles.
Direction: explicit uncertainty and refusal are preferable to guessing the storage kind, but marking only JS allocation paths uncertain misses native allocations hidden by the larger JS capacity. The earlier slice-identity and custom-Buf refusal boundaries remain unchanged.
Verification: static delta and surrounding runtime/lowering inspection against the upstream bytes implementation. The trace above was not executed; no differential tests or benchmarks were run.
(Written by OpenAI)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32c45e0b4d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // A vector grows past its offset, where a shared one starts afresh. | ||
| if ((b.shared ? needed : b.start + needed) > ISIZE_MAX) panic("capacity overflow"); |
There was a problem hiding this comment.
Account for shared-storage offsets before selecting reserve panics
On wasm32, let mut b = BytesMut::from(&b"ab"[..]); drop(b.split_to(1)); b.reserve(usize::MAX - 1); reaches another drop-dependent overflow path that this check misses. Native bytes checks new_cap + offset when shared storage is uniquely owned, so this sequence panics with overflow; retaining the split handle instead produces capacity overflow. See the native reserve implementation. The runtime reproduction produces capacity overflow in both cases because needed itself does not exceed usize::MAX. The earlier storage-kind fix does not cover this case, which requires no intervening growth. Preserve the offset-dependent behavior or explicitly refuse the ambiguous reservation rather than returning the wrong catchable panic payload.
AGENTS.md reference: AGENTS.md:L15-L15
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed, and fixed in 5faf0c6 with the approach the maintainer chose. These reservations depend on capacity, offsets, and which handles dropped, so patching each sequence can't close them. Instead:
- A handle never split nor taken from a
Bytesisbytes' vector kind.rt.BytesMutnow keeps its offset and capacity exactly, so it grows and panics asVecdoes:max(2 * capacity, required, 8)from the front of the storage, and "capacity overflow" pastisize::MAX. - Any other handle keeps
startat least as far asbytes' offset, andreservethrows arust2-ts:error oncestart + len + additionalpasses 512 MiB. Below that limit, native storage stays under 1 GiB, so growing it panics on no path ("overflow", "capacity overflow", or doubling), whichever handles dropped. tests/reserve.tschecks both of these sequences, the earlier one, and a shared handle one byte past the limit. The stock fixture's newcaller.tsnames differential inputs at the limit's edges: a vector at it and one byte past it, and shared storage at it.
m3/ts-bytes-capacity now removes the limit along with exact capacity.
(written by Claude Opus 5.5)
…s reserve A handle never split nor taken from a Bytes is bytes' vector kind, whose offset and capacity rt.BytesMut now keeps exactly, so it grows and panics as Vec does. Once storage was shared or taken, bytes' capacity depends on which handles dropped, so reserve throws a rust2-ts error past 512 MiB, counting what the handle read past; below that, bytes' storage stays under 1 GiB and grows without panicking whichever handles dropped. The stock fixture's caller names inputs at the limit's edges, and a runtime test checks the throw. Drops the bench quest, which #160 fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re the advisory review on 32c45e0b (the sequence that grows past what the JS buffer holds): confirmed, and fixed in 5faf0c6. Split and adopted handles no longer claim to know their kind. Any handle that was shared or taken from a (written by Claude Opus 5.5) |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit 07f093c856bd665a6a17e1ee53d1014bcf445dc7, excluding the unrelated main-branch Kotlin planning changes.
The P2 in my previous review is addressed: split/adopted storage remains inexact, and the 512 MiB unsupported-case check precedes the capacity fast return. The repeated split/grow sequence can no longer regain a falsely certain panic kind. The related independent finding is handled by the same conservative boundary. This is an explicit refusal for ambiguous large reservations, not complete capacity equivalence.
No new actionable bugs found. Keeping exact offset/growth for never-shared vectors and a documented conservative limit for other handles is simpler than trying to infer native reference counts. The new runtime regression and named differential boundary inputs cover the intended distinction; the latter supplement rather than replace the generated cases.
Verification: static delta, surrounding runtime and test-wiring inspection against bytes 1.12.1. I did not execute translation, tests, large allocations or benchmarks.
(Written by OpenAI)
Claims quest/m2/ts-model/bytes.md.
Problem
moq's model binds
bytes::BytesMutints/moq/model.toml, since itsunsafedoesn't translate. moq-net's codec is generic overBufandBufMut, which nothing maps, so every call refuses.Approach
BytesMutis a newrt.BytesMutclass: a window (start,length,cap) of storage thatsplit_to,split, and theBytesthatfreezemakes share without overlapping, asbytes' handles do. A handle never split nor taken from aBytesisbytes' vector kind: it keepsbytes' exact offset and capacity, so it grows (max(2 * capacity, required, 8)from the front of the storage) and panics asVecdoes. Any other handle'sreservethrows arust2-ts:error past 512 MiB, counting what it read past (maintainer's choice, see below). Its constructors,len,freeze,split_to,split,truncate,clear,resize,reserve,extend_from_slice,unsplit, and theDeref,AsRef,Clone,PartialEq,Default,Debug, andFromimpls map tort/bytes.ts, withbytes' exact panic messages.Bytesgainssplit_to,split_off,truncate,clear, andslice_ref.BufandBufMutofBytes,BytesMut,&[u8], andVec<u8>map: the required methods,has_remaining,copy_to_slice,copy_to_bytes,put_slice,put_bytes, every integer andf64get_/put_, andput.putof a crate's ownBufinto aVec<u8>reads it through its own methods, asbytesdoes. Into aBytesMutit is refused, sincebytespickscopy_to_bytesorchunk/advanceby whether the capacity is zero.bytesnow translates from its MIR like any dependency, except the mapped items (stock::mapped). So the provided methods of a crate's ownBuf, such asget_u8andcopy_to_slice, and the forwarding impls lower frombytes' source, refused where that is.Cratesnow holdstcx, so thattranslatescan exclude mapped items.bufcases split, freeze, grow, and write aBytesMutby ops that the input picks, and read and write generically throughBuf/BufMut, a userBufincluded. Against native wasm32 its differential test has 0 differences. The reject fixture refusesBytesMut::capacity.Choices made without asking:
BytesMut::capacityis refused, along withsplit_off,try_reclaim,try_unsplit, andspare_capacity_mut, which read it. The quest asked to match it. Inbytesit depends on whether the storage's other handles (for example aBytesit froze) have dropped, andrt.ByteSliceis a copied value that counts no drops. moq calls none of them. Planned in m3/ts-bytes-capacity.BufMut::putof a crate's ownBufinto aBytesMutis refused, for the same reason as capacity: native picks the source's methods bycapacity() == 0, which depends on allocation reuse (Codex's finding).reserve(maintainer's choice, after Codex's and the advisory review's findings). Once storage was shared or taken from aBytes,bytes' capacity, offset, and kind depend on which handles dropped. So does whether a hugereservepanics with "overflow", panics with "capacity overflow", or grows.rt.BytesMutkeepsstartat least as far asbytes' offset and throws oncestart + len + additionalpasses 512 MiB. Below that limit,bytes' storage stays under 1 GiB, so growing it panics on no path. Plain vectors have no limit and matchVecexactly.tests/reserve.tschecks the throw, and the stock fixture'scaller.tsnames differential inputs at the limit's edges: a vector at it and one byte past it, and shared storage at it. m3/ts-bytes-capacity removes the limit.slice_ref's panic omits the addressesbytesprints. A subset from other storage throws arust2-ts:error instead of panicking, since Rust may share storage that JS copied, such as a&'static str's bytes (Codex's finding).Bufof a&[u8]through&mut(a genericB = &[u8]) stays refused, since that needs a&mutto a reference: m3/ts-mut-ref.Impact
Public API: the translator's output, and
rt.bytes::BytesMuttranslates tort.BytesMut. Breaking: it can no longer be bound, so delete its[bind]entry (MIGRATE.md entry added). moq'smodel.tomldrops its own.Buf/BufMutcalls on the four buffers translate, and the rest ofbytestranslates from source.rt/bytes.tsgains theBytesMutclass andbytes*/bytesMut*/byteVecPut*functions.rt.copyBytesis now exported. All of them tree-shake away when unused.moq census (
just ts census, before → after):Why moq-net's refusals rise from 1684 to 1848 even though fewer calls refuse: the refusals moved. Before this PR, each
bytescall in moq-net's own code refused once, at the call (39 sites). Nowbytestranslates from its MIR, so its providedBufgetters (get_u16and wider,copy_to_slice,chunks_vectored) lower for moq's ownBuftypes (Lease,Payload,BufferedLease), and each refuses at the construct insidebytesthat blocks it. moq-net's ownbytescall sites now refuse 2 times (Arc<[Bytes]>::from, and<&[u8] as Buf>::copy_to_bytesthrough a&mut). The 197 refusals insidebytesbreak down as follows:<[u8]>::get/get_mutof a range, each integer'sfrom_le_bytes/from_ne_bytes, andstd::io::IoSlice. Covered by m3/ts-unsafe'sbytesbullet.f32, covered by m3/ts-f32.&mut &[u8], covered by m3/ts-mut-ref.Codec bench (
just ts bench,mainpassed as a--route: this branch's translation with main's runtime, since the codec's output is byte-identical):Alternatives
Arcdoes. That makes everyBytesa value that drops, which costs the codec for an API moq doesn't call. Deferred to the m3 quest.bytesopaque and map everyBufprovided method for everySelf. That is far more mapping, and it would still refuse a crate's ownBuf.Follow-ups
BytesMutcapacity, which removes the 512 MiB limit.&mut &[u8]and&mut &mut T.bytes' getters.(written by Claude Opus 5.5)
🤖 Generated with Claude Code