Skip to content

fix(ffi)!: export byte payloads as bytes - #197

Merged
kixelated merged 9 commits into
mainfrom
quest/m1/ffi-export/payload-bytes
Oct 29, 2026
Merged

kixelated merged 9 commits into
mainfrom
quest/m1/ffi-export/payload-bytes

Conversation

@kixelated

@kixelated kixelated commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

moq's frame payloads exported as String: C and C++ aborted on a payload that is not UTF-8, and Kotlin's writeFrame took a String. moq's media payloads are binary.

Root cause: choose read only a parameter's direct bounds. moq writes neither impl Into<Bytes> nor impl AsRef<[u8]>:

  • write_frame<B: IntoBytes>: the bytes meaning sits in the supertrait IntoBytes: AsRef<[u8]>.
  • binary::Snapshot::update(impl Into<Timed<Bytes>>): Timed<Bytes> doesn't map, and the bytes meaning sits in impl<B: Into<Bytes>> From<B> for Timed<Bytes>.

Either way no target ranked first, so the default String won, since String meets both bounds.

Approach

choose now reads what the bounds imply:

  • Supertraits: it elaborates the bounds, so IntoBytes: AsRef<[u8]> yields &[u8].
  • Owned fallback: a str or [T] target ranks &X, then its owned form (String or Vec<T>). That covers a bound with no borrowed impl and a 'static parameter, both of which used to fall to String.
  • From impls: for an Into<X> whose X doesn't map, it reads the From impls into X whose source is a type parameter, and takes that parameter's Into or AsRef targets.
  • Implied lifetimes: whether a parameter must outlive the call, so can't borrow what C lends, is read from the elaborated bounds too, both the item's and a From impl's source. Payload: AsRef<[u8]> + 'static takes Vec<u8>, not &[u8], and so does a From source when the parameter itself is 'static.

The default order (String, Bytes, Vec<u8>) is unchanged. I first tried ranking bytes ahead of String globally, but that regressed the census: generic instances like json::Snapshot<T: Serialize> need the String placeholder.

Results at moq c764ace, the pin before #196 (at #196's pin 30f25c0, binary::Snapshot::update and binary::Stream::append no longer export, and just ffi moq passes):

  • Census: identical before and after.
  • Changed parameters: 15, all byte payloads: both write_frames, append_datagram, insert_datagram, frame::Frame::new, frame::Producer::write, 7 codec decodes, binary::Snapshot::update, and binary::Stream::append. No text parameter changed.
  • Glue: 13 of them now borrow the C buffer directly instead of validating and copying it into a String. The two Timed ones copy it once into Bytes.

Tests: the api fixture adds hex(impl Payload), shaped like IntoBytes but 'static and implemented for &'static [u8], and timed_hex(impl Into<Timed<Bytes>> + 'static), whose From takes B: AsRef<[u8]>. moq's B: Into<Bytes> shape is covered by the nightly just ffi moq. Native, C, C++, and Kotlin all pass ff 00 61 and must produce the same transcript. Before the fix, both aborted with a string is not UTF-8. Without the lifetime reading, the glue fails to compile (p0 does not live long enough). In moq's transcripts, the last frame is now \xffworld in C, C++, and Kotlin.

Checks: just check, just test, and just ffi moq (C, C++, Kotlin) pass after merging origin/main (#189, #190, #196, #200, #205, #206). just ffi census is identical to origin/main's, and the generated export differs from main's only in those 15 parameters.

Impact

  • C: no signature changes (the parameters were already moq_buf). A payload that is not UTF-8 no longer aborts.
  • C++: no signature changes (view). Same behavior fix.
  • Kotlin (breaking): the 15 parameters above change from String to ByteArray. Entry added to ffi/MIGRATE.md.
  • Config: the api fixture excludes Payload (it consumes itself, so it can't be a callback) and Timed (no instance exports), as moq's config excludes IntoBytes and Timed.

Alternatives

  • Rank Bytes and Vec<u8> ahead of String by default. It's simpler but wrong: it regressed Serialize-bounded instances (json::Snapshot<String> became json::Snapshot<Bytes>, which is refused).
  • Rank the type arguments of an unmapped Into target (Timed<Bytes> → Bytes). It's shorter, but it's a heuristic, where reading the From impl reads the actual conversion.

Follow-ups

  • The gate fixture covers a 'static parameter through a From source, not a 'static bound on the source itself. That case is the same || and is checked by inspection.
  • each still reads only direct outlives bounds when it picks &X for an implementor. That predates this PR and isn't changed here.
  • sources reads one level of From impls. A chain such as Into<A> where A: From<B> and B: Into<C> isn't followed. moq doesn't need that today.

(written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 4 commits October 28, 2026 16:07
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed Into target

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Payloads cross as bytes fix(ffi)!: export byte payloads as bytes Oct 28, 2026
@kixelated
kixelated marked this pull request as ready for review October 28, 2026 23:48
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T04:28:34.845891Z 04e7836 Manual request
🔒 Security Review ✅ Completed 2026-10-29T04:30:22.246378Z 04e7836 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)
Reviewed commit: a19c18e

P2 — Carry implied lifetime bounds into the borrowed-candidate decision (ffi/src/collect.rs:1820–1846, with outlives computed at 1797–1802). The new elaboration discovers AsRef targets through supertraits, but outlives still examines only the function's direct clauses. For example, a Payload: AsRef<[u8]> + 'static trait implemented for &'static [u8], Vec, and String makes fn consume(p: impl Payload) select &[u8] ahead of its owned alternatives. The later bound check uses predicate_must_hold_modulo_regions, so it does not reject that call-scoped borrow. Generated glue obtains the slice from Buf::bytes(&self), which cannot satisfy the inherited 'static requirement, and the glue fails to compile; before this change the String fallback worked. Include elaborated outlives clauses when deciding whether to borrow, and add a supertrait-'static fixture with both borrowed and owned implementations that compiles the generated glue.

Direction: ranking types from actual supertraits/conversions rather than globally preferring bytes is sensible and preserves unrelated generic defaults. The non-UTF-8 cross-language fixtures target the reported payload bug well; the lifetime decision needs the same implied-bound treatment.

Verification: static inspection of candidate selection, modulo-region bound checking, generated argument borrowing, and fixture changes. No builds, tests, census, or runtime probes executed.

…for it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Fixed the P2 lifetime finding in 129f16b: choose now reads outlives from the elaborated bounds, and sources reports whether a From impl's source parameter must outlive the call. The fixture's Payload is now AsRef<[u8]> + 'static with a &'static [u8] impl, and Timed converts from B: Payload. Both paths failed to compile the glue before the fix (p0 does not live long enough) and pass now.

(written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 129f16bfd5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ffi/src/collect.rs Outdated
// An `Into` target that does not map takes what converts into it instead.
let into = (tcx.is_diagnostic_item(sym::Into, tr) && !self.maps(target(), Pos::Param))
.then(|| self.sources(def_id, target()));
for (target, outlives) in std::iter::once((target(), outlives)).chain(into.into_iter().flatten()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the parameter's lifetime requirement for From sources

For a signature such as fn consume(payload: impl Into<Timed<Bytes>> + 'static), with impl<B: AsRef<[u8]>> From<B> for Timed<Bytes>, the outer parameter requires owned data but sources returns ([u8], false). This loop replaces the outer outlives flag with that source flag and ranks &[u8] ahead of Vec<u8>. Because at checks obligations modulo regions, it accepts the borrowed candidate, and the generated glue then fails to compile when passing the call-local buffer to the 'static parameter. Combine the outer lifetime requirement with each source's requirement before choosing a borrowed candidate.

AGENTS.md reference: AGENTS.md:L15-L15

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a5bec55: a From source now owns its target when either the parameter or the source must outlive the call. timed_hex(impl Into<Timed<Bytes>> + 'static), whose From takes B: AsRef<[u8]>, reproduced the glue compile error before the fix and passes now.

(written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 129f16bfd5

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

kixelated and others added 2 commits October 28, 2026 20:03

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 20fb1a0

No new actionable bugs found in the substantive update since a19c18e, accounting for the merged main changes.

The earlier inherited-lifetime finding is fixed: choose now obtains outlives requirements from elaborated clauses, and the Payload fixture includes both the inherited 'static bound and a borrowed &'static [u8] implementation. The separate outer-lifetime issue reported in the independent review is also addressed by combining the parameter and From-source requirements with outlives || source; timed_hex exercises an outer 'static parameter with a non-static AsRef source bound.

Direction: keeping candidate ranking tied to actual bounds, with an owned fallback when either lifetime requirement demands it, is a focused correction and avoids globally changing generic defaults. The cross-language non-UTF-8 fixtures and Kotlin migration note remain appropriate. A From-source-only inherited 'static case is inspected but not retained in the final fixture, as the PR notes.

Verification: static review of the lifetime fixes, candidate selection and changed fixtures. No build, generated-glue compilation, tests or census run; the author's reported passes were not independently reproduced.

(Written by OpenAI)

@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 2d9d0752d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2d9d0752d8

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 04e7836e33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 04e7836e33

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@kixelated
kixelated merged commit a38f7c4 into main Oct 29, 2026
1 check passed
@kixelated
kixelated deleted the quest/m1/ffi-export/payload-bytes branch October 29, 2026 04:54
kixelated added a commit that referenced this pull request Oct 29, 2026
#197's From-impl rule now takes a source that is no type parameter as it
is, so impl Into<Timed<&'a T>> takes &T, and a later parameter's bounds
see each earlier one chosen. moq-mux's json::Snapshot::update and
json::Stream::append export, and moq.c, moq.cpp, and moq.kt publish a
JSON value and read it back off the track.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 29, 2026
…ts it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 29, 2026
Co-authored-by: Luke Curley <432854+kixelated@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 29, 2026
…destroyed C++ task's executor use (#211)

* chore(quest): claim quest/m1/ffi-export/session-tokio.md

Quest-Claim: 58e0b7aafe5e26e5

* test(ffi): serve and read a moq-tokio session from C, bound to port 0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(ffi): run the QUIC session round trip from C++ and Kotlin too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(ffi): keep the C++ worker's queue alive for a late post

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ffi): destroying a C++ task waits out a post under way

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(ffi): write the Kotlin session's payload as bytes, as #197 exports it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Luke Curley <432854+kixelated@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant