Skip to content

[Brook S5] Added the client Stream and Stream reads. - #375

Open
moetemp wants to merge 6 commits into
moe/brook/review/s4-backendsfrom
moe/brook/review/s5-client-reads
Open

moetemp wants to merge 6 commits into
moe/brook/review/s4-backendsfrom
moe/brook/review/s5-client-reads

Conversation

@moetemp

@moetemp moetemp commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

What was changed

This PR adds temporalio.client.Stream, which makes the Stream calls outside Workflow code, with Client.create_stream, Client.get_stream, Client.to_stream and Stream reads (Subscription).

  • Client.create_stream is create-or-get by Stream id. On a Stream that exists, it ignores the options and returns that Stream, and on a closed one it raises StreamClosedError. It takes an optional backend name, so it doesn't depend on the Store.
  • Client.get_stream(definition, *, id, backend=None) makes no call. Reads decode each Record with the definition's payload type.
  • Client.to_stream(handle, *, payload_type=None) gives the Stream of a handle that came as data, such as a query result. It makes no call. Reads decode each Record as a JSON value, unless payload_type is given. stream.handle gives the data to pass on.
  • publish starts the write at the call and returns a future. The writes of one Stream keep the order of the calls. A write error that no caller got raises at the next call of the Stream, and close closes first, then raises it.
  • rpc_metadata and rpc_timeout are keyword-only on publish, subscribe, get_offset and close.
  • close on a Workflow's Stream raises StreamNotOwnerError with no call, and so does async with at __aenter__.
  • A read yields Records, Superseded items that withdraw earlier Records, and Closed last. It long-polls and starts at an offset, at END or at the newest last Records. It matches each item on the oneof of data, superseded and close. The long poll's loop, which carries the offset across empty pages, is here.
  • A client Producer id has a key for its process. A forked child makes its own key, so two children of one parent never write as one Producer.
  • A read keeps the offsets that it gave of each Producer that a later Superseded item can withdraw, as runs of consecutive offsets. A batch has consecutive offsets, so a read of a million batched Records keeps almost nothing, and a read of a million single Records keeps about 17 MB, not 44 MB.
  • An in-Python fake of the crate's runtime lets the facade tests run with no Store.
  • _convert.py and _writer.py are here and not in S3 and S4, since their first users are here.

Depends on S4 (#374). No changelog fragment, since S9 has the Streams entry.
Part of the Brook review series.

Why?

This is the client half of the API, which S6 runs on Redis and in memory.

Checklist

  1. How was this tested:
    • poe lint, poe gen-docs and pytest tests/contrib/streams.
    • Fork CI passes, except the arm jobs, since the fork has no arm runners.

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 25, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from b146c64 to ce8bd6d Compare October 26, 2026 00:14
@moetemp
moetemp force-pushed the moe/brook/review/s5-client-reads branch from 252b606 to b70af4e Compare October 26, 2026 00:14
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from ce8bd6d to e453c8e Compare October 26, 2026 02:08
@moetemp
moetemp force-pushed the moe/brook/review/s5-client-reads branch 2 times, most recently from 50134b6 to 6636d64 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch 2 times, most recently from 0d0ab14 to 9f62387 Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/s5-client-reads branch 2 times, most recently from 86ae5d2 to c774176 Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from 9f62387 to 9afffa1 Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s5-client-reads branch 2 times, most recently from fe0790c to 62769cb Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from 36ed7f5 to c0c0a86 Compare October 28, 2026 07:54
`Client.create_stream` creates a Stream or gets the one that exists,
and `Client.stream` binds one with no call. A `ClientStream` in a query
result uses the client that made the query. A read yields Records, the
Superseded items that withdraw earlier Records, and the close.
@moetemp
moetemp force-pushed the moe/brook/review/s5-client-reads branch from 62769cb to 7265a04 Compare October 28, 2026 08:05
A closed or expired Stream fails the create, and a close sends only its failure, which sets the
close status. A read skips an item kind that it doesn't know.
The proposed API has one handle type in every context. A handle passes each call to the calls of
the context where it runs. In client code that's the client that made the handle or the client
that decoded it. `Client.get_stream_handle` replaces `Client.stream`, and `publish` starts the
write and returns its future.
# Conflicts:
#	temporalio/common/__init__.py
#	temporalio/common/_streams/_handle.py
@moetemp moetemp changed the title [Brook S5] Added client Stream handles and Stream reads. [Brook S5] Added the client Stream and Stream reads. Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant