Proposal: a Result Service to read execution results without a computing unit

[mengw15]

Follow-up to #5363. I'd like to propose moving result retrieval out of the computing unit into a new global service, and to hear objections or alternatives before opening sub-issues.

The problem

Execution results (result tables, runtime statistics, console logs) are Iceberg tables in the deployment's storage, and they outlive the computing unit that produced them. Reading them does not: the endpoints behind the result panel (result paging over the workflow websocket, /api/executions/{wid}/stats/{eid}, result export) are hosted by ComputingUnitMaster, and the gateway routes them to the CU pod. Once the CU is gone there is no route, and the panel stays empty even though the data is still there.

With per-user warehouses on (#8713), those results now sit in a warehouse the user owns, which makes not being able to see them harder to justify.

Why a service, and not Lakekeeper

Reading a result does not need the engine. Paging is a DB lookup for the result URI followed by an Iceberg read: load the table from Lakekeeper, read the Parquet files from S3. Only the deployment topology ties it to the CU.

Lakekeeper can't take this over. It is a catalog, not a query engine: its API serves table metadata and request signing, and has no endpoint that returns rows. Server-side scan planning is not implemented, and even in the Iceberg REST spec it returns a list of files, not data. Something on our side has to hold an Iceberg client and read the files.

Proposal

A new global result-service that serves execution results, whether the execution is still running or has finished: paging, column statistics, export and runtime statistics, and later their deletion under a retention policy. It plays the same role for Lakekeeper that file-service plays for LakeFS: a Texera service in front of a catalog, translating Texera concepts (execution, operator) into catalog ones (warehouse, namespace, table) and enforcing Texera's access checks.

	Today	Proposed
Execution running	The CU checks for new results and pushes which pages changed; paging over the websocket to the CU	The CU still checks and pushes which pages changed; the browser reads those pages from result-service over HTTP
Execution finished	Paging over the websocket to the CU	HTTP to result-service; no CU needed

Reads also have to name the execution. Today the CU resolves "the latest execution of this workflow on this CU" for every page; without a CU that question has no answer. The panel instead pages an execution by id: the one that is running, whose id the CU's pushed updates will carry, or by default the user's latest finished run.

A local prototype confirmed the shape: with every CU terminated and the CU process stopped, reopening a workflow showed the last run's results and paged through them, without opening a websocket.

Cleanup is affected too

Results are deleted from the CU today: when the last user leaves a workflow (after 30 seconds) and when a re-run starts on the same CU. So whether a result survives depends on how the user left. Once reads leave the CU, deletion should follow a retention policy in the same service rather than CU events.

Plan

1. Result-service with read endpoints; the frontend reads every result page over HTTP, with the CU only pushing which pages changed.
2. The CU stops deleting results when the last user closes the workflow, and retention in result-service replaces the CU-side deletion.
3. User-initiated deletion moves into the same service.

Comments welcome, especially from anyone who knows of other places that assume a CU is alive when results are read.

Edited Oct 20: result pages of running executions are read from result-service too, with the CU only pushing which pages changed; stopping the CU-side deletion moved to step 2. Tracked in #5363.

[kunwp1]

My understanding of this discussion is to move some of the result related logics from CU to result-service to decouple the lifecycles between CU and results. I like this idea overall. We had a concern that if the CU is deleted users can no longer see the results and this idea can resolve it.

[bobbai00]

I support this idea. I also verified that Lakekeeper is only providing Iceberg Metadata management. It does not provide ways for fetch data. We do need a result service

[chenlica]

I support this idea. @mengw15 An architecture diagram will be helpful.