Skip to content

A table as a frame: dtypes, cells and exact storage (ADR 0012) - #239

Merged
SimonHeybrock merged 3 commits into
mainfrom
236-table-form
Nov 19, 2026
Merged

SimonHeybrock merged 3 commits into
mainfrom
236-table-form

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

First of ten pull requests for #236, which implement ADR 0012 (a batch is a template and a table) and ADR 0013 (Follow adds the rows that a batch has not considered). They are merged one by one, in order, not stacked.

This one adds the frame form of a table to ess.apps.batch.table, with no caller yet, and sets both ADRs to accepted.

dtypes(model)              # the dtype of the column of each parameter: string, Float64, Int64, boolean, object
typed(frame, model)        # the frame as a batch holds it; refuses a key or a cell, naming it
as_request(model, cells)   # 'run:60339' -> FileRef, 'r17.centre' -> OutputRef, element-wise in a list
cell_of(value)             # the inverse
data_of(frame)             # the plain data of rows.json
frame_of(data)             # reads it back equal, dtypes included

ADR 0012 corrected

Probes showed three lines of its code false. The decision is unchanged.

  • to_json(orient='table') and read_json round floats (0.1 + 0.2 reads back as 0.3) and integers above 2**53. Every row would then read as outdated after a restart. rows.json keeps the same shape, but its records are written as Python values and each column is read back with pd.array(values, dtype).
  • rows.loc[key, blanks] gives one Series, so an Int64 cell becomes a float when all blanks are numbers. A row is read with to_dict('index').
  • A request refuses text at a file field. So the text of a file or an output is converted in one place, as_request, before Template.fill.

The dtype table of the ADR now says what the code does for a union: a file or an output is string; a value or an output is object.

import ess.apps.batch needs pandas from here on.

🤖 Generated with Claude Code

table.py gains the frame form of ADR 0012 next to RowTable, with no
caller yet:

- dtypes(model): the dtype of the column of each parameter.
- typed(frame, model): the frame as a batch holds it; refuses a key
  that is not text, empty or given twice, and a cell not of its
  column's dtype, naming each.
- as_request / cell_of: the one place where a cell (text for a file
  or an output) and a value of a request are converted.
- data_of / frame_of: the plain data of rows.json. pandas' to_json
  rounds floats and integers above 2**53, so the records are Python
  values and each column is read back with pd.array.

ADR 0012 and 0013 are accepted. ADR 0012 now states how rows.json is
written and read, takes a row with to_dict('index') (a row as one
Series turns Int64 into float), converts files and outputs with
as_request before fill, and names rows_of; ADR 0013 names rows_of.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock self-assigned this Nov 19, 2026
SimonHeybrock and others added 2 commits November 19, 2026 11:58
From review of the frame form:

- A cell must be of the kind of its column, with no conversion by
  pandas: string takes text, Float64 an int or a float, Int64 an int
  within 64 bits, boolean a bool, object plain data. A float NaN is
  empty. So text 'nan' cannot become a value that rows.json empties,
  and True is not a number.
- typed refuses a column named 'key' (data_of would overwrite the keys
  with it), a column not named by text, and a note whose dtype is not
  one of the five, which rows.json would change silently.
- frame_of builds each column with the same rule as typed, refuses a
  field given twice, and names row and column in its errors.
- as_request leaves empty text at a file for the request to refuse.
- cell_of gives the JSON form of anything that is not a reference or
  a list, such as a group or a time.
- dtypes refuses a parameter that takes text or an output, since a
  cell cannot tell the two apart; an enum of text and a date are text.

ADR 0012 says a time is text, that no column is named 'key', and that
its read_csv example holds for tables of files and text only.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Plain pandas gives whole numbers with a gap as a float column
(pd.DataFrame({'count': [1, None]})), so a column of whole numbers
takes 1.0 and keeps it as 1. 2.5, inf and a bool stay refused.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SimonHeybrock
SimonHeybrock merged commit 1d8e4d9 into main Nov 19, 2026
2 checks passed
@SimonHeybrock
SimonHeybrock deleted the 236-table-form branch November 19, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant