Skip to content

A batch keeps its table; Follow adds the rows not considered (ADR 0012, ADR 0013) - #241

Draft
SimonHeybrock wants to merge 10 commits into
mainfrom
236-batch-keeps-its-table
Draft

SimonHeybrock wants to merge 10 commits into
mainfrom
236-batch-keeps-its-table

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

Third pull request for #236: the code of ADR 0012 (a batch is a template and a table) and ADR 0013 (Follow adds the rows that a batch has not considered).

What a batch is after this

batch '20261118-130204'
  header.json     title, description
  settings.json   the template: spec, params (the settings), blanks (the columns)
  rows.json       one DataFrame: the key as index, a column per blank, notes
  source.json     only if made from a source: its name, and the keys considered
frame = batch.frame()                          # a copy of the table
frame.loc['60340', 'thickness'] = 2.0
batch.put(frame)                               # the only way to change the table; an equal frame writes nothing

manager.create('loki-iofq', title='Afternoon', source='sample-runs', rows=given[given.run >= 60339])
manager.step()                                 # Follow: add the rows not considered and filled, reduce those without a record
manager.offered(label); manager.take(label, keys)   # a person adds rows of the source
  • A source gives a DataFrame (Source(rows, fills, selector)), the same thing that put takes.
  • A cell is plain data ('run:60339', 'r17.centre'), converted to what a request takes in one place, table.as_request.
  • A table file (.csv, .tsv, .xlsx) is the frame as text; the command and the page read and write it with table.read_table and table.write_table.
  • The state of a row is computed from the records and never kept.

What goes

Row, RowTable, Scope, EMPTY, edits.json, Batch.scan, the spec, columns, source and scope in the header, "set by hand", and on the page: revert of a cell, "bring back", the scope fields, the scan after a correction, and "Start the next batch here". The batch layer shrinks from 2865 to about 2480 lines in the modules of the model; the pull request as a whole removes more than it adds.

Choices made here, not in the ADRs

  • The manager steps Follow and is the only writer of source.json, so the web layer copies no wiring.
  • "Start the next batch here" is not built again. A person makes the next batch with "New batch" and turns Follow on for it. ADR 0012 and 0013 still describe it; they are amended once this is confirmed.
  • Source keeps only rows, fills and selector. A row is named by its note name if the table has one, else by its key.

Not in this pull request

The page actions that the stored table makes possible ("New batch" from a range of rows or a table file, "Add rows from the runs", "Make this a column") and the rewrite of the guide batch.md follow.

🤖 Generated with Claude Code

SimonHeybrock and others added 10 commits November 19, 2026 13:04
A batch is a template and a table. The table is one DataFrame that only
`Batch.put` changes; its parameter columns are the blanks of the template,
and it is kept as rows.json. A batch is shown without its source and
without a scan, so the files no longer decide what rows it has.

- `Batch(client, spec, label=, settings=, rows=, definitions=)` with
  `frame()` and `put(frame)`. A put of an equal table writes nothing; a
  put with other columns rewrites the settings. The request of a row is
  the template filled with its cells, an empty cell giving the default, or
  None if the parameter takes None. Resolved cells are forgotten at each
  put. The Batch writes its definitions and never reads them.
- `source.py`: a source makes a frame (`rows_of`), with an empty column
  for each column it fills.
- Follow adds the rows of the source that are new and filled
  (`auto.added`), never changes a row the table has, and reduces the rows
  without a result. The keys considered are kept in source.json by the
  manager alone. `manager.step()`, `offered(label)` and `take(label,
  keys)` are what the page calls.
- The header is title and description. `create` takes the first rows and,
  with a source, considers every key it gives now.
- Gone: `RowTable`, `Row`, `EMPTY`, `Scope`, `Words`, the scan, edits.json,
  `start_next`, `check_fills`, `auto.run`, the problem "not a column".

The command, the page and the applications are adapted after this, so
their tests fail until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pair_by_sample returns a frame: the key as index, sample and
sample_transmission as the text of the file, and the notes run (Int64,
empty for a run that is not a number) and name. SAMPLE_RUNS is a plain
Source of these fills; the words, the row name and the index title go
with the Source fields that held them.

iofq_batch and COLUMNS go: only a test used them. A batch of I(Q)
without a manager is Batch(client, IOFQ, rows=rows_of(client,
SAMPLE_RUNS), staged=True), and the module docstring shows it. A test
checks that such a batch has its columns on a folder without files.

The stories of section E make their batch with rows=rows_of(...) and
pass the source to Automatic; E4 no longer scans. user-stories.md shows
the same code. E1 stays xfail for the reason of ADR 0013.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A table file (.csv, .tsv, .xlsx) is the frame of a batch as text, so it is
read and written in table.py with the dtypes of the parameters: read_table
needs a column key, reads each parameter column with its dtype, a list or
a group as JSON text, and drops status, record and failure, so a table
downloaded and loaded again unchanged changes nothing. read_text reads a
table that is not of a batch (a run table, a reduced table to export) as
text. read_cell and write_cell move from text.py to table.py.

frames.py keeps what the command runs. reduce_table makes a Batch from the
template's settings and the table, whose parameter columns are the blanks.
reduce_batch reduces rows of a batch that is already set up, and
with_results adds what came of each row to a frame indexed by key.
rows_of, table_of and batch_table go: a source and a file give the frame
that put takes.

text.py keeps templates. checked_settings no longer compares the blanks
with the columns, leaves out a setting at a column, and names files with
table.as_request; named_file and with_named_files go.

The command (ADR 0012, 0013): reduce on a new label makes the batch from
the table, or from every row of --source, or from one row keyed by the
label, and checks the template before anything is written. On a label that
exists, the template becomes the settings and the table replaces the table;
--source is refused. The clean-up of a batch made by a failed run goes.
export_table takes the key from the index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p kind

- A Follow step that raises after adding rows still keeps the keys it
  considered, so a row a person removes stays away.
- The table owns the lists and groups in its cells: typed and Batch.frame
  copy them, so only put changes a batch.
- A note with no value keeps its dtype, or is text where pandas made it
  float or object, so text can be set into it after a step or a take.
- create deletes what it wrote if a write is refused, so the label stays
  free; the header is written last. Tests of both write orders.
- Batch holds one Template; create makes its first table and template
  with a Batch.
- A step names the source of a row the batch refuses; take says which rows
  are in the batch already. FromSource is not exported.
- ADR 0012 and BatchManager.batch name the second case in which a batch
  cannot be shown: a version of the spec that refuses the cells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/essapps/src/ess/apps/batch/table.py
frames.table_of is gone, and a source gives the frame that a batch
takes, keyed by its index. The notebook takes the table from rows_of,
edits a cell by its key, names a run in a file column as 'run:60394'
(a cell of a file column is text), selects the key as the index, and
reads a run table from a file with table.read_text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each action that changes rows builds a frame and puts it to the batch:
an edit of cells, a row added or removed, and Load table, which replaces
the whole table. A value chosen in a picker is kept as table.cell_of
makes it. The forms of the cells and settings are built again when the
columns of the batch change.

The page shows the table as it is kept, without its source or files. A
row is named by its note `name`, else by its key, and every other note
is shown. "Rescan runs" lists the runs again and changes no row; a
correction of runs changes no row either.

A tick of Follow is `manager.step()`; the page of the batch that
follows shows what it refused, how many rows wait, and its failure.
Follow is offered on every page; the manager refuses a batch not made
from a source, and the page says why.

New batch makes a batch of every row that its source gives now, with
an empty column for each ticked column the source lacks, or, without a
source, of the ticked columns and no rows. The counts read each state
without a scan; `outside` counts the keys of each source that no table
has.

Gone from the page: revert and the edited/scanned marks of batch cells
(the Runs view keeps both), bring back, rows added by hand, strays,
scope, "Start the next batch here", and the scan after a correction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build made its first batch without rows, so the batch had no columns
and no blanks (plan, G10). It is now made with
rows=manager.rows_of('sample-runs') and source='sample-runs', and so
considers every sample run listed at the start; a run that arrives
later comes as a row through a step of Follow.

"Rescan runs" lists the runs again and changes no row. The tests of the
page and of the arrivals check that: a relist leaves the rows as they
are, and a step of Follow adds and reduces the sample runs whose
transmission run arrived. The docstrings of web.py and arrivals.py say
the same, and that a newer transmission run changes no row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock self-assigned this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant