Skip to content

ADR 0012: a batch is a template and a table - #237

Merged
SimonHeybrock merged 6 commits into
mainfrom
236-a-batch-keeps-its-rows
Nov 19, 2026
Merged

SimonHeybrock merged 6 commits into
mainfrom
236-a-batch-keeps-its-rows

Conversation

@SimonHeybrock

@SimonHeybrock SimonHeybrock commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

ADR 0012, proposed: a batch is a template and a table. A decision to review; no code. On top of #235. Part of #236.

Rewritten after the first review, which asked why a table carries what only a scan needs.

The question

What is a batch if it is made once and then changed by a person? The ADR answers that first. How a batch is amended from the files, Follow included, is decided on top of it, in ADR 0013 (the pull request on top of this one).

The decision proposed

batch '20261118-130204'
  header     title, description
  template   spec      loki-iofq
             params    {'wavelength': {...}, 'thickness': 1.0}    # the settings: one value per parameter, for all rows
             blanks    ('sample', 'sample_transmission')          # the parameters that each row gives
  rows       one DataFrame: a column per blank, and notes

key     run    name         sample      sample_transmission
60339   60339  D2O buffer   run:60339   run:60305
60340   60340  sample A     run:60340   <NA>
water   <NA>   by hand      run:60012   run:60013
  • Each row fills the template once: template.fill(params=row.dropna()), with the rules of Template unchanged. A parameter is a setting or a column, never both.
  • One value per cell, and only a person changes the table. No "set by hand", no marks per row. batch.frame() gives the table and batch.put(frame) replaces it.
  • A column named as a parameter of the spec is a parameter column, and these are the blanks. Any other column is a note. A parameter becomes a column at any time: frame['thickness'] = 1.0.
  • An empty cell gives no value: the parameter takes its default, None if it takes None, and the row is incomplete otherwise.
  • The source and the range of rows are arguments of making a batch, not parts of it.
  • pandas holds and stores the table, with a dtype per column from its parameter.

What changes for a person

  • A table shows without the files. A cell that names a file that is not found is marked.
  • A correction of a run changes no batch that exists.
  • A wrong value in a cell is refused when it is set, not when the row is reduced.
  • A spec cannot make a person choose between a file and None for each row; an empty cell of such a column is None.

To decide in review

  • An empty cell is None where the parameter takes None.
  • The order: the code of this ADR is written together with that of ADR 0013 (the pull request on top of this one), since Follow is in use.

🤖 Generated with Claude Code

Proposed. The rows of a batch are computed from the files at each scan
and stored nowhere, so the table needs the files, removing files removes
rows, and a table cannot go in and out without loss. The ADR proposes
to keep the rows as plain data, with a value from the source and a value
set by hand per cell, to make a scan a merge, and to let only a person
remove a row. It says how a pandas user gets the table (frame, put) and
which alternatives were considered.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock self-assigned this Nov 19, 2026
@SimonHeybrock SimonHeybrock mentioned this pull request Nov 19, 2026
5 tasks done
A second study built both cores with plain cells. Of the columns that
the page offers, all are files, outputs or numbers, so with a text form
for an output every such cell is text or a number, and the open point on
the form of a file is settled the same way. A value has one plain form:
a reference as a dict had two that compare unequal.

With plain cells nothing countable decides between rows of plain data
and a DataFrame inside the batch. The ADR chooses rows by a narrow
margin and says on which facts, and when the frame would be the one to
take.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Comment on lines +39 to +40
key '60339'
index 60339 # what the scope compares

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do this ever differ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. In both real sources the key is the text of the index. They differ only for a run without a number and for a row made by a person, which have no index.

In the rewritten ADR there is no index: the run number is an ordinary typed column (run) that whoever makes the table gives, and a range of runs is frame[frame.run >= start] when the batch is made.

Comment on lines +41 to +46
info {'name': 'D2O buffer'}
source {'sample': 'run:60339', 'transmission': 'run:60305'} # what the source gave, at the last scan that gave the row
hand {'thickness': 2.0} # what a person set
cleared ['transmission'] # the cells a person emptied
added False # True: made by a person, the source never gives it
removed False # True: removed by a person; kept so that a scan does not add it again

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel something is wrong with this. Unclear why a lot of info here is kept per-cell.

  • remove -- why do we need a scan once we have a batch table? If this is about follow-mode and adding new rows from that, then it is odd to have to influence our data structure / storage format.
  • same for added
  • cleared -- why can that not be encoded in the value?
  • source/hand/info -- why does this matter once the columns are created? If it matters, why is it stored on a per-row level (in the table), instead of e.g., a bunch of flags or metadata about columns (if there is really a need to know).
  • This layout seems hard to read and inefficient. Why can the "table" part of the batch not just be serialised by Pandas, alongside some other metadata/info bespoke to our Batch?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the layout stored the two inputs of a merge (what the source gave, what a person set) in place of the table. All of it was there for a scan that runs again.

Rewritten from the opposite end: what is a batch if it is made once and then changed by a person? A template (the settings; its blanks are the columns) and one DataFrame with one value per cell, stored by pandas. source, hand, cleared, added, removed, info and index are gone. An empty cell is the cleared cell.

What a scan needs (which source, which range, which rows a person removed) is left to an ADR on amending a batch, with the condition that it is state next to the batch, not part of the table.

The first version kept two values per cell, one from the source and one
set by hand, and marks per row for a scan that runs again. Review asked
why a table carries what only a scan needs.

The ADR now decides what a batch is when it is made once and changed by
a person: a header, settings, and one DataFrame of rows with one value
per cell. A cell overrides a setting, so a parameter can become a column
at any time and a cell needs no form for None. The source and the range
of rows are arguments of making a batch. pandas holds and stores the
table.

How a batch is amended from the files, Follow included, is left to an
ADR of its own, with two conditions that follow from this one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock changed the title ADR 0012: a batch keeps its rows ADR 0012: a batch is a table Nov 19, 2026
SimonHeybrock and others added 2 commits November 19, 2026 09:50
"A value for every row" read as a value per row. The settings map a
parameter to one value that all rows share.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The settings of a batch are a template, and "a cell overrides a
setting" was a second rule next to the template's own: fill refuses a
value for a parameter that the template fixes, and a blank has no
value. The ADR now uses the template unchanged. Its blanks are the
parameter columns of the table, a parameter is a setting or a column and
never both, and each row fills the template once.

An empty cell gives no value: the parameter takes its default, None if
it takes None, and the row is incomplete otherwise. The template names
the spec, so the header keeps the title and the description.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock changed the title ADR 0012: a batch is a table ADR 0012: a batch is a template and a table Nov 19, 2026
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Base automatically changed from 224-next-batch-in-manager to main November 19, 2026 11:14
@SimonHeybrock
SimonHeybrock merged commit a65283c into main Nov 19, 2026
2 checks passed
@SimonHeybrock
SimonHeybrock deleted the 236-a-batch-keeps-its-rows branch November 19, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant