From b907419acc3f39c0b0659348fb3504e84fcadff7 Mon Sep 17 00:00:00 2001 From: Simon Heybrock Date: Thu, 19 Nov 2026 06:43:47 +0000 Subject: [PATCH 1/6] ADR 0012: a batch keeps its rows Proposed. The rows of a batch are computed from the files at each scan and stored nowhere, so the table needs the files, removing files removes rows, and a table cannot go in and out without loss. The ADR proposes to keep the rows as plain data, with a value from the source and a value set by hand per cell, to make a scan a merge, and to let only a person remove a row. It says how a pandas user gets the table (frame, put) and which alternatives were considered. Co-Authored-By: Claude Fable 5.2 --- .../adr/0012-a-batch-keeps-its-rows.md | 215 ++++++++++++++++++ docs/developer/adr/index.md | 3 +- 2 files changed, 217 insertions(+), 1 deletion(-) create mode 100644 docs/developer/adr/0012-a-batch-keeps-its-rows.md diff --git a/docs/developer/adr/0012-a-batch-keeps-its-rows.md b/docs/developer/adr/0012-a-batch-keeps-its-rows.md new file mode 100644 index 00000000..67d63251 --- /dev/null +++ b/docs/developer/adr/0012-a-batch-keeps-its-rows.md @@ -0,0 +1,215 @@ +# ADR 0012: A batch keeps its rows + +- Status: proposed (#236) +- Deciders: Simon +- Date: 2026-11-19 + +## Context + +*Batch*, *label*, *header*, *scope*, *source*, *definitions* are the words of [ADR 0011](0011-a-person-makes-the-batches-of-a-workspace.md). +*Row*, *key*, *cell*, *scan*, *edit*, *result* are those of [batch.md, "Terms"](../../../packages/essapps/docs/batch.md#terms). + +Today the rows of a batch are stored nowhere. They are computed at each scan, and only what a person set on top is kept: + +```python +scan = source.rows(files with corrections) # code of the application +rows = scan in scope, with edits on top # edits.json: added, edited, cleared, removed, by key +``` + +So a batch is defined by how its rows are made, and the table a person sees is the output of that recipe now. Five problems follow. + +1. **The table needs the files.** If the file mount is not available, the scan gives no rows, and the table is empty. The results are in the workspace, but no row shows them. If the listing raises, the page of the batch cannot be shown. +2. **Removing files removes rows.** A row whose file is gone disappears, also one that was reduced or edited. Its edits are kept and apply to nothing. +3. **A change of the source's code changes every batch**, also one that was finished last week. +4. **The model has no table to put a table in.** The command takes one (`essapps-batch reduce iofq.yaml runs.csv`), and the page loads and saves one. Both are translated into "cells set by hand on top of the scan", in four structures of different shape (`added`, `edited`, `cleared`, `removed`) and a marker for a cleared cell. The origin of a cell is worked out again in six places. + +5. **A cell is held in two forms.** A cell is a stand-in, never a resolved value: a file is named by an identity, an output by the ids of its record and output. But the stand-in is a pydantic object if a source or the page made it (`FileRef(file='run:60339')`), and plain data if a table file did (`{'@ref': 'file', 'file': 'run:60339'}`). Four functions convert between the forms (`_values`, `with_named_files`, `write_cell`, `read_cell`) and do not agree: a table file loaded back unchanged marks a cell that holds a reference to an output as set by hand. + +A person thinks of a batch as a table: they edit its cells, remove its rows, reduce its rows and read their results. All of that refers to rows, so the rows must last as long as what refers to them. + +## Decision + +**A batch keeps its rows, as plain data. The source and the scope are what fills the table, and only a person removes a row.** + +### What is kept + +One table per batch, `rows.json`, in place of `edits.json`. Per row: + +```python +key '60339' +index 60339 # what the scope compares +info {'name': 'D2O buffer'} +source {'sample': 'run:60339', 'transmission': 'run:60305'} # what the source gave, at the last scan that gave the row +hand {'thickness': 2.0} # what a person set +cleared ['transmission'] # the cells a person emptied +added False # True: made by a person, the source never gives it +removed False # True: removed by a person; kept so that a scan does not add it again +``` + +A cell holds two values, one per writer. The cell shown is the value set by hand if there is one, nothing if a person emptied it, else the source's: + +```python +cells = {c: v for c, v in {**row.source, **row.hand}.items() if c not in row.cleared} +# in pandas words: source.mask(by_hand, hand) +``` + +Two values are needed, not one value and a mark: "revert" shows the source's value again, and the page shows it under a cell set by hand. + +### A cell holds plain data + +A cell holds the plain form of a value, as the parameters of a request are held before they are resolved: text, a number, a truth value, `None`, or a list or dict of these. + +```python +'run:60339' # a file, in a column whose field takes a file +2.0 # a thickness +None # "no direct beam": a value. An empty cell is a key left out. +['path:masks/beamstop.xml', 'path:masks/edges.xml'] # several files in one cell +{'start': 1.0, 'stop': 13.0, 'num_bins': 200, 'unit': 'Å'} # a group +{'@ref': 'output', 'record': 'r17', 'output': 'centre'} # an output of another record +``` + +A value is brought to this form once, where it enters the table: from a source, from an edit on the page, from a table file. The field of the column validates it when a request is made, as today. The four converters go. Two cells are equal if their plain data are equal, and `rows.json` is the table as it is. + +Whether a file is the text `run:60339` or the tagged form `{'@ref': 'file', 'file': 'run:60339'}` is open (see "Open"). + +### A scan is a merge + +```python +def scan(table, files): + given = {row.key: row for row in source.rows(files)} # the source sees every file + for key, row in given.items(): + if key in table: + table[key].update(index=row.index, info=row.info, source=row.values) # never `hand` + elif scope.holds(row): + table.add(row) + not_given = {key for key in table if key not in given and not table[key].added} +``` + +| The scan gives a row | The table has it | Then | +|---|---|---| +| yes, in the scope | no | the row is added | +| yes, outside the scope | no | nothing | +| yes | yes | `index`, `info` and `source` are replaced; `hand`, `removed` stay | +| no | yes | the row stays as it is, and is shown as *not in the files* | + +A scan that changes nothing writes nothing. A scan that cannot list the files changes nothing, and the page says why. + +*Not in the files* is not kept: it says what the last scan of this process found. A row not in the files keeps its cells, its result and its plot. A person can reduce it if its cells still resolve, and remove it. Follow leaves it alone. + +### The scope says which rows the batch shows and which rows a scan adds + +As in ADR 0011, the scope is a slice of the index, and a batch shows the rows inside it and the rows added by hand. New: the scan adds only rows inside the scope, so a batch stores its range of the rows, not every row of the source. + +- Widening the scope shows nothing new at once; the next scan adds the rows. +- Narrowing it hides the rows outside. They stay in `rows.json` with what was set by hand, so widening it again shows them as they were. + +### Only a person removes a row + +A removed row is kept, marked `removed`, so that the next scan does not add it again, and so that "Bring back" works as today. Nothing else removes a row: not a scan, not a correction of a run, not a change of the source's code. + +One action removes the rows that are not in the files, for the case below where a correction leaves a row behind. + +### "Start the next batch here" copies rows + +The next batch gets the rows from the index on, as they are: `source`, `hand`, `removed`. Then the scope of this batch ends there, which hides them here. Results stay under the label of this batch (ADR 0011). + +### A table goes in and out without loss + +```python +frame = batch.frame() # a DataFrame: the key as index, a column per column, then status and what was set by hand +batch.put(frame) # a cell that differs from the cell shown is set by hand; a key the batch lacks is added +batch.put(batch.frame()) # sets nothing: the contract, tested with every type of cell an application has +``` + +The CSV of the command and the table file of the page are this frame as text. The batch layer keeps rows of plain data, as JSON; `frames.py` stays the one place that converts to and from a `DataFrame` (see "pandas", below). + +## Examples + +A batch of sample runs with scope `[60300, None)`. + +**The files are not reachable.** + +``` +before mount gone, page opened +key sample transmission status key sample transmission status +60302 60302 60303 completed 60302 60302 60303 completed not in the files +60304 60304 60305 ready 60304 60304 60305 ready not in the files +``` + +Results and plots show. "Reduce" on 60304 is refused with the reason that its file is not found, as for any cell that does not resolve. + +**Two files are removed.** The two rows stay, shown as not in the files. Nothing else changes. + +**A run is corrected.** Run 60306 was recorded as a sample and is a transmission run. Before the correction the source gave a row `60306`; after it, it does not, and gives row `60304` the transmission run 60306. + +``` +key sample transmission status +60304 60304 60306 outdated # source cell replaced by the scan +60306 60306 - incomplete not in the files # stays until a person removes it +``` + +Today row `60306` disappears at the scan. Here it stays, marked, and "Remove the rows not in the files" removes it. This is the cost of the decision: a row that only wrong metadata made is not cleaned up for the person. + +**A cell is set by hand, then the file changes.** + +```python +batch.rows.edit('60304', {'transmission': 'run:60310'}) +# a correction makes the source give transmission 60306 for this row +# shown: 60310 (hand); under it: 60306 (source). Revert shows 60306. +``` + +**The command.** + +``` +essapps-batch reduce iofq.yaml runs.csv -l water # a batch without a source: the rows of runs.csv are its rows +essapps-batch reduce -s pkg:SOURCES -l 20261118-130204 -o done.csv +# edit done.csv in a spreadsheet, then: +essapps-batch reduce -s pkg:SOURCES -l 20261118-130204 done.csv # the cells changed are set by hand; the others are not +``` + +## pandas + +Developers of applications know pandas, so the table is given to them as a `DataFrame`, and the guide explains the model in its terms. + +- **`batch.frame()` and `batch.put(frame)`.** With plain cells the frame holds the same data as the table: most columns are text or numbers, and `put(frame())` loses nothing. +- **The guide states each concept with its pandas line:** + +| Concept | For a pandas user | +|---|---| +| a row's cells | `source.mask(by_hand, hand)` | +| a scan | an outer merge on the key that overwrites `source`, never `hand` | +| the scope | `rows[(rows.run >= start) & (rows.run < stop)]` | +| the results | `records.sort_values('created').drop_duplicates('member', keep='last')` | +| the state | `cells.join(results)`, a left join on the key | +| a rule, such as `left_to_reduce` | a boolean mask over the state | +| the settings | a column with one value for all rows | + +Whether the batch itself holds its rows as dicts of plain data or as a `DataFrame` is not decided here: it is not seen from outside. A batch writes before it changes and has one writer, so it does not hand out a frame that it holds, and a person works on a copy either way. Both forms were built and run through one session with equal results and equal size (162 lines against 156). Two things need care in a `DataFrame`: `None` is a value (`direct_beam: DirectBeamFile | None`), which the pandas operations for missing data treat as an empty cell; and a few columns hold a list or a dict. + +## Alternatives considered + +- **Keep the rows derived, and cache the last scan for when the files are away.** Two models: what a person sees depends on whether the cache or the scan answered, and removing files still removes rows. +- **Keep the rows, and let a scan remove those it no longer gives.** That is today's behaviour with a cache. It fails the second problem. +- **Remove a row not in the files if it has no result and nothing set by hand.** It cleans up after a correction. Rejected: a row then disappears or stays by a rule a person has to know, and a mount that is away for a moment removes rows that come back at the next scan. One explicit action does the same when a person wants it. +- **The scope only says what a scan adds; narrowing it hides nothing.** Then the scope in the list of batches no longer says which rows a batch has, and "Start the next batch here" has to move rows out of one table and into the other. +- **Cells as typed objects** (`FileRef`, `OutputRef`), as a source makes them today. Two forms of one value, and converters between them; see the context. + +## Consequences + +- `rows.json` replaces `edits.json`; `Batch.rows` stays its one writer. No code reads `edits.json`: a batch made before keeps its header and settings, and what was set by hand on it is lost. +- `RowTable` holds rows with `source` and `hand` in place of the layers `scan`, `added`, `edits`, `removed`. The marker `EMPTY` for a cell emptied by hand goes; `cleared` names such cells. The six places that work out the origin of a cell read it from the row. +- A batch is shown without a scan. A source that raises, or that is no longer registered, is a message on the page of the batch, not a batch that cannot be shown. That leaves one such case: the workspace binds no spec of the batch's name. +- `manager.start_next` copies rows; the code that carries edits over by key goes. +- A batch stores its range of the rows. A change of scope that widens it needs a scan to show the new rows. +- Rows that only wrong metadata made stay until a person removes them (the example above). +- The size of `rows.json` is that of the table: a few hundred bytes per row, bounded by the scope. It is written when a scan or a person changes the table. +- The rules over the state (`ready`, `arrived`, `left_to_reduce`) are unchanged, except that `arrived` leaves a row not in the files. +- A source may give a cell as an object, such as a `FileRef`; the table holds its plain form. The four converters between the forms go, and a table file loaded back unchanged sets nothing by hand. + +## Open + +- **The plain form of a file in a column that takes files**: the text `run:60339`, or the tagged form that a request holds. The text gives a column of text, and the field of the column says that it names a file. The tagged form needs no field to be read. + +- **Whether a removed row that is not in the files is dropped for good.** Kept here, for one rule less. The table then holds rows that nothing shows. +- **Rows shaped as a table.** A row has four kinds of fields (`key`, `index`, `values`, `info`), and five fields of `Source` say which is what. A table has columns with roles. This matters for several workflows on one batch (#222: a column has a third writer, a workflow), and is left for that decision. So is the name `index`, which means the key in pandas. +- **When the last scan was.** A page could say how old the table is when the files cannot be listed. Not needed for the decision. diff --git a/docs/developer/adr/index.md b/docs/developer/adr/index.md index f79d2256..dc041a5c 100644 --- a/docs/developer/adr/index.md +++ b/docs/developer/adr/index.md @@ -16,4 +16,5 @@ In reading order: - [ADR 0006: Runs are combined by an accumulator, not by combining the outputs of requests](0006-the-unit-is-an-accumulating-workflow.md). State n gives the outputs of the plain request over the first n pushes, and a freeze makes a record of the plain request of the logged state, state n for the n pushes logged so far. - [ADR 0003: An accumulator's driver adds in place, and push n+1 waits for the readers of state n](0003-accumulators-add-in-place.md). `client.output` copies a state and makes no record, a request reads an accumulator only through the record of its freeze, and freeze needs no copy. - [ADR 0007: Distributions are split by who depends on them and when they are released](0007-packages-split-by-dependencies.md). essspec, essapps, and essinstruments; code that needs dependencies only some users install is a subpackage behind an extra. -- [ADR 0011: A person makes the batches of a workspace, each with a scope over the rows of a source](0011-a-person-makes-the-batches-of-a-workspace.md). Proposed: a header per batch, a manager that lists the batches, and at most one batch that follows. Several workflows on one batch are not decided. +- [ADR 0011: A person makes the batches of a workspace, each with a scope over the rows of a source](0011-a-person-makes-the-batches-of-a-workspace.md). A header per batch, a manager that lists the batches, and at most one batch that follows. Several workflows on one batch are not decided. +- [ADR 0012: A batch keeps its rows](0012-a-batch-keeps-its-rows.md). Proposed: the rows of a batch are stored as plain data, a scan adds and updates them, and only a person removes a row. From f4eea3d534cad4b491098b6e97dcb4e26050c65c Mon Sep 17 00:00:00 2001 From: Simon Heybrock Date: Thu, 19 Nov 2026 07:27:40 +0000 Subject: [PATCH 2/6] ADR 0012: files and outputs are text; rows of plain data inside A second study built both cores with plain cells. Of the columns that the page offers, all are files, outputs or numbers, so with a text form for an output every such cell is text or a number, and the open point on the form of a file is settled the same way. A value has one plain form: a reference as a dict had two that compare unequal. With plain cells nothing countable decides between rows of plain data and a DataFrame inside the batch. The ADR chooses rows by a narrow margin and says on which facts, and when the frame would be the one to take. Co-Authored-By: Claude Fable 5.2 --- .../adr/0012-a-batch-keeps-its-rows.md | 38 +++++++++++++++---- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/docs/developer/adr/0012-a-batch-keeps-its-rows.md b/docs/developer/adr/0012-a-batch-keeps-its-rows.md index 67d63251..53fc6721 100644 --- a/docs/developer/adr/0012-a-batch-keeps-its-rows.md +++ b/docs/developer/adr/0012-a-batch-keeps-its-rows.md @@ -60,17 +60,19 @@ Two values are needed, not one value and a mark: "revert" shows the source's val A cell holds the plain form of a value, as the parameters of a request are held before they are resolved: text, a number, a truth value, `None`, or a list or dict of these. ```python -'run:60339' # a file, in a column whose field takes a file +'run:60339' # a file +'r17.centre' # an output of another record 2.0 # a thickness None # "no direct beam": a value. An empty cell is a key left out. ['path:masks/beamstop.xml', 'path:masks/edges.xml'] # several files in one cell {'start': 1.0, 'stop': 13.0, 'num_bins': 200, 'unit': 'Å'} # a group -{'@ref': 'output', 'record': 'r17', 'output': 'centre'} # an output of another record ``` -A value is brought to this form once, where it enters the table: from a source, from an edit on the page, from a table file. The field of the column validates it when a request is made, as today. The four converters go. Two cells are equal if their plain data are equal, and `rows.json` is the table as it is. +- **A file and an output are text.** The field of the column says that the text names a file or an output, as it does for a file named in a table today. The text of an output is what `str(OutputRef)` gives: `r17.centre`, `r18.slices[2]`. +- **Every column that the page offers is then text or a number.** Of the 17 parameters of the six specs of LoKI and the demo that can be a column on the page, 9 are files, 7 are outputs, and 1 is a number. A group or a list can be a column only of a batch made in code or with the command. +- **A value has one plain form.** It is brought to it once, where it enters the table: from a source, from an edit on the page, from a table file. For a dict or a list, that form is what the field of the column dumps, so a group typed without its defaults and the same group with them are one cell. (A reference as a dict has two forms today, with and without `'key': None`, which compare unequal.) -Whether a file is the text `run:60339` or the tagged form `{'@ref': 'file', 'file': 'run:60339'}` is open (see "Open"). +The field of the column validates a cell when a request is made, as today. The four converters go. Two cells are equal if their plain data are equal, and `rows.json` is the table as it is. ### A scan is a merge @@ -171,8 +173,8 @@ essapps-batch reduce -s pkg:SOURCES -l 20261118-130204 done.csv # the cells ch Developers of applications know pandas, so the table is given to them as a `DataFrame`, and the guide explains the model in its terms. -- **`batch.frame()` and `batch.put(frame)`.** With plain cells the frame holds the same data as the table: most columns are text or numbers, and `put(frame())` loses nothing. -- **The guide states each concept with its pandas line:** +- **`batch.frame()` and `batch.put(frame)`.** The frame holds the same data as the table: text and numbers. An empty cell is `NaN`; `None`, a value, is the text `null`, as in a table file. `put(frame())` sets nothing. +- **The guide states each concept with its pandas line**, which is literally true of the plain data: | Concept | For a pandas user | |---|---| @@ -184,7 +186,26 @@ Developers of applications know pandas, so the table is given to them as a `Data | a rule, such as `left_to_reduce` | a boolean mask over the state | | the settings | a column with one value for all rows | -Whether the batch itself holds its rows as dicts of plain data or as a `DataFrame` is not decided here: it is not seen from outside. A batch writes before it changes and has one writer, so it does not hand out a frame that it holds, and a person works on a copy either way. Both forms were built and run through one session with equal results and equal size (162 lines against 156). Two things need care in a `DataFrame`: `None` is a value (`direct_beam: DirectBeamFile | None`), which the pandas operations for missing data treat as an empty cell; and a few columns hold a list or a dict. +### What the batch holds inside: rows of plain data + +Inside, the table is rows of plain data, each a frozen value with `source`, `hand` and `cleared`; a table is replaced, never changed in place. A `DataFrame` inside was built too (four frames on one index: the rows, `source`, `hand`, `cleared`) and run through the same session with equal tables after each step. With plain cells nothing that can be counted decides between them: + +| | rows of plain data | four frames | +|---|---|---| +| lines, as written | 152 | 138 | +| lines, formatted at 88 columns | 209 | 241 | +| the state, the rules, `frame()`, `put()` | 70 lines | 38 lines | +| plain data out and in (a request, a record's params, `rows.json`) | 4 lines | 16 lines | +| a scan, the state, saving, at 3000 rows | 15, 10, 24 ms | 15, 14, 44 ms | +| a frame handed out cannot change the one held | a copy is made | yes, by copy-on-write (72 cases; one exception, `combine_first` on `object` columns) | + +The choice is made on three facts that cannot be counted, and by a narrow margin: + +1. **Three partners of a batch take plain data and one takes a frame.** A request, the params of a record, and `rows.json` are plain data, at every reduction, state and save. A person takes a frame, on demand. Rows hold the form that the three use. +2. **A frame inside needs rules that nothing checks.** Four traps were hit in 138 lines, two of them silent: `combine_first` shares memory with its argument, so a write changed the table held; `.at` at a key that is no row adds a row; `from_dict` turns 200 into 200.0; `ne` says that two empty cells differ. `None` next to an empty cell needs the text `null` in the cells. A type checker reads a row; it reads nothing of a `pd.DataFrame`. +3. **`ess.apps.batch` imports without pandas today.** + +For the frame: the state and the rules read as the formulas of the table above. It is a sound second choice, and the one to take if `None` stops being a cell value of its own (it is one at 3 of the 17 columns), or if several workflows on one batch (#222) make the column the unit of work. ## Alternatives considered @@ -208,7 +229,8 @@ Whether the batch itself holds its rows as dicts of plain data or as a `DataFram ## Open -- **The plain form of a file in a column that takes files**: the text `run:60339`, or the tagged form that a request holds. The text gives a column of text, and the field of the column says that it names a file. The tagged form needs no field to be read. +- **Whether `None` stays a cell value of its own.** `direct_beam` takes a file or `None`, "no direct beam", and an empty cell there makes the row incomplete. A named choice in its place would make every cell text or a number with nothing else to tell apart. +- **`put` of a frame that holds a row removed before** brings the row back. `frame()` leaves removed rows out, so this happens only if a person adds the key. - **Whether a removed row that is not in the files is dropped for good.** Kept here, for one rule less. The table then holds rows that nothing shows. - **Rows shaped as a table.** A row has four kinds of fields (`key`, `index`, `values`, `info`), and five fields of `Source` say which is what. A table has columns with roles. This matters for several workflows on one batch (#222: a column has a third writer, a workflow), and is left for that decision. So is the name `index`, which means the key in pandas. From 65dde158a908c8d58581a9cf38ce88a962ebd97f Mon Sep 17 00:00:00 2001 From: Simon Heybrock Date: Thu, 19 Nov 2026 09:44:04 +0000 Subject: [PATCH 3/6] ADR 0012 rewritten: a batch is a table The first version kept two values per cell, one from the source and one set by hand, and marks per row for a scan that runs again. Review asked why a table carries what only a scan needs. The ADR now decides what a batch is when it is made once and changed by a person: a header, settings, and one DataFrame of rows with one value per cell. A cell overrides a setting, so a parameter can become a column at any time and a cell needs no form for None. The source and the range of rows are arguments of making a batch. pandas holds and stores the table. How a batch is amended from the files, Follow included, is left to an ADR of its own, with two conditions that follow from this one. Co-Authored-By: Claude Fable 5.2 --- docs/developer/adr/0012-a-batch-is-a-table.md | 225 +++++++++++++++++ .../adr/0012-a-batch-keeps-its-rows.md | 237 ------------------ docs/developer/adr/index.md | 2 +- 3 files changed, 226 insertions(+), 238 deletions(-) create mode 100644 docs/developer/adr/0012-a-batch-is-a-table.md delete mode 100644 docs/developer/adr/0012-a-batch-keeps-its-rows.md diff --git a/docs/developer/adr/0012-a-batch-is-a-table.md b/docs/developer/adr/0012-a-batch-is-a-table.md new file mode 100644 index 00000000..ba2e17f5 --- /dev/null +++ b/docs/developer/adr/0012-a-batch-is-a-table.md @@ -0,0 +1,225 @@ +# ADR 0012: A batch is a table + +- Status: proposed (#236) +- Deciders: Simon +- Date: 2026-11-19 + +## Context + +*Batch*, *label*, *header*, *scope*, *source*, *definitions* are the words of [ADR 0011](0011-a-person-makes-the-batches-of-a-workspace.md). +*Row*, *key*, *cell*, *scan*, *result* are those of [batch.md, "Terms"](../../../packages/essapps/docs/batch.md#terms). + +Today the rows of a batch are stored nowhere. They are computed at each scan, and only what a person set on top is kept: + +```python +scan = source.rows(files with corrections) # code of the application +rows = scan in scope, with edits on top # edits.json: added, edited, cleared, removed, by key +``` + +So a batch is defined by how its rows are made, and the table a person sees is the output of that recipe now. Six problems follow. + +1. **The table needs the files.** If the file mount is not available, the scan gives no rows, and the table is empty. The results are in the workspace, but no row shows them. +2. **Removing files removes rows.** A row whose file is gone disappears, also one that was reduced or edited. +3. **A change of the source's code changes every batch**, also one that was finished last week. +4. **The model has no table to put a table in.** The command takes one (`essapps-batch reduce iofq.yaml runs.csv`), and the page loads and saves one. Both are translated into "cells set by hand on top of the scan", in four structures of different shape (`added`, `edited`, `cleared`, `removed`) and a marker for a cleared cell. +5. **A cell is held in two forms**: a pydantic object if a source or the page made it (`FileRef(file='run:60339')`), plain data if a table file did. Four functions convert between the forms and do not agree: a table file loaded back unchanged marks a cell that holds a reference to an output as set by hand. +6. **What a batch is depends on what amends it.** The source, the scope, and who wrote each cell are parts of every batch, because a scan must be able to run again at any time. A batch made from a table file has no source and carries the same parts. + +A person thinks of a batch as a table: they edit its cells, remove its rows, reduce its rows and read their results. + +## Decision + +**A batch is a table. It is a label with a header, settings, and rows; the rows are one table with one value per cell, and only a person changes it. How the table was made is not part of the batch.** + +``` +batch '20261118-130204' + header spec, title, description # what it is a batch of + settings {'wavelength': {...}, 'direct_beam': None} # a value for every row + rows one DataFrame # a value per row +``` + +``` +rows +key run name sample sample_transmission thickness +60339 60339 D2O buffer run:60339 run:60305 2.0 +60340 60340 sample A run:60340 +water by hand run:60012 run:60013 1.0 +``` + +This decides what a batch is when it is made once and then changed by a person. How a batch is amended from the files, Follow included, is decided after it and on top of it (see "Not decided here"). + +### Keys, columns, cells + +- **The key** of a row is text, unique in the batch, and the index of the frame. It joins a row to its results: the records under the label of the batch with that key, as today. +- **A column named as a parameter of the spec is a parameter column.** Any other column, such as `run` or `name`, is a note for the person. Nothing else tells them apart, and the header keeps no list of columns. +- **A cell holds one value, or is empty.** The value is the plain form of a parameter's value, as a request holds it before it is resolved. A file and an output are text: `run:60339`, and `r17.centre` for the output `centre` of record `r17`. +- **A column has the type of its parameter**, so a wrong value is refused where it enters the table: + +| The parameter takes | dtype | A cell | +|---|---|---| +| a file, an output, text | `string` | `'run:60339'` | +| a number | `Float64`, `Int64` | `2.0` | +| a truth value | `boolean` | `True` | +| a list, a group, or more than one of the above | `object` | `['run:1', 'run:2']` | + +### A cell overrides a setting + +```python +def request(key): + cells = rows.loc[key, parameter_columns].dropna() + return settings | cells.to_dict() # validated by the spec, as any request +``` + +- An empty cell means "as in the settings". A row is incomplete if a required parameter has neither a setting nor a cell. +- So a parameter is not either a setting or a column, fixed when the batch is made. A person adds a column for a parameter at any time: its cells are empty, every request is as before, and a value in one cell changes that row. +- A cell does not hold `None`. Where rows differ in whether they have a value, the setting says `None` and the rows that have one give it: `direct_beam: None` in the settings, and a file in the cells of the rows with a direct beam run. + +### A person changes the table + +A batch gives its table and takes one: + +```python +frame = batch.frame() # the rows, as above +frame.loc['60340', 'thickness'] = 1.0 # set a cell +frame.loc['60340', 'thickness'] = pd.NA # empty it: as in the settings +frame.loc['d2o', 'sample'] = 'run:7' # add a row +frame = frame.drop('60339') # remove a row +frame['can'] = pd.NA # add a column, for the parameter `can` +batch.put(frame) # replaces the table +batch.put(batch.frame()) # changes nothing and writes nothing +``` + +`put` refuses the whole table for a key that is empty or used twice, and for a cell that does not have the type of its column. Each action of the page is `put` with one such line before it. A table file of the page and the CSV of the command are this frame as text. + +Removing a row drops it. Its records stay under the label, as every record does, and show again if a row with that key is added. + +### Making a batch + +The first rows are an argument of making the batch, as its first settings are. Where they come from is the business of whoever makes it: + +```python +frame = table_of(source.rows(files)) # rows from the files: code of the application +manager.create('loki-iofq', title='Afternoon', rows=frame[frame.run >= 60339]) +manager.create('loki-iofq', title='Water', rows=pd.read_csv('runs.csv', index_col='key', dtype='string')) +manager.create('loki-iofq', title='By hand') # no rows yet +``` + +On the page, "New batch" takes a spec, and either a source with a range of its rows, a table file, or nothing. + +### The state is not stored + +```python +newest = records.sort_values('created').drop_duplicates('member', keep='last').set_index('member') +state = rows.join(newest) # a left join on the key +outdated = [request(key) != newest.params[key] for key in rows.index] +``` + +A cell that names a file that is not found is marked when the table is shown, and a request with it is refused, as for any cell that does not resolve. Nothing about the files is stored in the table. + +### What is kept + +``` +definitions/labels/20261118-130204/header.json spec, title, description +definitions/labels/20261118-130204/settings.json a template: the settings +definitions/labels/20261118-130204/rows.json the table: the plain data of frame.to_json(orient='table') +``` + +pandas writes and reads the table with the types of its columns. Checked on a table with `string`, `Float64`, `Int64` and `object` columns, empty cells, a list and a dict in a cell, and the key `'0342'`: the frame read back equals the frame written, with an empty cell of an `object` column read as `NaN`. + +### "Start the next batch here" splits the table + +```python +at = rows.index.get_loc('60400') +this, following = rows.iloc[:at], rows.iloc[at:] # the next batch gets `following`, and the same spec and settings +``` + +Results stay under the label of this batch (ADR 0011), so a row that moves has no result in the next batch. + +## Examples + +**The files are not reachable.** The page shows the table, the results and the plots. Each file cell is marked as not found, and "Reduce" on such a row is refused with that reason. + +**Two files are removed.** The two rows stay as they are, with their results. Their file cells are marked. + +**A run is corrected.** Run 60306 was recorded as a sample and is a transmission run. The batch has a row `60306`, and row `60304` names another transmission run. Nothing changes in the batch. The person removes row `60306` and sets the cell of `60304`. + +**A parameter becomes a column.** + +```python +settings['thickness'] # 1.0, for every row +frame['thickness'] = pd.NA # a column; every request is as before, no row is outdated +frame.loc['60340', 'thickness'] = 2.0 # row 60340 is outdated; the others are not +``` + +**The command.** + +``` +essapps-batch reduce iofq.yaml runs.csv -l water # makes the batch `water`: iofq.yaml its settings, runs.csv its rows +essapps-batch reduce -l water -o done.csv # the table, with a status and a record per row +# edit done.csv in a spreadsheet, then: +essapps-batch reduce -l water done.csv # done.csv replaces the table, and its rows are reduced +``` + +The columns of the state (`status`, `record`) are not columns of the table and are left out when a table file is read. + +## pandas + +The batch holds its rows as a `DataFrame`, so pandas becomes a dependency of `ess.apps.batch`; today only the command and the table files need it. Each concept is a line of pandas: + +| Concept | In pandas | +|---|---| +| a row, a key | a row of the frame, its index | +| a parameter column | `[c for c in rows.columns if c in parameters]` | +| the request of a row | `settings \| rows.loc[key, parameter_columns].dropna().to_dict()` | +| set, empty, add, remove | `.loc[key, column] = value`, `= pd.NA`, `.loc[key, column] = value` at a new key, `.drop(key)` | +| the results | `records.sort_values('created').drop_duplicates('member', keep='last')` | +| the state | `rows.join(results)`, a left join on the key | +| a rule, such as `left_to_reduce` | a boolean mask over the state | +| rows of a range of runs | `frame[(frame.run >= start) & (frame.run < stop)]` | +| the next batch | `rows.iloc[at:]` | + +## Not decided here: amending a batch from the files + +A source does three things to a batch today after it is made: + +| A scan | Example | +|---|---| +| adds rows as files arrive | Follow | +| fills a cell that was empty | a sample run measured before its transmission run | +| changes a cell that has a value | a transmission run measured again: the source gives the rows of its sample the last one | + +Whether each is wanted, and how it is done, gets its own ADR. Two conditions for it follow from this one: + +- **What amends a batch uses the table as a person does**: it adds a row or sets a cell with `put`. +- **What it needs to know is its own state, next to the batch**: which source, which range of rows, which rows a person removed. None of it is a column of `rows` or a field of every row. + +Until then the parts of ADR 0011 on the source, the scope and Follow stand, and the code of this ADR waits for that decision, since Follow is in use. Several workflows on one batch (#222) are the same kind of question: something other than a person fills cells. + +## Alternatives considered + +- **Keep the rows derived, and cache the last scan for when the files are away.** Two models: what a person sees depends on whether the cache or the scan answered, and removing files still removes rows. +- **Two values per cell, one that the source gave and one that a person set, and a scan that merges.** It stores the inputs of a merge in place of the table. Each row then carries what only a scan needs (`source`, `hand`, `cleared`, `added`, `removed`), and the table is not one that pandas holds or stores. It also decides the three uses of a scan above with "yes" without asking. +- **A parameter is a column or a setting, fixed at making** (today). The header then lists the columns, a column cannot be added, and a cell needs a form for `None`. +- **A list of the parameter columns in the header.** A second place that says what the columns of the frame say. +- **`None` as the value of a cell**, as the text `none`. Not needed with a cell that overrides a setting, and a text column would hold a word that is not text of its parameter. +- **Cells as typed objects** (`FileRef`, `OutputRef`). Two forms of one value, and converters between them; see the context. +- **Rows of plain data inside the batch, and a `DataFrame` only at its interface.** With one value per cell the frame is the table, and a second form of it needs converters and a rule for which one is right. + +## Consequences + +- `rows.json` replaces `edits.json`. No code reads `edits.json`: a batch made before keeps its settings, and what was set by hand on it is lost. +- `RowTable`, the layers of `Row` (`values`, `info`, `index`), the marker `EMPTY`, "set by hand", and the four converters of cells go. `Header.columns` goes. +- A batch is shown without a scan and without its source. A batch cannot be shown in one case: the workspace binds no spec of its name. +- A wrong value in a cell is refused when it is set or read from a table file, not when the row is reduced. +- A change of the settings changes the request of each row with an empty cell at that parameter, and of no other row. +- A row that only wrong metadata made stays until a person removes it, and a correction of a run changes no batch (the example above). +- The command and the page need no translation of a table: a table file is the table. +- `put` writes the whole table. At 3000 rows that is tens of milliseconds. +- What ADR 0011 says of the columns in the header and of `edits.json` is replaced by this. What it says of the source, the scope and Follow stands until the ADR on amending. + +## Open + +- **A note cannot have the name of a parameter.** A column `sample` is the parameter `sample` of the spec, whatever the maker of the table meant by it. +- **One row cannot say `None` where the setting gives a value.** The setting has to say `None`, and the other rows give the value. +- **The order of the rows** is the order of the frame, and a new row goes to its end. Whether sorting on the page changes the stored order is not decided. +- **The blanks of a template.** The settings stay a template, the format of the settings file a person saves. Its blanks no longer say what the columns are. The command could still check that each blank is a column of the table it is given. diff --git a/docs/developer/adr/0012-a-batch-keeps-its-rows.md b/docs/developer/adr/0012-a-batch-keeps-its-rows.md deleted file mode 100644 index 53fc6721..00000000 --- a/docs/developer/adr/0012-a-batch-keeps-its-rows.md +++ /dev/null @@ -1,237 +0,0 @@ -# ADR 0012: A batch keeps its rows - -- Status: proposed (#236) -- Deciders: Simon -- Date: 2026-11-19 - -## Context - -*Batch*, *label*, *header*, *scope*, *source*, *definitions* are the words of [ADR 0011](0011-a-person-makes-the-batches-of-a-workspace.md). -*Row*, *key*, *cell*, *scan*, *edit*, *result* are those of [batch.md, "Terms"](../../../packages/essapps/docs/batch.md#terms). - -Today the rows of a batch are stored nowhere. They are computed at each scan, and only what a person set on top is kept: - -```python -scan = source.rows(files with corrections) # code of the application -rows = scan in scope, with edits on top # edits.json: added, edited, cleared, removed, by key -``` - -So a batch is defined by how its rows are made, and the table a person sees is the output of that recipe now. Five problems follow. - -1. **The table needs the files.** If the file mount is not available, the scan gives no rows, and the table is empty. The results are in the workspace, but no row shows them. If the listing raises, the page of the batch cannot be shown. -2. **Removing files removes rows.** A row whose file is gone disappears, also one that was reduced or edited. Its edits are kept and apply to nothing. -3. **A change of the source's code changes every batch**, also one that was finished last week. -4. **The model has no table to put a table in.** The command takes one (`essapps-batch reduce iofq.yaml runs.csv`), and the page loads and saves one. Both are translated into "cells set by hand on top of the scan", in four structures of different shape (`added`, `edited`, `cleared`, `removed`) and a marker for a cleared cell. The origin of a cell is worked out again in six places. - -5. **A cell is held in two forms.** A cell is a stand-in, never a resolved value: a file is named by an identity, an output by the ids of its record and output. But the stand-in is a pydantic object if a source or the page made it (`FileRef(file='run:60339')`), and plain data if a table file did (`{'@ref': 'file', 'file': 'run:60339'}`). Four functions convert between the forms (`_values`, `with_named_files`, `write_cell`, `read_cell`) and do not agree: a table file loaded back unchanged marks a cell that holds a reference to an output as set by hand. - -A person thinks of a batch as a table: they edit its cells, remove its rows, reduce its rows and read their results. All of that refers to rows, so the rows must last as long as what refers to them. - -## Decision - -**A batch keeps its rows, as plain data. The source and the scope are what fills the table, and only a person removes a row.** - -### What is kept - -One table per batch, `rows.json`, in place of `edits.json`. Per row: - -```python -key '60339' -index 60339 # what the scope compares -info {'name': 'D2O buffer'} -source {'sample': 'run:60339', 'transmission': 'run:60305'} # what the source gave, at the last scan that gave the row -hand {'thickness': 2.0} # what a person set -cleared ['transmission'] # the cells a person emptied -added False # True: made by a person, the source never gives it -removed False # True: removed by a person; kept so that a scan does not add it again -``` - -A cell holds two values, one per writer. The cell shown is the value set by hand if there is one, nothing if a person emptied it, else the source's: - -```python -cells = {c: v for c, v in {**row.source, **row.hand}.items() if c not in row.cleared} -# in pandas words: source.mask(by_hand, hand) -``` - -Two values are needed, not one value and a mark: "revert" shows the source's value again, and the page shows it under a cell set by hand. - -### A cell holds plain data - -A cell holds the plain form of a value, as the parameters of a request are held before they are resolved: text, a number, a truth value, `None`, or a list or dict of these. - -```python -'run:60339' # a file -'r17.centre' # an output of another record -2.0 # a thickness -None # "no direct beam": a value. An empty cell is a key left out. -['path:masks/beamstop.xml', 'path:masks/edges.xml'] # several files in one cell -{'start': 1.0, 'stop': 13.0, 'num_bins': 200, 'unit': 'Å'} # a group -``` - -- **A file and an output are text.** The field of the column says that the text names a file or an output, as it does for a file named in a table today. The text of an output is what `str(OutputRef)` gives: `r17.centre`, `r18.slices[2]`. -- **Every column that the page offers is then text or a number.** Of the 17 parameters of the six specs of LoKI and the demo that can be a column on the page, 9 are files, 7 are outputs, and 1 is a number. A group or a list can be a column only of a batch made in code or with the command. -- **A value has one plain form.** It is brought to it once, where it enters the table: from a source, from an edit on the page, from a table file. For a dict or a list, that form is what the field of the column dumps, so a group typed without its defaults and the same group with them are one cell. (A reference as a dict has two forms today, with and without `'key': None`, which compare unequal.) - -The field of the column validates a cell when a request is made, as today. The four converters go. Two cells are equal if their plain data are equal, and `rows.json` is the table as it is. - -### A scan is a merge - -```python -def scan(table, files): - given = {row.key: row for row in source.rows(files)} # the source sees every file - for key, row in given.items(): - if key in table: - table[key].update(index=row.index, info=row.info, source=row.values) # never `hand` - elif scope.holds(row): - table.add(row) - not_given = {key for key in table if key not in given and not table[key].added} -``` - -| The scan gives a row | The table has it | Then | -|---|---|---| -| yes, in the scope | no | the row is added | -| yes, outside the scope | no | nothing | -| yes | yes | `index`, `info` and `source` are replaced; `hand`, `removed` stay | -| no | yes | the row stays as it is, and is shown as *not in the files* | - -A scan that changes nothing writes nothing. A scan that cannot list the files changes nothing, and the page says why. - -*Not in the files* is not kept: it says what the last scan of this process found. A row not in the files keeps its cells, its result and its plot. A person can reduce it if its cells still resolve, and remove it. Follow leaves it alone. - -### The scope says which rows the batch shows and which rows a scan adds - -As in ADR 0011, the scope is a slice of the index, and a batch shows the rows inside it and the rows added by hand. New: the scan adds only rows inside the scope, so a batch stores its range of the rows, not every row of the source. - -- Widening the scope shows nothing new at once; the next scan adds the rows. -- Narrowing it hides the rows outside. They stay in `rows.json` with what was set by hand, so widening it again shows them as they were. - -### Only a person removes a row - -A removed row is kept, marked `removed`, so that the next scan does not add it again, and so that "Bring back" works as today. Nothing else removes a row: not a scan, not a correction of a run, not a change of the source's code. - -One action removes the rows that are not in the files, for the case below where a correction leaves a row behind. - -### "Start the next batch here" copies rows - -The next batch gets the rows from the index on, as they are: `source`, `hand`, `removed`. Then the scope of this batch ends there, which hides them here. Results stay under the label of this batch (ADR 0011). - -### A table goes in and out without loss - -```python -frame = batch.frame() # a DataFrame: the key as index, a column per column, then status and what was set by hand -batch.put(frame) # a cell that differs from the cell shown is set by hand; a key the batch lacks is added -batch.put(batch.frame()) # sets nothing: the contract, tested with every type of cell an application has -``` - -The CSV of the command and the table file of the page are this frame as text. The batch layer keeps rows of plain data, as JSON; `frames.py` stays the one place that converts to and from a `DataFrame` (see "pandas", below). - -## Examples - -A batch of sample runs with scope `[60300, None)`. - -**The files are not reachable.** - -``` -before mount gone, page opened -key sample transmission status key sample transmission status -60302 60302 60303 completed 60302 60302 60303 completed not in the files -60304 60304 60305 ready 60304 60304 60305 ready not in the files -``` - -Results and plots show. "Reduce" on 60304 is refused with the reason that its file is not found, as for any cell that does not resolve. - -**Two files are removed.** The two rows stay, shown as not in the files. Nothing else changes. - -**A run is corrected.** Run 60306 was recorded as a sample and is a transmission run. Before the correction the source gave a row `60306`; after it, it does not, and gives row `60304` the transmission run 60306. - -``` -key sample transmission status -60304 60304 60306 outdated # source cell replaced by the scan -60306 60306 - incomplete not in the files # stays until a person removes it -``` - -Today row `60306` disappears at the scan. Here it stays, marked, and "Remove the rows not in the files" removes it. This is the cost of the decision: a row that only wrong metadata made is not cleaned up for the person. - -**A cell is set by hand, then the file changes.** - -```python -batch.rows.edit('60304', {'transmission': 'run:60310'}) -# a correction makes the source give transmission 60306 for this row -# shown: 60310 (hand); under it: 60306 (source). Revert shows 60306. -``` - -**The command.** - -``` -essapps-batch reduce iofq.yaml runs.csv -l water # a batch without a source: the rows of runs.csv are its rows -essapps-batch reduce -s pkg:SOURCES -l 20261118-130204 -o done.csv -# edit done.csv in a spreadsheet, then: -essapps-batch reduce -s pkg:SOURCES -l 20261118-130204 done.csv # the cells changed are set by hand; the others are not -``` - -## pandas - -Developers of applications know pandas, so the table is given to them as a `DataFrame`, and the guide explains the model in its terms. - -- **`batch.frame()` and `batch.put(frame)`.** The frame holds the same data as the table: text and numbers. An empty cell is `NaN`; `None`, a value, is the text `null`, as in a table file. `put(frame())` sets nothing. -- **The guide states each concept with its pandas line**, which is literally true of the plain data: - -| Concept | For a pandas user | -|---|---| -| a row's cells | `source.mask(by_hand, hand)` | -| a scan | an outer merge on the key that overwrites `source`, never `hand` | -| the scope | `rows[(rows.run >= start) & (rows.run < stop)]` | -| the results | `records.sort_values('created').drop_duplicates('member', keep='last')` | -| the state | `cells.join(results)`, a left join on the key | -| a rule, such as `left_to_reduce` | a boolean mask over the state | -| the settings | a column with one value for all rows | - -### What the batch holds inside: rows of plain data - -Inside, the table is rows of plain data, each a frozen value with `source`, `hand` and `cleared`; a table is replaced, never changed in place. A `DataFrame` inside was built too (four frames on one index: the rows, `source`, `hand`, `cleared`) and run through the same session with equal tables after each step. With plain cells nothing that can be counted decides between them: - -| | rows of plain data | four frames | -|---|---|---| -| lines, as written | 152 | 138 | -| lines, formatted at 88 columns | 209 | 241 | -| the state, the rules, `frame()`, `put()` | 70 lines | 38 lines | -| plain data out and in (a request, a record's params, `rows.json`) | 4 lines | 16 lines | -| a scan, the state, saving, at 3000 rows | 15, 10, 24 ms | 15, 14, 44 ms | -| a frame handed out cannot change the one held | a copy is made | yes, by copy-on-write (72 cases; one exception, `combine_first` on `object` columns) | - -The choice is made on three facts that cannot be counted, and by a narrow margin: - -1. **Three partners of a batch take plain data and one takes a frame.** A request, the params of a record, and `rows.json` are plain data, at every reduction, state and save. A person takes a frame, on demand. Rows hold the form that the three use. -2. **A frame inside needs rules that nothing checks.** Four traps were hit in 138 lines, two of them silent: `combine_first` shares memory with its argument, so a write changed the table held; `.at` at a key that is no row adds a row; `from_dict` turns 200 into 200.0; `ne` says that two empty cells differ. `None` next to an empty cell needs the text `null` in the cells. A type checker reads a row; it reads nothing of a `pd.DataFrame`. -3. **`ess.apps.batch` imports without pandas today.** - -For the frame: the state and the rules read as the formulas of the table above. It is a sound second choice, and the one to take if `None` stops being a cell value of its own (it is one at 3 of the 17 columns), or if several workflows on one batch (#222) make the column the unit of work. - -## Alternatives considered - -- **Keep the rows derived, and cache the last scan for when the files are away.** Two models: what a person sees depends on whether the cache or the scan answered, and removing files still removes rows. -- **Keep the rows, and let a scan remove those it no longer gives.** That is today's behaviour with a cache. It fails the second problem. -- **Remove a row not in the files if it has no result and nothing set by hand.** It cleans up after a correction. Rejected: a row then disappears or stays by a rule a person has to know, and a mount that is away for a moment removes rows that come back at the next scan. One explicit action does the same when a person wants it. -- **The scope only says what a scan adds; narrowing it hides nothing.** Then the scope in the list of batches no longer says which rows a batch has, and "Start the next batch here" has to move rows out of one table and into the other. -- **Cells as typed objects** (`FileRef`, `OutputRef`), as a source makes them today. Two forms of one value, and converters between them; see the context. - -## Consequences - -- `rows.json` replaces `edits.json`; `Batch.rows` stays its one writer. No code reads `edits.json`: a batch made before keeps its header and settings, and what was set by hand on it is lost. -- `RowTable` holds rows with `source` and `hand` in place of the layers `scan`, `added`, `edits`, `removed`. The marker `EMPTY` for a cell emptied by hand goes; `cleared` names such cells. The six places that work out the origin of a cell read it from the row. -- A batch is shown without a scan. A source that raises, or that is no longer registered, is a message on the page of the batch, not a batch that cannot be shown. That leaves one such case: the workspace binds no spec of the batch's name. -- `manager.start_next` copies rows; the code that carries edits over by key goes. -- A batch stores its range of the rows. A change of scope that widens it needs a scan to show the new rows. -- Rows that only wrong metadata made stay until a person removes them (the example above). -- The size of `rows.json` is that of the table: a few hundred bytes per row, bounded by the scope. It is written when a scan or a person changes the table. -- The rules over the state (`ready`, `arrived`, `left_to_reduce`) are unchanged, except that `arrived` leaves a row not in the files. -- A source may give a cell as an object, such as a `FileRef`; the table holds its plain form. The four converters between the forms go, and a table file loaded back unchanged sets nothing by hand. - -## Open - -- **Whether `None` stays a cell value of its own.** `direct_beam` takes a file or `None`, "no direct beam", and an empty cell there makes the row incomplete. A named choice in its place would make every cell text or a number with nothing else to tell apart. -- **`put` of a frame that holds a row removed before** brings the row back. `frame()` leaves removed rows out, so this happens only if a person adds the key. - -- **Whether a removed row that is not in the files is dropped for good.** Kept here, for one rule less. The table then holds rows that nothing shows. -- **Rows shaped as a table.** A row has four kinds of fields (`key`, `index`, `values`, `info`), and five fields of `Source` say which is what. A table has columns with roles. This matters for several workflows on one batch (#222: a column has a third writer, a workflow), and is left for that decision. So is the name `index`, which means the key in pandas. -- **When the last scan was.** A page could say how old the table is when the files cannot be listed. Not needed for the decision. diff --git a/docs/developer/adr/index.md b/docs/developer/adr/index.md index dc041a5c..5d31a437 100644 --- a/docs/developer/adr/index.md +++ b/docs/developer/adr/index.md @@ -17,4 +17,4 @@ In reading order: - [ADR 0003: An accumulator's driver adds in place, and push n+1 waits for the readers of state n](0003-accumulators-add-in-place.md). `client.output` copies a state and makes no record, a request reads an accumulator only through the record of its freeze, and freeze needs no copy. - [ADR 0007: Distributions are split by who depends on them and when they are released](0007-packages-split-by-dependencies.md). essspec, essapps, and essinstruments; code that needs dependencies only some users install is a subpackage behind an extra. - [ADR 0011: A person makes the batches of a workspace, each with a scope over the rows of a source](0011-a-person-makes-the-batches-of-a-workspace.md). A header per batch, a manager that lists the batches, and at most one batch that follows. Several workflows on one batch are not decided. -- [ADR 0012: A batch keeps its rows](0012-a-batch-keeps-its-rows.md). Proposed: the rows of a batch are stored as plain data, a scan adds and updates them, and only a person removes a row. +- [ADR 0012: A batch is a table](0012-a-batch-is-a-table.md). Proposed: a batch is a header, settings and one table of rows with one value per cell, which only a person changes. How a batch is amended from the files is not decided. From 7021ae1c45837e028af913b16532f447847d7724 Mon Sep 17 00:00:00 2001 From: Simon Heybrock Date: Thu, 19 Nov 2026 09:50:47 +0000 Subject: [PATCH 4/6] ADR 0012: say that the settings are one dict, not a table "A value for every row" read as a value per row. The settings map a parameter to one value that all rows share. Co-Authored-By: Claude Fable 5.2 --- docs/developer/adr/0012-a-batch-is-a-table.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/developer/adr/0012-a-batch-is-a-table.md b/docs/developer/adr/0012-a-batch-is-a-table.md index ba2e17f5..afaf1a4b 100644 --- a/docs/developer/adr/0012-a-batch-is-a-table.md +++ b/docs/developer/adr/0012-a-batch-is-a-table.md @@ -34,8 +34,8 @@ A person thinks of a batch as a table: they edit its cells, remove its rows, red ``` batch '20261118-130204' header spec, title, description # what it is a batch of - settings {'wavelength': {...}, 'direct_beam': None} # a value for every row - rows one DataFrame # a value per row + settings {'wavelength': {...}, 'direct_beam': None} # one dict: a parameter's value, the same for all rows + rows one DataFrame # a table: a parameter's value for one row ``` ``` @@ -70,6 +70,7 @@ def request(key): return settings | cells.to_dict() # validated by the spec, as any request ``` +- The settings are one dict, parameter to value, with no keys of rows: a single row of defaults. `rows.fillna(settings)` would be the table with every parameter in every row. - An empty cell means "as in the settings". A row is incomplete if a required parameter has neither a setting nor a cell. - So a parameter is not either a setting or a column, fixed when the batch is made. A person adds a column for a parameter at any time: its cells are empty, every request is as before, and a value in one cell changes that row. - A cell does not hold `None`. Where rows differ in whether they have a value, the setting says `None` and the rows that have one give it: `direct_beam: None` in the settings, and a file in the cells of the rows with a direct beam run. From 942b55ede89dd6aeb521d69350805a45c47387f7 Mon Sep 17 00:00:00 2001 From: Simon Heybrock Date: Thu, 19 Nov 2026 09:57:59 +0000 Subject: [PATCH 5/6] ADR 0012: a batch is a template and a table The settings of a batch are a template, and "a cell overrides a setting" was a second rule next to the template's own: fill refuses a value for a parameter that the template fixes, and a blank has no value. The ADR now uses the template unchanged. Its blanks are the parameter columns of the table, a parameter is a setting or a column and never both, and each row fills the template once. An empty cell gives no value: the parameter takes its default, None if it takes None, and the row is incomplete otherwise. The template names the spec, so the header keeps the title and the description. Co-Authored-By: Claude Fable 5.2 --- docs/developer/adr/0012-a-batch-is-a-table.md | 107 +++++++++++------- docs/developer/adr/index.md | 2 +- 2 files changed, 64 insertions(+), 45 deletions(-) diff --git a/docs/developer/adr/0012-a-batch-is-a-table.md b/docs/developer/adr/0012-a-batch-is-a-table.md index afaf1a4b..e13318d0 100644 --- a/docs/developer/adr/0012-a-batch-is-a-table.md +++ b/docs/developer/adr/0012-a-batch-is-a-table.md @@ -1,4 +1,4 @@ -# ADR 0012: A batch is a table +# ADR 0012: A batch is a template and a table - Status: proposed (#236) - Deciders: Simon @@ -7,6 +7,7 @@ ## Context *Batch*, *label*, *header*, *scope*, *source*, *definitions* are the words of [ADR 0011](0011-a-person-makes-the-batches-of-a-workspace.md). +*Template*, *blank* and *fill* are those of `ess.apps.Template`: a spec, some values, and blanks that each use fills. *Row*, *key*, *cell*, *scan*, *result* are those of [batch.md, "Terms"](../../../packages/essapps/docs/batch.md#terms). Today the rows of a batch are stored nowhere. They are computed at each scan, and only what a person set on top is kept: @@ -29,21 +30,23 @@ A person thinks of a batch as a table: they edit its cells, remove its rows, red ## Decision -**A batch is a table. It is a label with a header, settings, and rows; the rows are one table with one value per cell, and only a person changes it. How the table was made is not part of the batch.** +**A batch is a template and a table. The template holds the settings, and its blanks are the columns; each row of the table fills it once. The table has one value per cell, and only a person changes it. How the table was made is not part of the batch.** ``` batch '20261118-130204' - header spec, title, description # what it is a batch of - settings {'wavelength': {...}, 'direct_beam': None} # one dict: a parameter's value, the same for all rows - rows one DataFrame # a table: a parameter's value for one row + header title, description + template spec loki-iofq + params {'wavelength': {...}, 'thickness': 1.0} # the settings: one value per parameter, for all rows + blanks ('sample', 'sample_transmission') # the parameters that each row gives + rows one DataFrame: a column per blank, and notes ``` ``` rows -key run name sample sample_transmission thickness -60339 60339 D2O buffer run:60339 run:60305 2.0 -60340 60340 sample A run:60340 -water by hand run:60012 run:60013 1.0 +key run name sample sample_transmission +60339 60339 D2O buffer run:60339 run:60305 +60340 60340 sample A run:60340 +water by hand run:60012 run:60013 ``` This decides what a batch is when it is made once and then changed by a person. How a batch is amended from the files, Follow included, is decided after it and on top of it (see "Not decided here"). @@ -51,7 +54,7 @@ This decides what a batch is when it is made once and then changed by a person. ### Keys, columns, cells - **The key** of a row is text, unique in the batch, and the index of the frame. It joins a row to its results: the records under the label of the batch with that key, as today. -- **A column named as a parameter of the spec is a parameter column.** Any other column, such as `run` or `name`, is a note for the person. Nothing else tells them apart, and the header keeps no list of columns. +- **A column named as a parameter of the spec is a parameter column**, and the parameter columns are the blanks of the template. Any other column, such as `run` or `name`, is a note for the person. Nothing else tells them apart. - **A cell holds one value, or is empty.** The value is the plain form of a parameter's value, as a request holds it before it is resolved. A file and an output are text: `run:60339`, and `r17.centre` for the output `centre` of record `r17`. - **A column has the type of its parameter**, so a wrong value is refused where it enters the table: @@ -62,41 +65,52 @@ This decides what a batch is when it is made once and then changed by a person. | a truth value | `boolean` | `True` | | a list, a group, or more than one of the above | `object` | `['run:1', 'run:2']` | -### A cell overrides a setting +### A row fills the template ```python def request(key): - cells = rows.loc[key, parameter_columns].dropna() - return settings | cells.to_dict() # validated by the spec, as any request + cells = rows.loc[key, list(template.blanks)].dropna().to_dict() + for parameter in template.blanks - cells.keys(): # the empty cells + if has_default(parameter): + cells[parameter] = default(parameter) + elif takes_none(parameter): + cells[parameter] = None + return template.fill(params=cells) # refuses "a blank, not filled": the row is incomplete ``` -- The settings are one dict, parameter to value, with no keys of rows: a single row of defaults. `rows.fillna(settings)` would be the table with every parameter in every row. -- An empty cell means "as in the settings". A row is incomplete if a required parameter has neither a setting nor a cell. -- So a parameter is not either a setting or a column, fixed when the batch is made. A person adds a column for a parameter at any time: its cells are empty, every request is as before, and a value in one cell changes that row. -- A cell does not hold `None`. Where rows differ in whether they have a value, the setting says `None` and the rows that have one give it: `direct_beam: None` in the settings, and a file in the cells of the rows with a direct beam run. +- **A parameter is a setting or a column, never both.** These are the rules of a template, unchanged: `fill` refuses a value for a parameter that the template fixes, and naming a parameter as a blank drops its value. So every value that the table contributes is in the table, and no cell depends on a setting. +- **The settings are one dict**, parameter to value, with no keys of rows: `template.params`. +- **An empty cell gives no value.** The parameter then takes its default, or `None` if it takes `None`, and the row is incomplete otherwise. A cell holds no word for `None`: in a table, empty is `None`. + +| A column of | An empty cell gives | +|---|---| +| `thickness: float = 1.0` | `1.0` | +| `direct_beam: DirectBeamFile \| None` | `None`: no direct beam | +| `sample: SampleFile` | nothing: the row is incomplete | ### A person changes the table A batch gives its table and takes one: ```python -frame = batch.frame() # the rows, as above -frame.loc['60340', 'thickness'] = 1.0 # set a cell -frame.loc['60340', 'thickness'] = pd.NA # empty it: as in the settings -frame.loc['d2o', 'sample'] = 'run:7' # add a row -frame = frame.drop('60339') # remove a row -frame['can'] = pd.NA # add a column, for the parameter `can` -batch.put(frame) # replaces the table -batch.put(batch.frame()) # changes nothing and writes nothing +frame = batch.frame() # the rows, as above +frame.loc['60340', 'sample_transmission'] = 'run:60341' # set a cell +frame.loc['60340', 'sample_transmission'] = pd.NA # empty it +frame.loc['d2o', 'sample'] = 'run:7' # add a row +frame = frame.drop('60339') # remove a row +frame['thickness'] = 1.0 # add a column: `thickness` becomes a blank +frame = frame.drop(columns='thickness') # remove it: `thickness` is a setting again, with no value yet +batch.put(frame) # replaces the table; the blanks follow its columns +batch.put(batch.frame()) # changes nothing and writes nothing ``` -`put` refuses the whole table for a key that is empty or used twice, and for a cell that does not have the type of its column. Each action of the page is `put` with one such line before it. A table file of the page and the CSV of the command are this frame as text. +`put` refuses the whole table for a key that is empty or used twice, and for a cell that does not have the type of its column. It keeps the blanks of the template equal to the parameter columns of the table, so the columns are said in one place. Each action of the page is `put` with one such line before it. A table file of the page and the CSV of the command are this frame as text. Removing a row drops it. Its records stay under the label, as every record does, and show again if a row with that key is added. ### Making a batch -The first rows are an argument of making the batch, as its first settings are. Where they come from is the business of whoever makes it: +The first rows are an argument of making the batch, as its first settings are; the parameter columns of the rows are its blanks. Where they come from is the business of whoever makes it: ```python frame = table_of(source.rows(files)) # rows from the files: code of the application @@ -120,8 +134,8 @@ A cell that names a file that is not found is marked when the table is shown, an ### What is kept ``` -definitions/labels/20261118-130204/header.json spec, title, description -definitions/labels/20261118-130204/settings.json a template: the settings +definitions/labels/20261118-130204/header.json title, description +definitions/labels/20261118-130204/settings.json the template, as a settings file that a person saves: spec, params, blanks definitions/labels/20261118-130204/rows.json the table: the plain data of frame.to_json(orient='table') ``` @@ -131,7 +145,7 @@ pandas writes and reads the table with the types of its columns. Checked on a ta ```python at = rows.index.get_loc('60400') -this, following = rows.iloc[:at], rows.iloc[at:] # the next batch gets `following`, and the same spec and settings +this, following = rows.iloc[:at], rows.iloc[at:] # the next batch gets `following`, and the same template ``` Results stay under the label of this batch (ADR 0011), so a row that moves has no result in the next batch. @@ -147,15 +161,21 @@ Results stay under the label of this batch (ADR 0011), so a row that moves has n **A parameter becomes a column.** ```python -settings['thickness'] # 1.0, for every row -frame['thickness'] = pd.NA # a column; every request is as before, no row is outdated -frame.loc['60340', 'thickness'] = 2.0 # row 60340 is outdated; the others are not +template.params['thickness'] # 1.0, for every row +frame['thickness'] = 1.0 # a column with the value of the setting in every cell +batch.put(frame) # `thickness` is a blank; every request is as before, no row is outdated +frame.loc['60340', 'thickness'] = 2.0 +batch.put(frame) # row 60340 is outdated; the others are not ``` +The page does the first two lines for "Make this a column". The settings form no longer shows `thickness`. + +**Some rows have a direct beam run.** `direct_beam` takes a file or `None`. As a column, the rows with a run name it, and the cells of the others are empty. + **The command.** ``` -essapps-batch reduce iofq.yaml runs.csv -l water # makes the batch `water`: iofq.yaml its settings, runs.csv its rows +essapps-batch reduce iofq.yaml runs.csv -l water # makes the batch `water`: iofq.yaml its template, runs.csv its rows essapps-batch reduce -l water -o done.csv # the table, with a status and a record per row # edit done.csv in a spreadsheet, then: essapps-batch reduce -l water done.csv # done.csv replaces the table, and its rows are reduced @@ -170,8 +190,8 @@ The batch holds its rows as a `DataFrame`, so pandas becomes a dependency of `es | Concept | In pandas | |---|---| | a row, a key | a row of the frame, its index | -| a parameter column | `[c for c in rows.columns if c in parameters]` | -| the request of a row | `settings \| rows.loc[key, parameter_columns].dropna().to_dict()` | +| the parameter columns | `[c for c in rows.columns if c in parameters]`, equal to `template.blanks` | +| the request of a row | `template.fill(params=rows.loc[key, blanks].dropna().to_dict())` | | set, empty, add, remove | `.loc[key, column] = value`, `= pd.NA`, `.loc[key, column] = value` at a new key, `.drop(key)` | | the results | `records.sort_values('created').drop_duplicates('member', keep='last')` | | the state | `rows.join(results)`, a left join on the key | @@ -200,27 +220,26 @@ Until then the parts of ADR 0011 on the source, the scope and Follow stand, and - **Keep the rows derived, and cache the last scan for when the files are away.** Two models: what a person sees depends on whether the cache or the scan answered, and removing files still removes rows. - **Two values per cell, one that the source gave and one that a person set, and a scan that merges.** It stores the inputs of a merge in place of the table. Each row then carries what only a scan needs (`source`, `hand`, `cleared`, `added`, `removed`), and the table is not one that pandas holds or stores. It also decides the three uses of a scan above with "yes" without asking. -- **A parameter is a column or a setting, fixed at making** (today). The header then lists the columns, a column cannot be added, and a cell needs a form for `None`. -- **A list of the parameter columns in the header.** A second place that says what the columns of the frame say. -- **`None` as the value of a cell**, as the text `none`. Not needed with a cell that overrides a setting, and a text column would hold a word that is not text of its parameter. +- **A cell overrides a setting**: `settings | row.dropna()`, with an empty cell meaning "as in the settings". A second rule next to that of a template for the same two things, values for all uses and values of one use. An empty cell then depends on something outside the table. +- **The columns fixed when the batch is made** (today). Nothing needs it: a row is outdated by the comparison of its request with its record, whatever the columns were. +- **A list of the parameter columns in the header.** The template has it: its blanks. +- **A word for `None` in a cell**, such as the text `none`, so that an empty cell of `direct_beam` can mean "not said yet". A text column would hold a word that is not text of its parameter, and pandas reads empty as missing in every operation. - **Cells as typed objects** (`FileRef`, `OutputRef`). Two forms of one value, and converters between them; see the context. - **Rows of plain data inside the batch, and a `DataFrame` only at its interface.** With one value per cell the frame is the table, and a second form of it needs converters and a rule for which one is right. ## Consequences - `rows.json` replaces `edits.json`. No code reads `edits.json`: a batch made before keeps its settings, and what was set by hand on it is lost. -- `RowTable`, the layers of `Row` (`values`, `info`, `index`), the marker `EMPTY`, "set by hand", and the four converters of cells go. `Header.columns` goes. +- `RowTable`, the layers of `Row` (`values`, `info`, `index`), the marker `EMPTY`, "set by hand", and the four converters of cells go. `Header.spec` and `Header.columns` go: the template names the spec, and its blanks are the columns. - A batch is shown without a scan and without its source. A batch cannot be shown in one case: the workspace binds no spec of its name. - A wrong value in a cell is refused when it is set or read from a table file, not when the row is reduced. -- A change of the settings changes the request of each row with an empty cell at that parameter, and of no other row. +- A spec cannot make a person choose between a value and `None` for each row: an empty cell of such a column is `None`. It still can for a setting. - A row that only wrong metadata made stays until a person removes it, and a correction of a run changes no batch (the example above). - The command and the page need no translation of a table: a table file is the table. - `put` writes the whole table. At 3000 rows that is tens of milliseconds. -- What ADR 0011 says of the columns in the header and of `edits.json` is replaced by this. What it says of the source, the scope and Follow stands until the ADR on amending. +- What ADR 0011 says of the spec and the columns in the header and of `edits.json` is replaced by this. What it says of the source, the scope and Follow stands until the ADR on amending. ## Open - **A note cannot have the name of a parameter.** A column `sample` is the parameter `sample` of the spec, whatever the maker of the table meant by it. -- **One row cannot say `None` where the setting gives a value.** The setting has to say `None`, and the other rows give the value. - **The order of the rows** is the order of the frame, and a new row goes to its end. Whether sorting on the page changes the stored order is not decided. -- **The blanks of a template.** The settings stay a template, the format of the settings file a person saves. Its blanks no longer say what the columns are. The command could still check that each blank is a column of the table it is given. diff --git a/docs/developer/adr/index.md b/docs/developer/adr/index.md index 5d31a437..f6237027 100644 --- a/docs/developer/adr/index.md +++ b/docs/developer/adr/index.md @@ -17,4 +17,4 @@ In reading order: - [ADR 0003: An accumulator's driver adds in place, and push n+1 waits for the readers of state n](0003-accumulators-add-in-place.md). `client.output` copies a state and makes no record, a request reads an accumulator only through the record of its freeze, and freeze needs no copy. - [ADR 0007: Distributions are split by who depends on them and when they are released](0007-packages-split-by-dependencies.md). essspec, essapps, and essinstruments; code that needs dependencies only some users install is a subpackage behind an extra. - [ADR 0011: A person makes the batches of a workspace, each with a scope over the rows of a source](0011-a-person-makes-the-batches-of-a-workspace.md). A header per batch, a manager that lists the batches, and at most one batch that follows. Several workflows on one batch are not decided. -- [ADR 0012: A batch is a table](0012-a-batch-is-a-table.md). Proposed: a batch is a header, settings and one table of rows with one value per cell, which only a person changes. How a batch is amended from the files is not decided. +- [ADR 0012: A batch is a template and a table](0012-a-batch-is-a-table.md). Proposed: the template holds the settings and its blanks are the columns; the rows are one table with one value per cell, which only a person changes. How a batch is amended from the files is not decided. From 66adb3f070fceb7bbe9b329d226d14336e1fb5ec Mon Sep 17 00:00:00 2001 From: Simon Heybrock Date: Thu, 19 Nov 2026 11:00:19 +0000 Subject: [PATCH 6/6] ADR 0011 points to ADR 0012 for the parts it replaces Co-Authored-By: Claude Fable 5.2 --- .../adr/0011-a-person-makes-the-batches-of-a-workspace.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer/adr/0011-a-person-makes-the-batches-of-a-workspace.md b/docs/developer/adr/0011-a-person-makes-the-batches-of-a-workspace.md index 82aff084..420e7275 100644 --- a/docs/developer/adr/0011-a-person-makes-the-batches-of-a-workspace.md +++ b/docs/developer/adr/0011-a-person-makes-the-batches-of-a-workspace.md @@ -1,6 +1,6 @@ # ADR 0011: A person makes the batches of a workspace, each with a scope over the rows of a source -- Status: accepted. Several workflows on one batch are described below and not decided. Amended 2026-11-18: the header holds the spec and the columns. Amended 2026-11-19: the manager keeps which batch follows; "Start the next batch here" leaves Follow where it is. +- Status: accepted. Several workflows on one batch are described below and not decided. Amended 2026-11-18: the header holds the spec and the columns. Amended 2026-11-19: the manager keeps which batch follows; "Start the next batch here" leaves Follow where it is. [ADR 0012](0012-a-batch-is-a-table.md), proposed, replaces what is said here of the spec and the columns in the header and of `edits.json`. - Deciders: Simon - Date: 2026-11-18