Repository navigation
ADR 0012: a batch is a template and a table - #237
Conversation
Proposed. The rows of a batch are computed from the files at each scan and stored nowhere, so the table needs the files, removing files removes rows, and a table cannot go in and out without loss. The ADR proposes to keep the rows as plain data, with a value from the source and a value set by hand per cell, to make a scan a merge, and to let only a person remove a row. It says how a pandas user gets the table (frame, put) and which alternatives were considered. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A second study built both cores with plain cells. Of the columns that the page offers, all are files, outputs or numbers, so with a text form for an output every such cell is text or a number, and the open point on the form of a file is settled the same way. A value has one plain form: a reference as a dict had two that compare unequal. With plain cells nothing countable decides between rows of plain data and a DataFrame inside the batch. The ADR chooses rows by a narrow margin and says on which facts, and when the frame would be the one to take. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
| key '60339' | ||
| index 60339 # what the scope compares |
There was a problem hiding this comment.
Do this ever differ?
There was a problem hiding this comment.
Good question. In both real sources the key is the text of the index. They differ only for a run without a number and for a row made by a person, which have no index.
In the rewritten ADR there is no index: the run number is an ordinary typed column (run) that whoever makes the table gives, and a range of runs is frame[frame.run >= start] when the batch is made.
| info {'name': 'D2O buffer'} | ||
| source {'sample': 'run:60339', 'transmission': 'run:60305'} # what the source gave, at the last scan that gave the row | ||
| hand {'thickness': 2.0} # what a person set | ||
| cleared ['transmission'] # the cells a person emptied | ||
| added False # True: made by a person, the source never gives it | ||
| removed False # True: removed by a person; kept so that a scan does not add it again |
There was a problem hiding this comment.
I feel something is wrong with this. Unclear why a lot of info here is kept per-cell.
remove-- why do we need a scan once we have a batch table? If this is about follow-mode and adding new rows from that, then it is odd to have to influence our data structure / storage format.- same for
added cleared-- why can that not be encoded in the value?source/hand/info-- why does this matter once the columns are created? If it matters, why is it stored on a per-row level (in the table), instead of e.g., a bunch of flags or metadata about columns (if there is really a need to know).- This layout seems hard to read and inefficient. Why can the "table" part of the batch not just be serialised by Pandas, alongside some other metadata/info bespoke to our
Batch?
There was a problem hiding this comment.
Agreed, the layout stored the two inputs of a merge (what the source gave, what a person set) in place of the table. All of it was there for a scan that runs again.
Rewritten from the opposite end: what is a batch if it is made once and then changed by a person? A template (the settings; its blanks are the columns) and one DataFrame with one value per cell, stored by pandas. source, hand, cleared, added, removed, info and index are gone. An empty cell is the cleared cell.
What a scan needs (which source, which range, which rows a person removed) is left to an ADR on amending a batch, with the condition that it is state next to the batch, not part of the table.
The first version kept two values per cell, one from the source and one set by hand, and marks per row for a scan that runs again. Review asked why a table carries what only a scan needs. The ADR now decides what a batch is when it is made once and changed by a person: a header, settings, and one DataFrame of rows with one value per cell. A cell overrides a setting, so a parameter can become a column at any time and a cell needs no form for None. The source and the range of rows are arguments of making a batch. pandas holds and stores the table. How a batch is amended from the files, Follow included, is left to an ADR of its own, with two conditions that follow from this one. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
"A value for every row" read as a value per row. The settings map a parameter to one value that all rows share. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The settings of a batch are a template, and "a cell overrides a setting" was a second rule next to the template's own: fill refuses a value for a parameter that the template fixes, and a blank has no value. The ADR now uses the template unchanged. Its blanks are the parameter columns of the table, a parameter is a setting or a column and never both, and each row fills the template once. An empty cell gives no value: the parameter takes its default, None if it takes None, and the row is incomplete otherwise. The template names the spec, so the header keeps the title and the description. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
ADR 0012, proposed: a batch is a template and a table. A decision to review; no code. On top of #235. Part of #236.
Rewritten after the first review, which asked why a table carries what only a scan needs.
The question
What is a batch if it is made once and then changed by a person? The ADR answers that first. How a batch is amended from the files, Follow included, is decided on top of it, in ADR 0013 (the pull request on top of this one).
The decision proposed
template.fill(params=row.dropna()), with the rules ofTemplateunchanged. A parameter is a setting or a column, never both.batch.frame()gives the table andbatch.put(frame)replaces it.frame['thickness'] = 1.0.Noneif it takesNone, and the row is incomplete otherwise.What changes for a person
Nonefor each row; an empty cell of such a column isNone.To decide in review
Nonewhere the parameter takesNone.🤖 Generated with Claude Code