Skip to content

Consolidate after two adversarial reviews: the manager owns the rules and which batch follows - #235

Merged
SimonHeybrock merged 17 commits into
mainfrom
224-next-batch-in-manager
Nov 19, 2026
Merged

SimonHeybrock merged 17 commits into
mainfrom
224-next-batch-in-manager

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

Second consolidation of the stack of #224, on top of #234. Two adversarial reviews read the whole stack against a one-page statement of the architecture (a picture and twelve claims). Two claims were false and four held only by convention. This pull request makes them true, mostly by deleting: 35 files, +1309/−1710; manager.py 539 → 482 lines; the class Follow, the type Defaults, the scope of keys and two preview routes are gone.

The picture after it

   person, browser                      person, shell              notebook
        │ HTTP                               │                        │
 ┌──────┴──────────────────────┐             │                        │
 │ web: create_app, Pages, Page│      ┌──────┴───────┐                │
 └──────┬──────────────────────┘      │ cli          │                │
────────┼─────────────────────────────┴──────┬───────┴────────────────┼────────
        ▼                                    ▼                        │
      BatchManager ◄─────────────────────────┘                        │
        │  label → Header          (kept)                             │
        │  the label that follows  (kept)                             │
        │  label → Batch           (made once per process)            ▼
        ▼                                                  Batch made in code:
      Batch ── Source, Client, DefinitionStore              no header, not listed,
                                                            keeps nothing
manager = BatchManager(client, sources=SOURCES, definitions=definitions,
                       settings={IOFQ.name: SETTINGS}, staged={IOFQ.name})
label = manager.create(IOFQ.name, title='Morning', source='sample-runs', scope=Scope(start='60339'))
manager.follow(label)                                  # this batch follows, and no other
manager.following                                      # its label, kept in the definitions
nxt = manager.start_next(label, 60421, title='Afternoon')   # ends one batch, makes the next from there
manager.remove(label)                                  # also ends Follow, if that batch followed
Automatic(manager.batch(manager.following)).step()     # a step of Follow; whoever serves owns the timer

What changed, and the finding behind it

Finding Change
"Batch" meant two things: a batch of the workspace, and the object Batch, which wrote definitions under any label a batch is a label with a header; only the manager gives a Batch its definitions; a Batch made in code keeps nothing
which batch follows was kept consistent in four places, and Follow could be wired to another manager or other definitions the manager keeps it (following, follow(label)); remove ends it; Pages(manager, interval=...) steps it
"Start the next batch here" lived in the web presenter manager.start_next
it moved Follow, which reduced the rows that moved again, with the same settings, before a person could change them Follow stays where it is; a person turns it on for the next batch
a refusal was KeyError, ValueError or NotKept by case, so the command checked beforehand a refusal of the manager is a ValueError; KeyError only for a label without a batch
the ends of a scope were converted in the page and again in the manager Scope is a plain pair; the manager converts and checks, once
a scope of keys: nothing but tests made one; a branch in seven files removed
clean-up for definitions that refuse a write removed; a refused write raises. One clean-up stays, in start_next
two previews behind confirmation dialogs; one copied Follow's rule and miscounted removed
counts() and its result type on the manager, with one caller moved to Pages
Defaults: two lifetimes in one type settings= and staged= of the manager
bound_spec exported from the batch package Client.spec_named(name)

Three changes to what ADR 0011 says

The ADR is amended in place. Each is a decision I took after the reviews; object if wrong.

  1. "Start the next batch here" does not move Follow. For "new settings after lunch", moving it cost three reductions per row, two of them wasted.
  2. A scope is a slice only. A row without an index is held only by a batch without a start and a stop.
  3. The manager keeps which batch follows. The ADR rejected that because "the manager would get a thread, a timer and a lock". What was built has none of them: a kept label and one step.

Declined

  • Source keeps the words for its rows: they describe the source, as a title describes a spec.
  • "No source" stays None.
  • settings.json stays a template, the format of the settings file a person saves; its blanks are not read.
  • Batch(definitions=) stays a parameter, used by the manager only.

To try

  • python -m ess.apps.batch.web.demo --arrive 4 --interval 2: Follow on; "Start the next batch here"; Follow is still on the first batch; turn it on for the next.
  • Type text as the start of a scope, in "New batch" and in the "Batch" tab: refused with a message.

Part of #224.

🤖 Generated with Claude Code

SimonHeybrock and others added 17 commits November 19, 2026 05:13
"Start the next batch here" decided in the web presenter what can be
split and how: a scope of keys has no end to set, the batch must keep a
row of the scan, the next batch takes [index, old stop) with the cells
set by hand and the removals of the rows that move, and it is removed
again if it cannot take them. A notebook or the command could not use
any of it.

BatchManager.next_scope and BatchManager.start_next hold these rules
now; NotEnded tells the caller that the next batch was made but the
batch before it was not ended. The page finds the index of the row,
counts what moves, moves Follow and says the refusals in its words.
The tests of the rules move to the manager.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Which spec a name stands for, the highest version that the workspace
binds, is a fact of the workspace, used by the manager, frames, the
command and the pages. Client.spec_named gives it next to Client.spec
and Client.specs, and bound_spec of the manager module goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope(keys=...) named the rows of a batch by key. Nothing but tests made
one, yet it had a branch in the table, the header data, the manager, the
page, the list of batches and the browser. A scope is now a slice of the
index; a row without an index is held only by a batch without a start and
a stop. The ADR says that naming rows by key is not built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope refused a stop below the start before any conversion, so
Scope(start='9', stop='10') raised and the web layer had to convert the
ends (scoped) before the manager converted them again (_typed). A Scope is
now a pair of ends, checked only for their type; scope_of in the manager
converts them to the type of the index of the source and then refuses a
stop below the start, in create, change and next_scope. The page and the
list of batches pass the ends as typed and show what the manager refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The manager refused with KeyError (a spec not bound, a source not
registered, a kept column the bound spec lacks), ValueError or NotKept by
case. Callers caught all three, and the command checked beforehand to tell
a refusal from a bug. The docstring of BatchManager states one contract:
every refusal of what is asked is a ValueError with a message for a
person; KeyError only for "no batch has this label"; NotKept when the
definitions refuse a write; anything else is an error of the code of the
application. _source and _bound are the one place for the first two
refusals, in create, batch and change.

The command drops its checks beforehand. It checks a template with
text.checked_settings, as the page's "Load settings" does, so a template
of another version of the spec is refused by both. It adds the -s/--sources
hint to the manager's refusal of an unregistered source, and shows NotKept
and a folder held by another process as a message, not a traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Page.added counted the rows that a wider scope would make Follow reduce,
for a confirmation before Save; it copied Follow's rule (arrived) and
miscounted it. Turning Follow on for a batch with many rows asks nothing,
so asking only before a scope change was inconsistent. Page.moved counted
the rows that "Start the next batch here" moves, for the text of its
prompt, and validated the row a second time before start_next did.

Both go with their routes, their model and their dialogs. Save saves;
"Start the next batch here" asks for the title and calls the manager's
start_next once, which shows what it refuses. The text under the button
says that the results of the rows that move stay in this batch.
next_scope is inlined into start_next, and scope_of becomes private.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The definitions refuse a write only when the disk does, such as when it is
full; the next action fails as well, and the person learns it from the
first message. Clean-up code for that case goes: NotEnded, the deletion of
the settings in create when the header is refused, the two messages and
the finally of Pages.remove, and the messages of Page.start_next for
partial states. A refused write raises NotKept, which the page shows as
"The change was not kept". The order of writes stays: write first, then
hold in memory.

One clean-up stays: start_next removes the next batch again if making or
scanning its Batch, or giving it the edits, raises, since a batch that
cannot be shown cannot be removed on the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BatchManager.counts had one caller, Pages.counts, which turned its result
type Counts into the statuses of the page. It is a view over the public
interface of the manager (headers, batch, sources, client), not a rule of
the batches, so Pages.counts does the loop itself; Counts and _said go.
A batch that the manager had not made yet is scanned twice by a count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Defaults held two things of two lifetimes: the settings that a new batch
of a spec starts with, read once by create, and whether a spec runs
staged, read each time the manager makes a Batch. They are now the
arguments settings and staged of BatchManager. The fallback "a header
without settings.json gets the settings registered" goes: create always
writes the settings before the header, so a batch has them. LoKI, the
demo, the command and the docs follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Batch made in code with definitions was a second writer of the settings
and edits of a label, which the manager does not know of. The manager is
now the only code that passes definitions to a Batch: iofq_batch of LoKI
loses them, the guide shows the manager in "Where the definition is
kept", and the test that built a second Batch with definitions on a
manager's label goes; the test of settings kept at a column writes the
file as a person would.

settings.json was described as holding a copy of the header's spec and
blanks; reading it checks the name of its spec, and neither its version
nor its blanks are read, since the header says the columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- create does not check fits: _bound refuses a column that is not a
  parameter, and check_fills a source that fills one that is not a
  column, so the check refused nothing that they let through. fits stays
  for the New batch form.
- Automatic.reduce had one caller, step, and one test; it is folded into
  step and the test goes with it.
- A generated label is not checked by check_label and _taken again.
- ess.apps.batch exports what applications, the guide and the tests use;
  EARLIER, Index, compared and fits go. page.py and pages.py import from
  the modules of the package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`BatchManager.following` and `BatchManager.follow(label)` replace the
label that `Follow` kept. The manager reads the label from the
definitions when it is made, counts a label without a header as none,
refuses a batch without a source or with a source not registered
(ValueError), and `remove` ends it. So "the batch that follows exists"
holds in one place: the self-off of the step, the Follow lines of
`Pages.remove` and the second source check in `Page.follow` go.

What remained of `Follow` was one step of `Automatic` on
`manager.batch(manager.following)` and why it refused rows. That step is
now `Pages.tick`, which keeps `refused` next to `failure`, so the class
goes, and with it the `definitions` an application could give it apart
from the manager's. `Pages(manager, *, interval=...)`: the interval is of
whoever owns the timer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moving Follow to the next batch made its next step reduce every row that
moved, again, with the settings copied from the batch that ended, before
a person could change them. Rows that move have no result under the new
label, so the step took them all. `Page.start_next` now only calls the
manager; the next batch opens with Follow off, and the hint under the
button says to turn it on there once its settings are right.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The state carried a version of the list of batches, and the browser asked
for `api/batches` when it moved. The version of a page already moves when
the headers of the batches or the batch that follows change, so the state
now carries the list itself (tens of rows, built from the headers), and
`Pages.batches_version` with its two fields goes. Only the page without a
batch asks for `api/batches`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR 0011 is amended in place: the manager keeps which batch follows, a
step is one step of `Automatic` on it, whoever serves owns the timer, and
"Start the next batch here" leaves Follow where it is, with the reason.
The rejected alternative "Follow in the manager" now says what was built.

batch.md defines a batch (a label with a header) against `Batch` (the
object that works on the rows of one label) and the definitions once, in
"Terms". "The batches of a workspace" follows the code (`follow`,
`following`, `start_next`, `settings=`/`staged=`), says that a label is
used once and what a restart keeps, and drops the table that restated
what `manager.batch()` gives a `Batch`. "Automatic reduction" states
Follow's rule as a rule on the state and lists the actions of a person
that make it reduce rows; the module docstrings of `auto.py` and
`page.py` say the same.

The docstrings of LoKI's and the demo's `build` say that they make a
first batch whenever no batch is listed, also after a person removed the
last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A removal that the definitions refuse left the batch listed but ended
  Follow for good, since `remove` wrote `following` first. The header goes
  first now; a kept label without a header already reads as none.
- `start_next` refuses an index at the stop of the batch, which made a next
  batch that could never hold a row, and a refused clean-up of the next
  batch no longer hides the error that caused it.
- The reasons why Follow refused rows are kept with the label of the batch
  they were given for, so a batch that follows next, also by
  `manager.follow` from code, never shows them; `Page.follow` no longer
  empties them.
- A template of another version is refused with what to do about it.
- The count of the batches scans a batch not made yet twice: said in its
  docstring, and the test pins it instead of making the batches first.
- Tests at the page: a header change and Follow refused by the
  definitions answer 422 and change nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The demo and LoKI make a first batch at a start when no batch is listed,
not whenever none is. batch.md: a `Batch` made in code writes nothing,
"batch" and `Batch` each in the one sense of "Terms", the stored tables
include `following`, and a template must be of the version bound. The
`reduce` bullet of the command's docstring is one bullet again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock self-assigned this Nov 19, 2026
Base automatically changed from 224-command-header to main November 19, 2026 11:13
@SimonHeybrock
SimonHeybrock merged commit 0144e8a into main Nov 19, 2026
2 checks passed
@SimonHeybrock
SimonHeybrock deleted the 224-next-batch-in-manager branch November 19, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant