Repository navigation
A table as a frame: dtypes, cells and exact storage (ADR 0012) - #239
Merged
Merged
Conversation
table.py gains the frame form of ADR 0012 next to RowTable, with no
caller yet:
- dtypes(model): the dtype of the column of each parameter.
- typed(frame, model): the frame as a batch holds it; refuses a key
that is not text, empty or given twice, and a cell not of its
column's dtype, naming each.
- as_request / cell_of: the one place where a cell (text for a file
or an output) and a value of a request are converted.
- data_of / frame_of: the plain data of rows.json. pandas' to_json
rounds floats and integers above 2**53, so the records are Python
values and each column is read back with pd.array.
ADR 0012 and 0013 are accepted. ADR 0012 now states how rows.json is
written and read, takes a row with to_dict('index') (a row as one
Series turns Int64 into float), converts files and outputs with
as_request before fill, and names rows_of; ADR 0013 names rows_of.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
From review of the frame form: - A cell must be of the kind of its column, with no conversion by pandas: string takes text, Float64 an int or a float, Int64 an int within 64 bits, boolean a bool, object plain data. A float NaN is empty. So text 'nan' cannot become a value that rows.json empties, and True is not a number. - typed refuses a column named 'key' (data_of would overwrite the keys with it), a column not named by text, and a note whose dtype is not one of the five, which rows.json would change silently. - frame_of builds each column with the same rule as typed, refuses a field given twice, and names row and column in its errors. - as_request leaves empty text at a file for the request to refuse. - cell_of gives the JSON form of anything that is not a reference or a list, such as a group or a time. - dtypes refuses a parameter that takes text or an output, since a cell cannot tell the two apart; an enum of text and a date are text. ADR 0012 says a time is text, that no column is named 'key', and that its read_csv example holds for tables of files and text only. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Plain pandas gives whole numbers with a gap as a float column
(pd.DataFrame({'count': [1, None]})), so a column of whole numbers
takes 1.0 and keeps it as 1. 2.5, inf and a bool stay refused.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of ten pull requests for #236, which implement ADR 0012 (a batch is a template and a table) and ADR 0013 (Follow adds the rows that a batch has not considered). They are merged one by one, in order, not stacked.
This one adds the frame form of a table to
ess.apps.batch.table, with no caller yet, and sets both ADRs to accepted.ADR 0012 corrected
Probes showed three lines of its code false. The decision is unchanged.
to_json(orient='table')andread_jsonround floats (0.1 + 0.2reads back as0.3) and integers above 2**53. Every row would then read as outdated after a restart.rows.jsonkeeps the same shape, but its records are written as Python values and each column is read back withpd.array(values, dtype).rows.loc[key, blanks]gives oneSeries, so anInt64cell becomes a float when all blanks are numbers. A row is read withto_dict('index').as_request, beforeTemplate.fill.The dtype table of the ADR now says what the code does for a union: a file or an output is
string; a value or an output isobject.import ess.apps.batchneeds pandas from here on.🤖 Generated with Claude Code