Repository navigation
[RF][HS3] Axis handling: categorical axes and HS3 bin order (HS3 #108, #126; depends on #23840) - #24001
Draft
cburgard wants to merge 12 commits into
Draft
[RF][HS3] Axis handling: categorical axes and HS3 bin order (HS3 #108, #126; depends on #23840)#24001cburgard wants to merge 12 commits into
cburgard wants to merge 12 commits into
Conversation
This was referenced Nov 5, 2026
HS3 PR "json-fixes-0.3" defines `hs3_version` as a `"MAJOR.MINOR"` string and the version of the revised standard as `"0.3"`. RooFit now writes this version. Documents with a newer version are imported with a warning. This commit depends on the HS3 PR and should only be merged together with or after it. Assisted-by: claude-code:claude-opus-5-5
HS3 PR "json-fixes-0.3" renames the analysis keys to `parameters_of_interest` and `domains` (a list of domain names), which is what RooFit already writes, including one entry each for the nuisance parameter, global observable and parameter-of-interest domains. The keys of HS3 v0.2.9, `parameter_of_interest` and `domain` (a single name), are now also read, with a deprecation warning. The plural keys take precedence. Assisted-by: claude-code:claude-opus-5-5
…f writing null RooFit wrote infinite bounds of product-domain axes as JSON `null`, which HS3 doesn't allow. HS3 PR "json-fixes-0.3" makes `min` and `max` of domain axes optional, with an absent bound meaning an unbounded range. Infinite bounds are now omitted on export. Since that PR also requires both bounds for a binning, the default binning is only written for bounded axes. Axes without any bound are still written (with only their `name`), which the HS3 PR keeps valid. RooFit needs them to tell an unbounded RooRealVar from a RooConstVar. On import, an absent bound means -inf/+inf, and `null` is still accepted for files written by older ROOT versions. The RooGaussianConstVarSigmaExport and ProductDomainEntriesExportExplicitBounds tests are updated to expect omitted instead of null bounds. Assisted-by: claude-code:claude-opus-5-5
…ll_dist and crystalball_doublesided_dist HS3 PR root-project#135 defines `crystalball_dist` as the single-sided Crystal Ball (`sigma`, `alpha`, `n`, as RooCBShape) and the new `crystalball_doublesided_dist` as the double-sided one (`alpha_L`, `n_L`, `alpha_R`, `n_R`, and `sigma` or `sigma_L`/`sigma_R`, without a symmetric shorthand). * RooCBShape is exported as `crystalball_dist`, as before. * A single-sided RooCrystalBall is exported as `crystalball_dist`, and read back as a RooCBShape (same function, same sign convention for alpha). Before, it was written as an incomplete `crystalball_doublesided_dist` without `alpha_R`/`n_R`, which could not be read back. * A double-sided RooCrystalBall is exported as `crystalball_doublesided_dist`, with `sigma` if both sides share it, and the tails always as `alpha_L`/`n_L`/`alpha_R`/`n_R`. * `crystalball_doublesided_dist` is imported as RooCrystalBall, also with a single `sigma`, and invalid field combinations are rejected. * `crystalball_dist` with any `_L`/`_R` field is the double-sided form of HS3 versions before 0.3. It is still read as RooCrystalBall (including `alpha`/`n` for both tails with `sigma_L`/`sigma_R`), with a one-time deprecation warning. The `crystalball_doublesided_dist` factory expression and the RooCrystalBall export keys are removed, since the new importer and exporter handle all cases. Assisted-by: claude-code:claude-opus-5-5
…on't contribute to the staterror HS3 PRs root-project#135 and root-project#124 make the "errors" of a HistFactory sample optional: a sample without "errors" contributes no statistical uncertainty to a staterror modifier. RooFit used the content as squared error instead, i.e. assumed Poisson uncertainties. That is kept for documents before HS3 0.3. In HS3 >= 0.3 documents, this only matters if the staterror is constrained through the legacy "constraint" key, since otherwise the gammas are either unconstrained or use the named constraint distributions. Assisted-by: claude-code:claude-opus-5-5
…eference constraints by name HS3 PR root-project#124 replaces the `parameter` and `constraint` (a constraint type like `Gauss`) of HistFactory modifiers with the lists `parameters` and `constraints`. Each entry of `constraints` names a distribution in the top-level `distributions`, and `null` leaves a parameter unconstrained. `shapesys` loses its `data.vals`, since the uncertainties are encoded in the constraint distributions. HS3 PR root-project#129 states that the constraint terms belong to the prediction, which is how RooFit already builds the model. Import: * `parameters` and `constraints` are read for all modifiers, and take precedence over the legacy `parameter`, `constraint`, `constraint_name` and `constraint_type`, which are still read for older files. * For `staterror`, the named constraint distributions are used instead of creating Poisson terms. Without any constraint information, the gammas are unconstrained in documents with `hs3_version` >= 0.3; older documents still get the Poisson constraints that RooFit used to create implicitly. * The `nom_*` global observable is now only created when RooFit builds a constraint itself (legacy `Gauss`). Export: * All modifiers write `parameters` and, if any parameter is constrained, `constraints` with the names of the constraint distributions, which are exported alongside. `parameter`, `constraint` and `shapesys` `data.vals` are no longer written. * The RooPoisson (or RooGaussian) constraint terms of the staterror gammas are exported as named `poisson_dist` (or `gaussian_dist`) and referenced from the staterror modifiers. root-project#124 recommends a `barlow_beeston_lite_poisson_constraint_dist`, which RooFit doesn't support yet; any distribution type is allowed. * The channel-level `staterror.constraint_type` and the `constraint` type on staterror modifiers (written since the previous HistFactory commits) are no longer written, but still read. Tests that checked the exported `constraint` key are updated to the new form, and a new test covers import and export by name. This commit depends on HS3 PRs root-project#124 and root-project#129 and should only be merged together with or after them. Assisted-by: claude-code:claude-opus-5-5
…max on irregular binned axes RooFit wrote only `edges` for binned axes with an irregular binning. HS3 PR root-project#124 requires `name`, `min`, `max` and exactly one of `nbins` or `edges` on the axes of a `histfactory_dist`, while HS3 PR root-project#135 makes `min`/`max` optional next to `edges`, but allows them if they equal the first and last edge. Writing them satisfies both, so binned axes written via `exportAxis()` (binned data, histograms, ParamHistFunc and HistFactory axes) now always carry `min` and `max`. The `axes` of generic formulas are unchanged, because their reader requires either `edges` or `min`/`max`/`nbins`. Assisted-by: claude-code:claude-opus-5-5
… named constraints in rf515_hfJSON Write the HistFactory modifiers of the rf515_hfJSON tutorial in the form of HS3 PR root-project#124: every modifier names its parameters in `parameters`, and its constraint distributions by name in `constraints`. The constraint terms that RooFit used to create implicitly are now explicit: * unit Gaussians (mean 0, sigma 1) for the normsys parameters, as recommended by root-project#124; * Poisson terms for the two staterror gammas, with the tau values (400 and 100) that follow from the sample errors. root-project#124 recommends a `barlow_beeston_lite_poisson_constraint_dist`, which RooFit doesn't support yet, so `poisson_dist` is used. The gammas are declared in a domain and in the default parameter point, so that the functions referring to them can be imported. The document now declares `hs3_version` 0.3. The likelihood is the same as before up to an additive constant of 1.3e-5 (checked at several parameter points), and the fit result agrees within the minimizer tolerance. Assisted-by: claude-code:claude-opus-5-5
… shapefactor as "shapefactor" Following the proposal for HS3 PR root-project#124, HS3 has a single per-bin modifier type, `shapefactor`, whose parameters are constrained by the distributions named in `constraints`, if given, and free otherwise. `shapesys` is a deprecated alias. * Export: HistFactory's ShapeSys and ShapeFactor are both written as `shapefactor`, with the constraint distributions found for the gammas in `constraints` (null for free gammas, omitted if all are free). The type is no longer derived from the name of the ParamHistFunc, so free gammas no longer make the export fail with "cannot find constraint", and constraints of any type are allowed (the unused relative errors, which only supported Gaussian and Poisson constraints, are no longer computed). * Import: `shapesys` is read as `shapefactor`, with a deprecation warning (once per channel) for HS3 >= 0.3. The ParamHistFunc is named `<channel>_<name>_ShapeSys` if constrained and `<channel>_<name>_shapeFactor` if free, as in HistFactory. * Without `constraints`, RooFit only creates constraints from `data.vals` for documents before HS3 0.3 or with a legacy constraint key; otherwise the parameters are free and `data.vals` is ignored with a warning. The HistFactory test of the HS3 import of ShapeFactors now round-trips a free ShapeFactor, which needed constant gammas before. Assisted-by: claude-code:claude-opus-5-5
This was referenced Nov 6, 2026
HS3 PR root-project#108 added categorical axes (`{"name": ..., "categories": [...]}`) to domains and data sets. RooFit used to read a categorical domain axis as an unbounded RooRealVar and could not read categorical axes of data. * Domains: a categorical axis creates a RooCategory with one state per category, in the order of the document and with the position as index. An existing RooCategory of that name must have the same labels. The axis must not have `min`, `max`, `nbins` or `edges`, and the labels must be unique non-empty strings. A string value in a parameter point sets the state. * Unbinned data: the entries of a categorical axis are labels of the category, which is declared by a domain. * Binned data: a categorical axis is one bin per category. HS3 identifies categories by name, so the data may list them in another order than the domain. RooDataHist looks up the bin of a category by the order of the states of the category it is given, so the data's category takes over the states and their order from the workspace category, and the contents and errors are permuted from the order of the document accordingly. * histfactory_dist: categorical axes are rejected with an explicit error. Export is unchanged: datasets and distributions with categories are still written with the ROOT-internal combined form until the HS3 representation of combined data (HS3 PR root-project#119) is final. Assisted-by: claude-code:claude-opus-5-5
…ned contents HS3 PR root-project#126 defines the storage order of multi-dimensional binned contents: the last axis listed in "axes" varies fastest. This is RooDataHist's own order, and RooFit writes the axes in the order of the variables of the RooDataHist, so no change is needed for binned data. A new test pins it. HistFactory channels write the axes from the first sample's histogram, and the contents of every sample from its own histogram. If two histograms of a channel had their observables in different orders, the contents would be written along different axes. This is now an explicit error. Assisted-by: claude-code:claude-opus-5-5
…amHistFunc in HS3 bin order HS3 PR root-project#126 lays out every per-bin array with the last axis varying fastest, like RooDataHist. ParamHistFunc, however, orders its parameters like the bins of a TH1, with the first variable varying fastest. For more than one dimension, RooFit therefore wrote the parameters of ParamHistFunc ("step") and of the staterror and shapesys/shapefactor modifiers of HistFactory in a different order than the bin contents. On import, it paired the relative uncertainty of bin k with the k-th gamma in TH1 order, so the gammas of multi-dimensional channels got the constraints of other bins. The parameters are now converted between the two orders at the HS3 boundary: they are written in bin order (via ParamHistFunc::getParameter), paired with the per-bin values in bin order, and only then put into TH1 order for the ParamHistFunc. One-dimensional functions are unchanged. Documents before HS3 0.3 with multi-dimensional ParamHistFuncs don't define the order; none are known, and they are rejected with an error instead of being guessed. New tests check the order in the written file for binned data, RooHistFunc and ParamHistFunc (each with both orders of the variables), and for a two-dimensional HistFactory channel with staterror. Assisted-by: claude-code:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Depends on #23840. This branch is stacked on it: please review only the last 3 commits. It stays in draft until HS3 PR #126 is merged, and will be rebased when the PRs below it are merged.
Rebased onto
masterafter the merge of #23996, including the follow-up commits added there during review (e.g.isHS3VersionBefore()instead ofisLegacyHS3Version(), analyses named after their ModelConfig).This Pull request:
Axis handling of RooFit's HS3 import/export, following two HS3 changes:
The export of datasets and distributions with categories is not touched: it still uses the ROOT-internal combined form, until the HS3 representation of combined data (HS3 PR #119) is final. That is #24005, stacked on this one.
Changes or fixes:
1. Read categorical axes of domains and data (HS3 #108)
RooFit used to read a categorical domain axis (
{"name": ..., "categories": [...]}) as an unboundedRooRealVar, losing the categories, and could not read categorical axes of data.RooCategory, one state per category, in the order of the document and with the position as index. An existingRooCategoryof that name must have the same labels. The axis must not havemin/max/nbins/edges; the labels must be unique non-empty strings. A string value in a parameter point sets the state.histfactory_distCategory order in binned data.
RooDataHist::calcTreeIndex()determines the bin of a category from the order of the states of the category it is given, not of its own. Data whose category lists the states in another order than the workspace category therefore gave swapped bins when looked up through the workspace category (asRooHistFunc/RooHistPdf, plotting or slicing do), even though the stored data was right. The data's category now takes over the states and their order from the workspace category, andreadBinnedData()permutescontentsanderrorsfrom the order of the document by label.Test:
CategoricalAxesInDomainsAndData(including data that lists the categories in another order than the domain, and invalid documents).2. Pin the flattening order of binned contents (HS3 #126)
RooDataHistalready stores its bins with the last variable varying fastest, and RooFit writes the axes in the order of the variables of theRooDataHist, so binned data needs no change. Test:FlatteningOrderOfBinnedContents.HistFactory channels write the axes from the first sample's histogram, but the contents of every sample from its own histogram. If two histograms of a channel had their observables in different orders, the contents were silently written along different axes. This is now an explicit error.
3. Per-bin parameters of
ParamHistFuncin HS3 bin order (HS3 #126)ParamHistFuncorders its parameters like the bins of a TH1 (first variable fastest), the opposite ofRooDataHistand of HS3. For two or more dimensions:ParamHistFunc(step) and of thestaterrorandshapesys/shapefactormodifiers ofhistfactory_distwere written in a different order than the bin contents;The parameters are now converted at the HS3 boundary: written in bin order, paired with per-bin values in bin order, and only then put into TH1 order for the
ParamHistFunc. One-dimensional functions are unchanged. Documents before HS3 0.3 with multi-dimensionalParamHistFuncs don't define the order; none are known, so they are rejected with an error instead of being guessed.Tests:
ExportedBinOrderFollowsAxeschecks the order in the written file for binned data,RooHistFuncandParamHistFunc, each built with both orders of its variables and a known value per bin, plus the round trip.HistFactoryMultiDimensionalBinOrderchecks a two-dimensional channel withstaterror: each gamma gets the uncertainty of its own bin, theParamHistFuncreturns the right gamma in every bin, the export keeps the order, and pre-0.3 documents are rejected.Note:
step(RooHistFunc,ParamHistFunc) is a RooFit-specific type, not part of the standard.RooParamHistFuncgets an HS3 export in #24002, as aproductof the two kinds ofstep.Checklist:
Tests run (Linux, Ubuntu 26.04, GCC 15.2, RelWithDebInfo,
-Droofit=ON -Dtesting=ON):testRooFitHS3111 → 115 tests (4 new),testJSONInterface1,testHistFactory69, all passing; each commit built and tested on its own. clang-format clean.AI assistance: Claude Code (claude-opus-5-5) implemented the changes, the tests and this description under review of the author. The commits carry
Assisted-by: claude-code:claude-opus-5-5.🤖 Generated with Claude Code