Skip to content

[RF][HS3] Axis handling: categorical axes and HS3 bin order (HS3 #108, #126; depends on #23840) - #24001

Draft
cburgard wants to merge 12 commits into
root-project:masterfrom
cburgard:hs3-axes
Draft

cburgard wants to merge 12 commits into
root-project:masterfrom
cburgard:hs3-axes

Conversation

@cburgard

@cburgard cburgard commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Important

Depends on #23840. This branch is stacked on it: please review only the last 3 commits. It stays in draft until HS3 PR #126 is merged, and will be rebased when the PRs below it are merged.

Rebased onto master after the merge of #23996, including the follow-up commits added there during review (e.g. isHS3VersionBefore() instead of isLegacyHS3Version(), analyses named after their ModelConfig).

This Pull request:

Axis handling of RooFit's HS3 import/export, following two HS3 changes:

  • HS3 PR #108 (merged): categorical axes in domains and data sets.
  • HS3 PR #126 "Refactor axes": the storage order of multi-dimensional binned contents and of every other per-bin array is defined, with the last axis varying fastest.

The export of datasets and distributions with categories is not touched: it still uses the ROOT-internal combined form, until the HS3 representation of combined data (HS3 PR #119) is final. That is #24005, stacked on this one.

Changes or fixes:

1. Read categorical axes of domains and data (HS3 #108)

RooFit used to read a categorical domain axis ({"name": ..., "categories": [...]}) as an unbounded RooRealVar, losing the categories, and could not read categorical axes of data.

Where Change
Domains A categorical axis creates a RooCategory, one state per category, in the order of the document and with the position as index. An existing RooCategory of that name must have the same labels. The axis must not have min/max/nbins/edges; the labels must be unique non-empty strings. A string value in a parameter point sets the state.
Unbinned data The entries of a categorical axis are labels of the category (declared by a domain).
Binned data A categorical axis has one bin per category. HS3 identifies categories by name, so the data may list them in another order than the domain.
histfactory_dist Categorical axes are rejected with an explicit error.

Category order in binned data. RooDataHist::calcTreeIndex() determines the bin of a category from the order of the states of the category it is given, not of its own. Data whose category lists the states in another order than the workspace category therefore gave swapped bins when looked up through the workspace category (as RooHistFunc/RooHistPdf, plotting or slicing do), even though the stored data was right. The data's category now takes over the states and their order from the workspace category, and readBinnedData() permutes contents and errors from the order of the document by label.

Test: CategoricalAxesInDomainsAndData (including data that lists the categories in another order than the domain, and invalid documents).

2. Pin the flattening order of binned contents (HS3 #126)

RooDataHist already stores its bins with the last variable varying fastest, and RooFit writes the axes in the order of the variables of the RooDataHist, so binned data needs no change. Test: FlatteningOrderOfBinnedContents.

HistFactory channels write the axes from the first sample's histogram, but the contents of every sample from its own histogram. If two histograms of a channel had their observables in different orders, the contents were silently written along different axes. This is now an explicit error.

3. Per-bin parameters of ParamHistFunc in HS3 bin order (HS3 #126)

ParamHistFunc orders its parameters like the bins of a TH1 (first variable fastest), the opposite of RooDataHist and of HS3. For two or more dimensions:

  • the parameters of ParamHistFunc (step) and of the staterror and shapesys/shapefactor modifiers of histfactory_dist were written in a different order than the bin contents;
  • on import, the relative uncertainty of bin k was paired with the k-th gamma in TH1 order, so the gammas of multi-dimensional HistFactory channels got the constraints of other bins.

The parameters are now converted at the HS3 boundary: written in bin order, paired with per-bin values in bin order, and only then put into TH1 order for the ParamHistFunc. One-dimensional functions are unchanged. Documents before HS3 0.3 with multi-dimensional ParamHistFuncs don't define the order; none are known, so they are rejected with an error instead of being guessed.

Tests: ExportedBinOrderFollowsAxes checks the order in the written file for binned data, RooHistFunc and ParamHistFunc, each built with both orders of its variables and a known value per bin, plus the round trip. HistFactoryMultiDimensionalBinOrder checks a two-dimensional channel with staterror: each gamma gets the uncertainty of its own bin, the ParamHistFunc returns the right gamma in every bin, the export keeps the order, and pre-0.3 documents are rejected.

Note: step (RooHistFunc, ParamHistFunc) is a RooFit-specific type, not part of the standard. RooParamHistFunc gets an HS3 export in #24002, as a product of the two kinds of step.

Checklist:

  • tested changes locally
  • updated the docs (if necessary)

Tests run (Linux, Ubuntu 26.04, GCC 15.2, RelWithDebInfo, -Droofit=ON -Dtesting=ON): testRooFitHS3 111 → 115 tests (4 new), testJSONInterface 1, testHistFactory 69, all passing; each commit built and tested on its own. clang-format clean.

AI assistance: Claude Code (claude-opus-5-5) implemented the changes, the tests and this description under review of the author. The commits carry Assisted-by: claude-code:claude-opus-5-5.

🤖 Generated with Claude Code

HS3 PR "json-fixes-0.3" defines `hs3_version` as a `"MAJOR.MINOR"` string
and the version of the revised standard as `"0.3"`. RooFit now writes this
version. Documents with a newer version are imported with a warning.

This commit depends on the HS3 PR and should only be merged together with
or after it.

Assisted-by: claude-code:claude-opus-5-5
HS3 PR "json-fixes-0.3" renames the analysis keys to
`parameters_of_interest` and `domains` (a list of domain names), which is
what RooFit already writes, including one entry each for the nuisance
parameter, global observable and parameter-of-interest domains.

The keys of HS3 v0.2.9, `parameter_of_interest` and `domain` (a single
name), are now also read, with a deprecation warning. The plural keys take
precedence.

Assisted-by: claude-code:claude-opus-5-5
…f writing null

RooFit wrote infinite bounds of product-domain axes as JSON `null`, which
HS3 doesn't allow. HS3 PR "json-fixes-0.3" makes `min` and `max` of domain
axes optional, with an absent bound meaning an unbounded range. Infinite
bounds are now omitted on export. Since that PR also requires both bounds
for a binning, the default binning is only written for bounded axes.

Axes without any bound are still written (with only their `name`), which
the HS3 PR keeps valid. RooFit needs them to tell an unbounded RooRealVar
from a RooConstVar. On import, an absent bound means -inf/+inf, and `null`
is still accepted for files written by older ROOT versions.

The RooGaussianConstVarSigmaExport and ProductDomainEntriesExportExplicitBounds
tests are updated to expect omitted instead of null bounds.

Assisted-by: claude-code:claude-opus-5-5
…ll_dist and crystalball_doublesided_dist

HS3 PR root-project#135 defines `crystalball_dist` as the single-sided Crystal Ball
(`sigma`, `alpha`, `n`, as RooCBShape) and the new
`crystalball_doublesided_dist` as the double-sided one (`alpha_L`, `n_L`,
`alpha_R`, `n_R`, and `sigma` or `sigma_L`/`sigma_R`, without a symmetric
shorthand).

* RooCBShape is exported as `crystalball_dist`, as before.
* A single-sided RooCrystalBall is exported as `crystalball_dist`, and read
  back as a RooCBShape (same function, same sign convention for alpha).
  Before, it was written as an incomplete `crystalball_doublesided_dist`
  without `alpha_R`/`n_R`, which could not be read back.
* A double-sided RooCrystalBall is exported as
  `crystalball_doublesided_dist`, with `sigma` if both sides share it, and
  the tails always as `alpha_L`/`n_L`/`alpha_R`/`n_R`.
* `crystalball_doublesided_dist` is imported as RooCrystalBall, also with a
  single `sigma`, and invalid field combinations are rejected.
* `crystalball_dist` with any `_L`/`_R` field is the double-sided form of
  HS3 versions before 0.3. It is still read as RooCrystalBall (including
  `alpha`/`n` for both tails with `sigma_L`/`sigma_R`), with a one-time
  deprecation warning.

The `crystalball_doublesided_dist` factory expression and the RooCrystalBall
export keys are removed, since the new importer and exporter handle all
cases.

Assisted-by: claude-code:claude-opus-5-5
…on't contribute to the staterror

HS3 PRs root-project#135 and root-project#124 make the "errors" of a HistFactory sample optional: a
sample without "errors" contributes no statistical uncertainty to a
staterror modifier. RooFit used the content as squared error instead, i.e.
assumed Poisson uncertainties. That is kept for documents before HS3 0.3.

In HS3 >= 0.3 documents, this only matters if the staterror is constrained
through the legacy "constraint" key, since otherwise the gammas are either
unconstrained or use the named constraint distributions.

Assisted-by: claude-code:claude-opus-5-5
…eference constraints by name

HS3 PR root-project#124 replaces the `parameter` and `constraint` (a constraint type
like `Gauss`) of HistFactory modifiers with the lists `parameters` and
`constraints`. Each entry of `constraints` names a distribution in the
top-level `distributions`, and `null` leaves a parameter unconstrained.
`shapesys` loses its `data.vals`, since the uncertainties are encoded in
the constraint distributions. HS3 PR root-project#129 states that the constraint terms
belong to the prediction, which is how RooFit already builds the model.

Import:
* `parameters` and `constraints` are read for all modifiers, and take
  precedence over the legacy `parameter`, `constraint`, `constraint_name`
  and `constraint_type`, which are still read for older files.
* For `staterror`, the named constraint distributions are used instead of
  creating Poisson terms. Without any constraint information, the gammas
  are unconstrained in documents with `hs3_version` >= 0.3; older documents
  still get the Poisson constraints that RooFit used to create implicitly.
* The `nom_*` global observable is now only created when RooFit builds a
  constraint itself (legacy `Gauss`).

Export:
* All modifiers write `parameters` and, if any parameter is constrained,
  `constraints` with the names of the constraint distributions, which are
  exported alongside. `parameter`, `constraint` and `shapesys` `data.vals`
  are no longer written.
* The RooPoisson (or RooGaussian) constraint terms of the staterror gammas
  are exported as named `poisson_dist` (or `gaussian_dist`) and referenced
  from the staterror modifiers. root-project#124 recommends a
  `barlow_beeston_lite_poisson_constraint_dist`, which RooFit doesn't
  support yet; any distribution type is allowed.
* The channel-level `staterror.constraint_type` and the `constraint` type
  on staterror modifiers (written since the previous HistFactory commits)
  are no longer written, but still read.

Tests that checked the exported `constraint` key are updated to the new
form, and a new test covers import and export by name.

This commit depends on HS3 PRs root-project#124 and root-project#129 and should only be merged
together with or after them.

Assisted-by: claude-code:claude-opus-5-5
…max on irregular binned axes

RooFit wrote only `edges` for binned axes with an irregular binning. HS3
PR root-project#124 requires `name`, `min`, `max` and exactly one of `nbins` or
`edges` on the axes of a `histfactory_dist`, while HS3 PR root-project#135 makes
`min`/`max` optional next to `edges`, but allows them if they equal the
first and last edge. Writing them satisfies both, so binned axes written
via `exportAxis()` (binned data, histograms, ParamHistFunc and HistFactory
axes) now always carry `min` and `max`. The `axes` of generic formulas are
unchanged, because their reader requires either `edges` or
`min`/`max`/`nbins`.

Assisted-by: claude-code:claude-opus-5-5
… named constraints in rf515_hfJSON

Write the HistFactory modifiers of the rf515_hfJSON tutorial in the form
of HS3 PR root-project#124: every modifier names its parameters in `parameters`, and
its constraint distributions by name in `constraints`. The constraint
terms that RooFit used to create implicitly are now explicit:

* unit Gaussians (mean 0, sigma 1) for the normsys parameters, as
  recommended by root-project#124;
* Poisson terms for the two staterror gammas, with the tau values
  (400 and 100) that follow from the sample errors. root-project#124 recommends a
  `barlow_beeston_lite_poisson_constraint_dist`, which RooFit doesn't
  support yet, so `poisson_dist` is used.

The gammas are declared in a domain and in the default parameter point, so
that the functions referring to them can be imported. The document now
declares `hs3_version` 0.3.

The likelihood is the same as before up to an additive constant of
1.3e-5 (checked at several parameter points), and the fit result agrees
within the minimizer tolerance.

Assisted-by: claude-code:claude-opus-5-5
… shapefactor as "shapefactor"

Following the proposal for HS3 PR root-project#124, HS3 has a single per-bin modifier
type, `shapefactor`, whose parameters are constrained by the distributions
named in `constraints`, if given, and free otherwise. `shapesys` is a
deprecated alias.

* Export: HistFactory's ShapeSys and ShapeFactor are both written as
  `shapefactor`, with the constraint distributions found for the gammas in
  `constraints` (null for free gammas, omitted if all are free). The type is
  no longer derived from the name of the ParamHistFunc, so free gammas no
  longer make the export fail with "cannot find constraint", and constraints
  of any type are allowed (the unused relative errors, which only supported
  Gaussian and Poisson constraints, are no longer computed).
* Import: `shapesys` is read as `shapefactor`, with a deprecation warning
  (once per channel) for HS3 >= 0.3. The ParamHistFunc is named
  `<channel>_<name>_ShapeSys` if constrained and
  `<channel>_<name>_shapeFactor` if free, as in HistFactory.
* Without `constraints`, RooFit only creates constraints from `data.vals`
  for documents before HS3 0.3 or with a legacy constraint key; otherwise the
  parameters are free and `data.vals` is ignored with a warning.

The HistFactory test of the HS3 import of ShapeFactors now round-trips a
free ShapeFactor, which needed constant gammas before.

Assisted-by: claude-code:claude-opus-5-5
HS3 PR root-project#108 added categorical axes (`{"name": ..., "categories": [...]}`) to
domains and data sets. RooFit used to read a categorical domain axis as an
unbounded RooRealVar and could not read categorical axes of data.

* Domains: a categorical axis creates a RooCategory with one state per
  category, in the order of the document and with the position as index. An
  existing RooCategory of that name must have the same labels. The axis must
  not have `min`, `max`, `nbins` or `edges`, and the labels must be unique
  non-empty strings. A string value in a parameter point sets the state.
* Unbinned data: the entries of a categorical axis are labels of the
  category, which is declared by a domain.
* Binned data: a categorical axis is one bin per category. HS3 identifies
  categories by name, so the data may list them in another order than the
  domain. RooDataHist looks up the bin of a category by the order of the
  states of the category it is given, so the data's category takes over the
  states and their order from the workspace category, and the contents and
  errors are permuted from the order of the document accordingly.
* histfactory_dist: categorical axes are rejected with an explicit error.

Export is unchanged: datasets and distributions with categories are still
written with the ROOT-internal combined form until the HS3 representation of
combined data (HS3 PR root-project#119) is final.

Assisted-by: claude-code:claude-opus-5-5
…ned contents

HS3 PR root-project#126 defines the storage order of multi-dimensional binned contents:
the last axis listed in "axes" varies fastest. This is RooDataHist's own
order, and RooFit writes the axes in the order of the variables of the
RooDataHist, so no change is needed for binned data. A new test pins it.

HistFactory channels write the axes from the first sample's histogram, and
the contents of every sample from its own histogram. If two histograms of a
channel had their observables in different orders, the contents would be
written along different axes. This is now an explicit error.

Assisted-by: claude-code:claude-opus-5-5
…amHistFunc in HS3 bin order

HS3 PR root-project#126 lays out every per-bin array with the last axis varying fastest,
like RooDataHist. ParamHistFunc, however, orders its parameters like the bins
of a TH1, with the first variable varying fastest. For more than one
dimension, RooFit therefore wrote the parameters of ParamHistFunc ("step")
and of the staterror and shapesys/shapefactor modifiers of HistFactory in a
different order than the bin contents. On import, it paired the relative
uncertainty of bin k with the k-th gamma in TH1 order, so the gammas of
multi-dimensional channels got the constraints of other bins.

The parameters are now converted between the two orders at the HS3
boundary: they are written in bin order (via ParamHistFunc::getParameter),
paired with the per-bin values in bin order, and only then put into TH1
order for the ParamHistFunc. One-dimensional functions are unchanged.

Documents before HS3 0.3 with multi-dimensional ParamHistFuncs don't define
the order; none are known, and they are rejected with an error instead of
being guessed.

New tests check the order in the written file for binned data, RooHistFunc
and ParamHistFunc (each with both orders of the variables), and for a
two-dimensional HistFactory channel with staterror.

Assisted-by: claude-code:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants