Repository navigation
Conversation
HS3 PR "json-fixes-0.3" defines `hs3_version` as a `"MAJOR.MINOR"` string and the version of the revised standard as `"0.3"`. RooFit now writes this version. Documents with a newer version are imported with a warning. This commit depends on the HS3 PR and should only be merged together with or after it. Assisted-by: claude-code:claude-opus-5-5
HS3 PR "json-fixes-0.3" renames the analysis keys to `parameters_of_interest` and `domains` (a list of domain names), which is what RooFit already writes, including one entry each for the nuisance parameter, global observable and parameter-of-interest domains. The keys of HS3 v0.2.9, `parameter_of_interest` and `domain` (a single name), are now also read, with a deprecation warning. The plural keys take precedence. Assisted-by: claude-code:claude-opus-5-5
…f writing null RooFit wrote infinite bounds of product-domain axes as JSON `null`, which HS3 doesn't allow. HS3 PR "json-fixes-0.3" makes `min` and `max` of domain axes optional, with an absent bound meaning an unbounded range. Infinite bounds are now omitted on export. Since that PR also requires both bounds for a binning, the default binning is only written for bounded axes. Axes without any bound are still written (with only their `name`), which the HS3 PR keeps valid. RooFit needs them to tell an unbounded RooRealVar from a RooConstVar. On import, an absent bound means -inf/+inf, and `null` is still accepted for files written by older ROOT versions. The RooGaussianConstVarSigmaExport and ProductDomainEntriesExportExplicitBounds tests are updated to expect omitted instead of null bounds. Assisted-by: claude-code:claude-opus-5-5
…ll_dist and crystalball_doublesided_dist HS3 PR root-project#135 defines `crystalball_dist` as the single-sided Crystal Ball (`sigma`, `alpha`, `n`, as RooCBShape) and the new `crystalball_doublesided_dist` as the double-sided one (`alpha_L`, `n_L`, `alpha_R`, `n_R`, and `sigma` or `sigma_L`/`sigma_R`, without a symmetric shorthand). * RooCBShape is exported as `crystalball_dist`, as before. * A single-sided RooCrystalBall is exported as `crystalball_dist`, and read back as a RooCBShape (same function, same sign convention for alpha). Before, it was written as an incomplete `crystalball_doublesided_dist` without `alpha_R`/`n_R`, which could not be read back. * A double-sided RooCrystalBall is exported as `crystalball_doublesided_dist`, with `sigma` if both sides share it, and the tails always as `alpha_L`/`n_L`/`alpha_R`/`n_R`. * `crystalball_doublesided_dist` is imported as RooCrystalBall, also with a single `sigma`, and invalid field combinations are rejected. * `crystalball_dist` with any `_L`/`_R` field is the double-sided form of HS3 versions before 0.3. It is still read as RooCrystalBall (including `alpha`/`n` for both tails with `sigma_L`/`sigma_R`), with a one-time deprecation warning. The `crystalball_doublesided_dist` factory expression and the RooCrystalBall export keys are removed, since the new importer and exporter handle all cases. Assisted-by: claude-code:claude-opus-5-5
…on't contribute to the staterror HS3 PRs root-project#135 and root-project#124 make the "errors" of a HistFactory sample optional: a sample without "errors" contributes no statistical uncertainty to a staterror modifier. RooFit used the content as squared error instead, i.e. assumed Poisson uncertainties. That is kept for documents before HS3 0.3. In HS3 >= 0.3 documents, this only matters if the staterror is constrained through the legacy "constraint" key, since otherwise the gammas are either unconstrained or use the named constraint distributions. Assisted-by: claude-code:claude-opus-5-5
…eference constraints by name HS3 PR root-project#124 replaces the `parameter` and `constraint` (a constraint type like `Gauss`) of HistFactory modifiers with the lists `parameters` and `constraints`. Each entry of `constraints` names a distribution in the top-level `distributions`, and `null` leaves a parameter unconstrained. `shapesys` loses its `data.vals`, since the uncertainties are encoded in the constraint distributions. HS3 PR root-project#129 states that the constraint terms belong to the prediction, which is how RooFit already builds the model. Import: * `parameters` and `constraints` are read for all modifiers, and take precedence over the legacy `parameter`, `constraint`, `constraint_name` and `constraint_type`, which are still read for older files. * For `staterror`, the named constraint distributions are used instead of creating Poisson terms. Without any constraint information, the gammas are unconstrained in documents with `hs3_version` >= 0.3; older documents still get the Poisson constraints that RooFit used to create implicitly. * The `nom_*` global observable is now only created when RooFit builds a constraint itself (legacy `Gauss`). Export: * All modifiers write `parameters` and, if any parameter is constrained, `constraints` with the names of the constraint distributions, which are exported alongside. `parameter`, `constraint` and `shapesys` `data.vals` are no longer written. * The RooPoisson (or RooGaussian) constraint terms of the staterror gammas are exported as named `poisson_dist` (or `gaussian_dist`) and referenced from the staterror modifiers. root-project#124 recommends a `barlow_beeston_lite_poisson_constraint_dist`, which RooFit doesn't support yet; any distribution type is allowed. * The channel-level `staterror.constraint_type` and the `constraint` type on staterror modifiers (written since the previous HistFactory commits) are no longer written, but still read. Tests that checked the exported `constraint` key are updated to the new form, and a new test covers import and export by name. This commit depends on HS3 PRs root-project#124 and root-project#129 and should only be merged together with or after them. Assisted-by: claude-code:claude-opus-5-5
…max on irregular binned axes RooFit wrote only `edges` for binned axes with an irregular binning. HS3 PR root-project#124 requires `name`, `min`, `max` and exactly one of `nbins` or `edges` on the axes of a `histfactory_dist`, while HS3 PR root-project#135 makes `min`/`max` optional next to `edges`, but allows them if they equal the first and last edge. Writing them satisfies both, so binned axes written via `exportAxis()` (binned data, histograms, ParamHistFunc and HistFactory axes) now always carry `min` and `max`. The `axes` of generic formulas are unchanged, because their reader requires either `edges` or `min`/`max`/`nbins`. Assisted-by: claude-code:claude-opus-5-5
… named constraints in rf515_hfJSON Write the HistFactory modifiers of the rf515_hfJSON tutorial in the form of HS3 PR root-project#124: every modifier names its parameters in `parameters`, and its constraint distributions by name in `constraints`. The constraint terms that RooFit used to create implicitly are now explicit: * unit Gaussians (mean 0, sigma 1) for the normsys parameters, as recommended by root-project#124; * Poisson terms for the two staterror gammas, with the tau values (400 and 100) that follow from the sample errors. root-project#124 recommends a `barlow_beeston_lite_poisson_constraint_dist`, which RooFit doesn't support yet, so `poisson_dist` is used. The gammas are declared in a domain and in the default parameter point, so that the functions referring to them can be imported. The document now declares `hs3_version` 0.3. The likelihood is the same as before up to an additive constant of 1.3e-5 (checked at several parameter points), and the fit result agrees within the minimizer tolerance. Assisted-by: claude-code:claude-opus-5-5
… shapefactor as "shapefactor" Following the proposal for HS3 PR root-project#124, HS3 has a single per-bin modifier type, `shapefactor`, whose parameters are constrained by the distributions named in `constraints`, if given, and free otherwise. `shapesys` is a deprecated alias. * Export: HistFactory's ShapeSys and ShapeFactor are both written as `shapefactor`, with the constraint distributions found for the gammas in `constraints` (null for free gammas, omitted if all are free). The type is no longer derived from the name of the ParamHistFunc, so free gammas no longer make the export fail with "cannot find constraint", and constraints of any type are allowed (the unused relative errors, which only supported Gaussian and Poisson constraints, are no longer computed). * Import: `shapesys` is read as `shapefactor`, with a deprecation warning (once per channel) for HS3 >= 0.3. The ParamHistFunc is named `<channel>_<name>_ShapeSys` if constrained and `<channel>_<name>_shapeFactor` if free, as in HistFactory. * Without `constraints`, RooFit only creates constraints from `data.vals` for documents before HS3 0.3 or with a legacy constraint key; otherwise the parameters are free and `data.vals` is ignored with a warning. The HistFactory test of the HS3 import of ShapeFactors now round-trips a free ShapeFactor, which needed constant gammas before. Assisted-by: claude-code:claude-opus-5-5
…actors The HistFactory exporter decomposes RooFormulaVars that are products into their factors, to recognise normfactors and explicitly encoded normsys modifiers. splitTopLevelProduct() split an expression at every "*" at the top level, also if the expression is a sum or difference at the top level: "1 + 0.2 * k" became the factors "1 + 0.2" and "k", i.e. 1.2 * k instead of 1 + 0.2 * k, which changes the model. A binary "+" or "-" at the top level now means that the expression is not a product (signs and exponents like in "2e-1" are not binary operators). Test: FormulaWithTopLevelSumIsNotSplit (fails without the fix). Assisted-by: claude-code:claude-opus-5-5
…ucture histfactory_dist implies the division by the bin width, so the HistFactory exporter doesn't write it as a modifier. It recognised it by the name: every factor of a sample with "binWidth" in its name was dropped. That silently dropped e.g. a normfactor called "signal_binWidth_scale", and a renamed RooBinWidthFunction was written as a custom modifier, so that the import divided by the bin width twice. * A RooBinWidthFunction that divides by the bin width is identified by its type. * HistFactory before ROOT 6.26 multiplied every sample with a constant RooRealVar "binWidth_<observables>_<i>" in the coefficients of the RooRealSumPdf, with numBins/(max-min) as value. Such a factor is now identified by its structure: a RooConstVar or a constant RooRealVar that is not floating in any snapshot of the workspace, whose only clients are the RooRealSumPdf or products of the channel, and whose value is the inverse of the volume of the first bin, or of the average bin volume, of one of the histograms of the channel. If its name doesn't contain "binWidth", a warning says that it was identified. Identified factors are not written as model parameters either. * RooJSONFactoryWSTool::config().identifyLegacyBinWidthFactors (default true) switches the identification of such constants off; they are then written as normfactors. Test: HS3LegacyBinWidthFactor in testHistFactory, with the ROOT 6.16 reference workspace: original and renamed factors, a factor floating in a snapshot, and the switch. HS3ExportDoesNotDependOnNames (next commit) also covers a renamed legacy factor with bins of width pi/10. Assisted-by: claude-code:claude-opus-5-5
Many workspaces were made with old or custom versions of HistFactory, or modified after their creation, and still have the structure of a HistFactory model. Their channels should be written as histfactory_dist, not as primitives. The new test HS3ExportDoesNotDependOnNames builds a two-channel HistFactory model (normfactor, normsys, histosys, shapesys, staterror, shapefactor, lumi), changes it (adds a custom modifier, removes a normsys, swaps a constraint for a narrower one, adds a second FlexibleInterpVar with a new constrained parameter, and in one channel the constant inverse bin width of HistFactory before ROOT 6.26 with 7 significant digits instead of the RooBinWidthFunction; bins of width pi/10), and renames every object except the observables, the channel category and the constants: the names are reversed, so no prefix or suffix of the HistFactory naming scheme survives, and the POI gets a name with "binWidth". It checks that both channels are written as histfactory_dist with the expected modifiers, that the likelihood is the same after the round trip, and that the document is stable from the second export on. Without the two preceding commits, the POI is dropped from the model and the custom modifier is split into wrong factors. Assisted-by: claude-code:claude-opus-5-5
A histfactory_dist is a product of Poisson distributions of the bin counts.
RooFit's binned likelihood ("BinnedLikelihood" attribute of the
RooRealSumPdf) computes exactly that; it differs from the event-wise
likelihood only by a constant (checked with both evaluation backends, for
binned and unbinned data), and is faster. Binned generation
("GenerateBinned") generates bin counts.
HistFactory enables both by default, but the HS3 importer didn't: the
attributes only came back from "misc", keyed by the name of the
RooRealSumPdf, so they were lost for channels whose sum pdf has another
name than the one the importer gives it. The importer now enables both for
every histfactory_dist.
Tests: HistFactoryImportEnablesBinnedLikelihood, and the renamed model of
HS3ExportDoesNotDependOnNames.
Assisted-by: claude-code:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Depends on #23840. This branch is stacked on it: please review only the last 4 commits. It stays in draft until #23840 is merged, and will be rebased then.
This Pull request:
Makes the HistFactory exporter of the HS3 I/O independent of the names of the objects in the workspace. Many workspaces were made with old or custom versions of HistFactory, or modified after their creation (e.g. with
custommodifiers, which the XML-based HistFactory doesn't support), and still have the structure of a HistFactory model. Their channels should be written ashistfactory_dist, not split into primitives, and the model must stay the same.A new test renames everything in a HistFactory model and changes its modifiers (see below). It found two bugs:
histfactory_distimplies the division by the bin width, so the exporter doesn't write it as a modifier. It recognised it by the name: every factor of a sample withbinWidthin its name was dropped. A normfactor called e.g.signal_binWidth_scalesilently disappeared from the model, and a renamedRooBinWidthFunctionwas written as acustommodifier, so that the import divided by the bin width twice.RooFormulaVars that are products into their factors. It split an expression at every top-level*, also if the expression is a sum at the top level:1 + 0.2 * kbecame the factors1 + 0.2andk, i.e.1.2 * k. This doesn't depend on names; the test just happened to use such a formula.Changes or fixes:
+or-at the top level means that the expression is not a product (signs and exponents like2e-1are not binary operators).FormulaWithTopLevelSumIsNotSplit(fails without the fix)RooBinWidthFunctionthat divides by the bin width is identified by its type. The constant inverse bin width of HistFactory before ROOT 6.26 (a constantRooRealVarbinWidth_<observables>_<i>in the coefficients of theRooRealSumPdf, valuenumBins/(max-min)) is identified by its structure: aRooConstVaror constantRooRealVarthat is not floating in any snapshot, whose only clients are theRooRealSumPdfor products of the channel, and whose value is the inverse of the volume of the first bin, or of the average bin volume, of one of the histograms of the channel. If its name doesn't containbinWidth, a warning says that it was identified. Identified factors are not written as model parameters either.RooJSONFactoryWSTool::config().identifyLegacyBinWidthFactors = falseswitches this off; such constants are then written as normfactors.HS3LegacyBinWidthFactor(ROOT 6.16 reference workspace: original and renamed factors, a factor floating in a snapshot, the switch)HS3ExportDoesNotDependOnNames: a two-channel HistFactory model (normfactor, normsys, histosys, shapesys, staterror, shapefactor, lumi; bins of width π/10, which is not exactly representable), changed (a custom modifier added, a normsys removed, a constraint swapped for a narrower Gaussian, a secondFlexibleInterpVarwith a new constrained parameter added, and in one channel theRooBinWidthFunctionreplaced by the constant inverse bin width of HistFactory before ROOT 6.26, with 7 significant digits as HistFactory wrote it), and every object renamed except the observables, the channel category and the constants (names reversed, so that no prefix or suffix of the HistFactory naming scheme survives; the POI is calledsignal_binWidth_scale). Checks: both channels arehistfactory_distwith the expected modifiers (the renamed legacy bin width is identified by its value, with a warning), the likelihood is the same after the round trip at several parameter points, and the document is stable from the second export on. Without the two fixes, the POI is dropped and the custom modifier is split into wrong factors.histfactory_disthistfactory_distis a product of Poisson distributions of the bin counts, which is what RooFit's binned likelihood (BinnedLikelihoodattribute of theRooRealSumPdf) computes; binned generation (GenerateBinned) generates bin counts. HistFactory enables both by default, but the importer didn't: they only came back frommisc, keyed by the name of theRooRealSumPdf, so they were lost for renamed channels. The importer now enables both for everyhistfactory_dist. The binned likelihood differs from the event-wise one only by a constant (checked with both evaluation backends, for binned and unbinned data), so only absolute NLL values change, and only for workspaces that didn't have the attribute.HistFactoryImportEnablesBinnedLikelihood, and the renamed model ofHS3ExportDoesNotDependOnNamesRemaining name dependencies
Found while reading the exporter, not changed here:
ParamHistFuncis taken asstaterrorif its name starts withmc_stat_, otherwise asshapefactor. A renamed staterror is therefore written as ashapefactorper sample, with the same parameters and constraints: a valid description of the same likelihood, but without the staterror semantics. Telling them apart by structure (gammas shared by all samples of a channel, constraints derived from the sample errors) is possible but not obviously unambiguous.nom_are not exported as model parameters. Renamed global observables are exported as parameters in addition; harmless._tausuffix only as fallback.model_,L_x_,_shapesetc.; cosmetic.On the import side, HistFactory objects are created with the usual names, so the
miscattributes of a renamedRooRealSumPdfare lost in the first round trip (the binned likelihood is enabled anyway, see the last commit); from the second export on, the document is stable.Checklist:
Tests run (Linux, Ubuntu 26.04, GCC 15.2, RelWithDebInfo,
-Droofit=ON -Dtesting=ON):testRooFitHS3111 → 113 (2 new),testJSONInterface1,testHistFactory69 → 71 (2 new), all passing; each commit built and tested on its own. clang-format clean. The commits also apply on top of #24005 and of #24013 with trivial conflicts (newconfig()fields, includes), and pass there as well (on #24005:testRooFitHS3128, on #24013: 127;testHistFactory72 on both). The round-trip tests of the ROOT 6.16 reference workspace in #24005 and #24013 compared absolute NLL values, which change when the binned likelihood is enabled; they now compare the likelihood up to a constant.AI assistance: Claude Code (claude-opus-5-5) implemented the changes, the tests and this description under review of the author, following the author's design for the structural identification of the bin width factor. The commits carry
Assisted-by: claude-code:claude-opus-5-5.🤖 Generated with Claude Code