Skip to content

[essreduce] Fix mask order when reading from file - #879

Open
nvaytet wants to merge 4 commits into
mainfrom
fix-mask-reader
Open

nvaytet wants to merge 4 commits into
mainfrom
fix-mask-reader

Conversation

@nvaytet

@nvaytet nvaytet commented Nov 14, 2026

Copy link
Copy Markdown
Member

In #874 we fixed the order in which masks were written to file.
We apparently forgot to fix the reader as well, which we do here.

Most importantly, we add a round-trip test which verifies that masks saved to disk and loaded again match the original masks.

@nvaytet nvaytet added the essreduce Issues for essreduce. label Nov 14, 2026
- Names of `NXdetector` groups are the names of the groups in the NeXus files of the runs.
A name that is not a bank of the instrument is an error, since a misspelled name would otherwise mask nothing.
- The name of a dataset in `masks` is the name of the mask in the reduced data.
Datasets with the same name in several banks form one mask.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot decided this should go. I also fail to see how they could form just one mask, since detector banks are loaded separately in the nexus workflow. It would imply that the mask has a name and then lists the detector banks inside it, which leads to the previous data structure.

I think we are still checking that masks with the same name in different detector banks should have the same dims.

@nvaytet
nvaytet requested a review from jl-wynen November 16, 2026 07:31
Readers accept any integer type, and the boolean enumeration that h5py writes for numpy booleans.
- Dimensions of a mask are named with HDF5 dimension labels.
They are a subset of the logical dimensions of the bank, in any order, with matching sizes.
Masks with the same name in different banks use the same dimension labels, though their sizes may differ.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this added? There now is a new function to enforce this requirement. But do we actually need it?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this is not possible, e.g., for DREAM, right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In which case is it now possible for Dream? Do you mean because some dimensions do not exist in all banks?

I guess if you are in bank 1 and have a mask named mask1, and also a mask1 in bank 2, you don't need to enforce anything.
I guess I was just thinking about mask names like bad_tubes, but that is just a user-defined name. We shouldn't deduce any logic from the names of the masks.

But it's still important to raise if there is a mask for the same bank, with the same name, in two different mask files that are both being loaded.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean because some dimensions do not exist in all banks?

Yes, that is what I mean.

for mask_name, mask in content.items():
if mask_name in masks:
raise ValueError(f"Several masks have the name '{mask_name}'.")
masks[mask_name] = _align_bank_mask(mask, detector.sizes, mask_name)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire function looks fishy. name is a bank name, so why does the first branch blindly assign that to masks? And why does the last branch check the name? And why does this loop at all instead of calling pixel_masks.get(bank)? The bank is None case doesn't seem to be supported by PixelMasks.

@nvaytet nvaytet Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, this function takes in a list of masks for all detectors, and tries to extract the masks for the requested detector.

First branch blindly assigns to masks

The isinstance(content, sc.Variable): case is for the ISIS xml masks case, where the loaded masks look like:

{'mask_new_July2022.xml': <scipp.Variable> (detector_id: 249022)      int32        <no unit>  [1, 2, ..., 458751, 458752]}

I believe this is also why we can't just pixel_masks.get(bank). In the case above, if we try to load some xml masks for Loki, it would fail because bank which is loki_detector_0 would not be in the dict of masks.
Instead, we have to lookup the detector ids and determine if they are in the requested bank or not.

So the asumption is that if the value is a Variable, it must be a mask from an xml file (which only contains det ids, has no concept of bank names).
I guess this is quite a bold assumption, but maybe ok? I fail to see how you could save masks such as

{'bad_tubes': sc.zeros(sizes={'layer':4, 'tube':32}),
 'bad_straws': sc.zeros(sizes={'layer':4, 'tube':32, 'straw':7}),}

that would be for multiple detector banks, as each bank usually has its own number of layers, tubes, straws. And you don't want to have a mechanism to identify the right shape, as it can be ambiguous.

You can of course save a mask in a scipp hdf file, that would hold detector ids, like the xml mask, but then we would just treat it as the xml case above.

The bank is None case

I don't understand fully this case. For isis xml masks, this part of the code is not used.
For sans (Loki), the bank name is always specified.
Powder diffraction is the only use case I found where bank is None, but couldn't really wrap my head around how the pixel_masks were defined (they are just empty dicts in the docs examples).

if known_banks is not None and name not in known_banks:

I guess that's to protect against bank names that are misspelled, so that we raise an error instead of sliently dropping the masks?

@jl-wynen jl-wynen Nov 17, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems too complicated. I would normalise when loading instead of having a complex extraction logic because we might look at the masks dict manually or using other tools as well. So I'd rather have a consistent in-memory structure.

@nvaytet nvaytet Nov 17, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also thought that would be better, but I think it would mean you need to know the detector name to be able to load the masks, meaning that you can't just load one nexus file with masks all in one go, you just load the part for the detector that you're interested in.
Which in turn makes it more difficult to just load the masks and inspect them?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only a concern for XML mask files where we don't have a bank name in the file? I see two options for adjusting the representation:

  1. Use a catch-all bank name like "__all__". But that would mean checking for it in the extraction function.
  2. Use a dedicated type that allows us to check whether we have this case or the ESS case of one entry per bank. E.g., two separate classes or a class with an attribute (returning an enum) to distinguish the cases.

I just don't like having to inspect the contents of a dict and guessing that if you see a variable, the dict key is a mask name, otherwise it is a bank name.

@nvaytet nvaytet Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we have a special provider for reading xml masks?
It could take in the detector numbers and the detector name as inputs, to output a mask just for the required bank, as for the others.
But it means it needs to load the detector numbers before it can read the masks (but maybe that's fine?).

Do we expect to need the detector id based masks for non-xml mask files?

@SimonHeybrock I could use some input here 🙏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

essreduce Issues for essreduce.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants