[woutdenolf]

Closes #1668.
Closes #1670.

This PR makes five changes. Each is described below as before and after, followed by what the NIAC should decide.

Review tips: start with manual/source/design.rst (sections "Terms at five levels" and "Links"), then base_classes/NXobject.nxdl.xml, and in nxdl.xsd the types fieldAttributesType, groupAttributeType (which carries valueFrom), valueFromType and linkType.

1. fieldAttributes: NeXus attributes of every field

Before

• The NeXus attributes that any field may have (@units, @target, @long_name, …) were described only in prose. No NXDL file defined them, so a validator that reads only NXDL files did not know them.
• nxdl.xsd declared XML attributes of the NXDL field with the same names (long_name, interpretation, stride, data_offset). Such a declaration constrains only .nxdl.xml files, never a data file.

After

• A new NXDL term, fieldAttributes, the last child of an NXDL group or of the NXDL definition of a base class (at most one). Its NXDL attributes define NeXus attributes that any field directly in that NXDL group or class may have, whether or not the class defines that field; not fields in nested groups.
• NXobject uses it for every field: @target, @units, @long_name, @interpretation, @calibration_status, @stride, @data_offset and @variant. NXroot, which does not extend NXobject, has a copy; a test keeps both the same. This supersedes add target attribute to NXobject using new field super-type #1558, which proposed an NXDL field FIELDNAME with nameType="any" in NXobject: that matches only names that no other NeXus item of the group defines, so it would miss named fields such as data of NXdetector.
• When several define the same NeXus attribute, the closest applies: an NXDL attribute in the NXDL field itself, then the fieldAttributes of the enclosing NXDL group, then those of the NXDL groups that define the same NeXus group in the application definitions this one extends, then those of its class, then those of the classes that class extends (see fieldAttributesType in nxdl.xsd).
• In a class path, these NeXus attributes are written with * for the field name, such as /NXobject/*@units.
• fieldAttributes directly in an application definition is rejected by the repository's Python syntax check (dev_tools, run in CI): no NeXus group has an application definition as its type.
• @units: a content field that a NeXus field with a units category describes (such as units="NX_LENGTH") must have the content attribute @units, with units of that category.
• The XML attributes long_name, interpretation, stride and data_offset of the NXDL field are deprecated. No NXDL file uses them.

To decide

• Do we agree on fieldAttributes and its precedence? The precedence covers application definitions that extend each other, but not yet base classes that extend base classes (for example the 62 that extend NXcomponent).
• Should @interpretation and @long_name move from NXobject to NXdata? The prose ties them to plotting, but they are about the field itself, not about a plot, and content links share content attributes: defined only in NXdata, no NeXus attribute would describe them at the other end, in content groups of class NXdetector, NXsample, ….
• Should @stride and @data_offset stay? They describe how values are stored when not in C order (Data Rules > Non C Storage Order), which looks like a matter of storage. Zarr stores the order itself; HDF5 cannot.
• Do we remove the declarations of the deprecated XML attributes from nxdl.xsd in a later NXDL version? Until then, stride and data_offset keep default="1" there, so a schema-aware reader still gives every NXDL field these XML attributes.

2. valueFrom="class": the value of @NX_class

Before

• No NXDL file could say what the value of the NeXus attribute @NX_class is, except a one-value enumeration repeated in every class.

After

• A new XML attribute, valueFrom="class", of NXDL attributes in NXDL groups and definitions: the value of this NeXus attribute is the type of the NeXus group it is an attribute of, a class name such as NXsample.
• NXobject and NXroot use it for @NX_class. The class page of NXsample, for example, shows "Obligatory value: NXsample".
• valueFrom combined with an NXDL enumeration is rejected by the Python syntax check, not by nxdl.xsd: XSD 1.0 cannot express that rule.

To decide

• Keep valueFrom as an XML attribute? As a child element instead, nxdl.xsd could offer a choice between it and an NXDL enumeration, so any XSD validator would reject the combination. The alternative, a one-value enumeration in every class, also needs a Python check and repeats the class name in every definition.

3. @NX_class is required

Before

• No class defined @NX_class. The prose (File Format, Data Rules) said that the class of a group is in NX_class, but not that it is required, and nothing for the root.

After

• NXobject and NXroot define @NX_class with optional="false".
• The manual also states it as a rule of data files: every content group in a NeXus file, the root included, must have @NX_class, whether or not a NeXus group describes it. Without it, a validator could not tell the class of the group, and the class rule would never reach it.
• This is an exception to the prose "all NeXus items of a base class are optional"; the PR adds the exception to that prose (nxdl.xsd, Design).
• NXobject also defines @target for every group.

To decide

• Required, or only recommended? In base classes and in application definitions alike? To check: how do validators handle a content group without @NX_class today?

4. Content links: the same content, however it is stored

Before

• "NeXus links are HDF5 hard links with an additional target attribute". A Zarr file, which has no storage links, could not satisfy an application definition with a NeXus link.
• A NeXus link required the content field or group of its name to exist and have @target; nxdl.xsd called the XML attribute target of the NXDL link only a suggestion of the value of @target.

After

• A content link is a content field or group whose @target is the path of another one, its target, with the same shared content:
  • of a content field: its values, data type and content attributes, except @target;
  • of a content group: its content attributes, except @target, and the shared content of its content fields and groups, under their names.
• The target is not a content link itself (no chains). It needs no @target; if it has one, for example because an HDF5 hard link makes both paths one object, its value is its own path.
• How a content link is stored is up to the file format: in HDF5 a hard, soft or external link, a virtual dataset, a dataset with external binary data, or a copy; in Zarr a copy. @target is always a path in the content tree (fix target attribute for external file links #842).
• A NeXus link now describes a content field or group with the same shared content as one that its target describes; one of the two is a content link to the other (or both to the same target). Either may be the content link.
• The XML attribute target of the NXDL link is a class path. It names the NeXus field or group whose content must be the same. The value of @target does not have to match it.
• The XML attribute napimount of the NXDL link is deprecated, and the manual marks the HDF5 attribute napimount of NAPI files as deprecated.

To decide

• Is "same shared content" exact: the same values, data type and content attributes, except @target?
• May a validator trust @target instead of comparing? Nothing needs comparing when the storage makes both paths one object (HDF5 hard or soft link). A 1-to-1 virtual dataset guarantees the values only: its content attributes must still be compared. A large copy is expensive to compare.
• Is an HDF5 external link acceptable as a content link? It has no HDF5 attributes of its own, so @target is stored in the other file, as a path in the linking file; that file must be writable, and only one linking file can set it.
• Should a NeXus attribute that describes one end of a content link also describe the other end? The shared content is the same at both ends; today the class of the content group at each end must define a NeXus attribute for every content attribute.
• Does this PR also close Using soft links instead of hard #1042 (HDF5 soft links are now one way to store a content link)?

5. Five levels of terms

Before

• "group", "field", "attribute" and "link" were used without saying which kind is meant: something in nxdl.xsd, an element of an NXDL file, an item of a class, part of a data file, or a feature of HDF5. That is how the XML attribute interpretation of the NXDL field was read as if it defined the NeXus attribute @interpretation.

After

• A new section, Design > Terms at five levels, names each level once, for example for "attribute":

  Level	Term	Example
  XML Schema (nxdl.xsd)	declaration of the NXDL attribute	<xs:element name="attribute"/>
  NXDL (.nxdl.xml)	NXDL attribute	<attribute name="units"/>
  NeXus (the classes)	NeXus attribute	/NXentry/NXdata/data@units
  Content (one file, any format)	content attribute	/entry/data/data@units="eV"
  Storage	storage attribute	an HDF5 or Zarr attribute

• Each level defines or uses a vocabulary (the names and values that may be written), a grammar (how they may be combined: inside what, how often) and semantics (what it means):

  • XML Schema: nxdl.xsd uses those of XSD, and declares the vocabulary and grammar of NXDL. It declares no semantics: the manual gives the semantics of NXDL, in prose.
  • NXDL: the NXDL files use the vocabulary and grammar of NXDL, and define the NeXus classes.
  • NeXus: the classes define the vocabulary, grammar and semantics (documentation) of content.
  • Content: uses those of the classes, and may contain more than they define.
  • Storage: uses those of its file format; the File Format chapter states how content is stored in it.

• The text this PR touches in the manual, the examples and nxdl.xsd says the level of these words. Older text elsewhere still uses synonyms such as "data item".

To decide

• Do we adopt the five levels and their names for the manual?

Other changes

• NXdata: @primary (deprecated) is defined on AXISNAME, next to @axis.
• Class pages show @NX_class with its obligatory value and the fieldAttributes, and give the target of a NeXus link as "target:" instead of "suggested target:".
• The nxdl.xsd page shows XML attributes as "XML attribute name" instead of "@name", since @ marks a NeXus or content attribute.
• The type of an NXDL group must be a base class, not an application definition (checked by the Python syntax check). No definition uses one today.
• nameType="any" means any name that no other NeXus item of the same NeXus group defines: a specified name takes precedence.
• Advice to writers of data files is stated as rules of the classes, for example "NXobject defines the NeXus field FIELDNAME_errors".
• The MATLAB example writer_2_1.m now writes @target.
• The documentation of about 45 NXDL files is reworded to say the level of these terms (for example "content link" where a data file is meant).

Compatibility

• All existing NXDL files stay valid: nxdl.xsd only gains the declarations of the optional fieldAttributes and the optional valueFrom.
• A data file now fails validation if a content group has no @NX_class, if a content field whose NeXus field has a units category has no @units, or if the content field or group that a NeXus link describes and the one its target describes do not have the same shared content.

Follow-ups, not in this PR

• NXDL: Common base class for NXroot and NXobject #1672: a common base class for NXroot and NXobject, instead of the copy in NXroot.
• NXDL: replace the XML attributes signal, axes, axis, primary of NXDL fields by NXDL attributes #1682: convert the XML attributes signal, axes, axis and primary of the NXDL field (81 NXDL fields in 27 NXDL files) to NXDL attributes.
• NXDL: replace the NXDL link with an XML attribute sameAs of NXDL fields and groups #1671: replace the NXDL link with an XML attribute sameAs of NXDL fields and groups.
• Definitions: clarify what "link" means in some docs #1683: in some definitions it is unclear what "link" means (a content link, a path or a reference): NXimage, NXspectrum, NXellipsometry, NXreflections, NXoptical_spectroscopy, NXmpes_arpes, NXxpcs, NXdelocalization.

[woutdenolf]

Alternative to #1558

[github-actions]

Documentation preview: https://manual.nexusformat.org/pr-preview/pr-1669/ (built from 4d09f5f)

This comment is updated on every push to the branch.

[rayosborn]

I have a fundamental objection to defining NX_class as an attribute in NXobject. It has nothing to do with the NXDL definitions. It is the way that group classes are implemented. By including it as an attribute, you are implicitly allowing a group that is meant to conform to the, e.g., NXsample, class to have NX_class="NXinstrument", which is nonsensical. The push to add it is an unfortunate consequence of us not supporting an API, in which implementation details are contained within the code.

I feel slightly less strongly about the target attribute, although it too is an implementation issue.

[woutdenolf]

you are implicitly allowing a group that is meant to conform to the, e.g., NXsample, class to have NX_class="NXinstrument", which is nonsensical

This PR does not allow that. @NX_class in NXobject has valueFrom="class": its value must be the name of the class the group is validated against.

• A group validated against NXsample must have NX_class="NXsample". A validator rejects NX_class="NXinstrument" on it. The NXsample class page shows "Obligatory value: NXsample".
• This is stated once, in NXobject, and holds for every class that extends it. No class needs its own one-value enumeration.

Why in NXDL: the definitions are the contract that validators and readers use. Every NeXus file must have @NX_class, but today only prose says so. Declaring it in NXDL makes the rule machine-readable. It does not change how group classes are implemented.

@target is in NXDL for the same reason: it is the only way a file marks a link, so the definitions should say what its value must be. The PR description, section "Links", defines this for any storage (hard link, soft link, copy, …).

FYI this PR is still a draft, there are some other corner cases to be dealt with before I propose it to the NIAC. I added comments to other PRs that are somewhat related so the authors are aware of the approach in this PR.

[lukaspie]

@woutdenolf thanks for the draft, I appreciate that you tried to address the points we discussed w.r.t., among others, #1410 and #1558 directly.

fieldType currently defines even more attributes, some of which are (I would guess) meant to be NeXus attributes (like @interpretation, @stride, @data_offset). It gets a bit messy here: some of them are really only relevant for NXdatafields, some are deprecated, for some of them there are even alternatives (e.g. /NXdata/FIELDNAME_offset-field.

How should these fit into this new structure? @interpretation seems to be in used in hyperspy/rosettasciio at least: https://github.com/hyperspy/rosettasciio/blob/8d55492c0d5f28c0d6b62ae96844f6e1cb8eb9c2/doc/supported_formats/nexus.rst?plain=1#L40 and we are seeing issues because of it.

[lukaspie]

Regarding @NX_class, I don't quite understand why it should not be defined in NXDL. Most downstream software depends on it, so it's not simply an implementation detail. I would think the cleanest way to avoid NXsample instances having NX_class="NXinstrument would be to restrict NXsample/@NX_class to NXsample with an enum. This is exactly the same as it is done for the NXentry/definition field in application definitions. This restriction would need to be part of every base class though.

[woutdenolf]

How should these fit into this new structure?

I only added target, units and long_name for now to investigate the approach. I did not look at the others yet (messy as you pointed out). Ideally they would be added in the same way.

This restriction would need to be part of every base class though.

Yes I think that's Ray's comment, not necessarily the presence of NX_class itself. Something needs to fail (ideally the XML validation itself) when the base class does not fix the NX_class attribute to the correct value (enumeration with one value as usual).