Skip to content

BUG: C18: File parsing and output path checks in 22 filters #1877

Open

Description

@imikejackson

Summary

2026-11-13: PRs #1961, #1962, #1965, #1967, #1968, #1969, #1970, #1971 and #1972 were closed and combined into #1973, which is built and tested with the combined EbsdLib branch BlueQuartzSoftware/EbsdLib#86 (it holds EbsdLib #70 to #85, which were closed into it). Items that named one of these pull requests now also name #1973. No item is ticked until #1973 is merged.

Some filters read text files, parse numbers from strings, or derive output paths, and do not check the result. Hand-written readers index tokens without a size check and loop on eof(). String-to-number conversion accepts text that is only partly a number. Output names are made from array names or suffix rules, and preflight does not compare the final paths. Malformed input then crashes or hangs preflight, or the filter writes wrong data or overwrites a file with no error.

As of develop 419f9db (2026-10-21) this class has 30 open findings in 22 filters: 8 CRASH, 10 WRONG_DATA, 12 MINOR. An automated code-reading sweep found them on develop 4a7b71e5f (2026-10-08). Each finding was re-checked by reading the code on develop 419f9db. Each finding is a candidate: nobody has run it yet.

Mechanism

This class has three sub-patterns.

1. String-to-number parsing accepts partial input. The shared converter StringInterpreterFromType calls std::stoll, std::stoull, std::stof and std::stod without the pos argument. These functions stop at the first character that is not part of a number and do not report it. StringInterpretationUtilities.hpp#L86-L94:

      else
      {
        // Default: is signed and not float
        int64 value = std::stoll(input);
        if(value > std::numeric_limits<T>::max() || value < std::numeric_limits<T>::min())
        {
          return nx::core::MakeErrorResult<T>(-10353, fmt::format("Overflow error trying to convert '{}' to type '{}' using function '{}'", input, typeName, detail::TypeToFuncName<T>()));
        }
        outputValue = static_cast<T>(value);

So "2.5" becomes 2, "1e3" becomes 1 and "12abc" becomes 12, with no error. The unsigned branch rejects a leading - only at index 0 (line 74), so " -1" wraps to UINT64_MAX. Every filter that uses this helper (or the CsvParser helpers) gets the same behavior.

2. Text header and line parsing without token-count, EOF, line-length or CR checks. Each reader has its own copy of the line and token code. For example ReadVolumeGraphicsFileFilter.cpp#L144-L146:

  while(buf[0] != '[' && !inputFile.eof())
  {
    std::vector<std::string> tokens = StringUtilities::split(buf, ' ');
    if(tokens[0] == k_FileFormat)

An empty line gives an empty tokens vector, so tokens[0] reads out of bounds. Related forms in other readers: while(!eof()) that never ends on a failed stream, gcount() == length checks that are never true after getline(buf, length), width(length) with no terminating NUL, and a trailing \r (CRLF) or UTF-8 BOM that stays in the token.

3. Output path or name collisions and nested selections are not detected. The output name is made from getTargetName() or from a suffix rule, and preflight compares the raw parameter values (or nothing). Two outputs can then resolve to one file or one DataPath, the output can be the input, the destination can be inside the source, or a selection can contain a path twice or a path and its parent. The result is a silent overwrite, an infinite recursion, or a late generic 'already exists' / 'does not exist' error.

The correct form for sub-pattern 3 already exists on develop in WriteASCIIDataFilter.cpp#L138-L149 (from #1765). Preflight compares the final output names, ignoring case, and returns a clear error:

    std::map<std::string, DataPath> outputNames;
    for(const auto& arrayPath : selectedDataArrayPaths)
    {
      // Some file systems treat names that differ only by case as the same file.
      const auto [iter, inserted] = outputNames.emplace(StringUtilities::toLower(arrayPath.getTargetName()), arrayPath);
      if(!inserted)
      {
        return MakePreflightErrorResult(k_DuplicateOutputName,
                                        fmt::format("In Multiple Files mode, DataArrays '{}' and '{}' have the same output name when case is ignored. Select DataArrays with different names.",
                                                    iter->second.toString(), arrayPath.toString()));
      }
    }

WriteBinaryDataFilter writes files with the same naming rule but does not have this check.

Rule

Decision needed: no rule for this class is decided yet. Proposed rule:

  • Numbers: A string converts to a number only if the whole string (after trimming whitespace) is consumed. Otherwise the conversion returns an error. Parsing does not depend on the user's locale. Unparsable text for a bool is an error.
  • Text readers: Every reader checks that the file opened, loops on the success of the read (not on eof()), removes a trailing \r and a leading UTF-8 BOM, and checks the token count before it reads a token. A malformed line gives an error with the line number.
  • Output paths: Preflight resolves every output to its final path or DataPath (after suffixes and prefixes; absolute and normalized; case-insensitive where the file system can be). Preflight rejects duplicate outputs, output equal to an input, a destination equal to or inside the source, and selections that contain a path twice or a path and its ancestor (or removes them with a warning; decide which).

How to fix

  1. Numbers (ConditionalSetValueFilter, CreateDataArrayAdvancedFilter, ReadTextDataArrayFilter#3, ArrayCalculatorFilter): fix StringInterpreterFromType and Convert<T> in src/simplnx/Utilities/StringInterpretationUtilities.hpp once. Trim the input with StringUtilities::trimmed (StringUtilities.hpp#L224), then parse with std::from_chars and require that it consumes the whole string. (std::from_chars is already used in ReadOnScaleTableFile.cpp; floating-point from_chars support depends on the standard library version, so confirm it on all CI compilers, or use std::stod with pos and a C-locale stream.) Keep the existing error codes -10350, -10351 and -10353 where they fit. In ArrayCalculator.cpp, require that each bracket token is only digits and that the commas are exact.
  2. Text readers: Add one shared line reader (for example in src/simplnx/Utilities/) that reads with std::getline into std::string, removes \r and a BOM, and reports the line number. Add one token accessor that checks the count and returns an error. Use them in ReadBinaryCTNorthstarFilter, ReadVolumeGraphicsFileFilter, ReadDeformKeyFileV12Filter, ReadVtkStructuredPointsFilter, ReadCSVFileFilter, ConvertHexGridToSquareGridFilter and CsvParser. In CsvParser.hpp, check the result of in >> value for every element, including the last. In ReadDeformKeyFileV12.cpp, resize to numVars, not 12.
  3. Output paths and selections: Add one shared preflight helper that takes the final output paths and returns an error for duplicates. Use the WriteASCIIDataFilter check as the model. Apply it to the file writers (WriteBinaryDataFilter, WriteNodesAndElementsFilesFilter, ComputeGBCDMetricBasedFilter, ComputeGBPDMetricBasedFilter, ConvertHexGridToSquareGridFilter) and to the DataPath outputs (ComputeArrayHistogramFilter, QuickSurfaceMeshFilter, CreateGeometryFilter, CombineStlFilesFilter). Add a destination-under-source check to CopyDataObjectFilter, and duplicate/ancestor checks to DeleteDataFilter, MoveDataFilter and CombineAttributeArraysFilter. In CreateGeometry*Action, also check the result of deepCopy for nullptr.
  4. Put each check in preflight. Only a check that needs file contents that preflight does not read can go in execute.
  5. New errors use the error-code range of that filter. Codes are more negative than -10000, and each filter uses one range of 100 codes (for example, first code -47500 means all codes -47500..-47599). Check uniqueness with git grep -nE -- "-NNNN([^0-9]|$)" (do not use \b; git grep -E ignores it) on develop and on open PR refs.
  6. Remove execute-time checks that a new preflight check makes unreachable (for example the raw-path compare in ConvertHexGridToSquareGrid.cpp). Do not keep duplicate checks.

How to test

  • Write the failing test first. Each checklist line gives the trigger input.
  • Numbers: Call the converter (or run the filter) with 2.5, 1e3, 12abc and -1 for integer types, and check that each gives an error. Check that valid values (2, -1, 1.5 for float) still convert to the correct value.
  • Text readers: Write small malformed files in the test (empty line, missing closing tag, short line, CRLF line endings, BOM, line longer than the buffer, last line with no newline). Check that preflight returns an error with a clear message and does not crash or hang. For ReadTextDataArrayFilter#1, use the file 1,2,3 with 4 tuples and check that the filter returns an error. For ReadCSVFileFilter#3, check that the created DataArray names have no \r and no BOM.
  • Output paths: Set two outputs that resolve to the same file or DataPath and check for the new preflight error. For ConvertHexGridToSquareGridFilter#3, check that the input file is not changed. For CopyDataObjectFilter#1, check that preflight returns an error and does not recurse.

Conventions

  • Make one PR for the whole class. Make one commit for each filter.
  • Start each commit message with BUG:. Add a Signed-off-by line. Do not add AI attribution.
  • Start every fix with a unit test that fails on develop. If you cannot reproduce a finding, write that in the PR and on this issue with the reason. Do not force a fix.
  • Value checks need an independent expected result (oracle). Do not use the output of the code under test as the expected result.
  • Run ctest serially. Do not use -j, because the tests depend on each other.
  • Format with clang-format 19.1.1 and the repository .clang-format.
  • Do not add sentences to the filter documentation about input requirements. The generated parameter table documents them.
  • If a fix touches a filter that has a V&V report (src/Plugins/<Plugin>/vv/<Filter>.md), update every file of the V&V package of that filter (report, deviations, provenance, test tables). V&V documents reference DREAM3D 6.5.171 only.
  • Rebase on develop. Do not merge develop into the branch.

Checklist

Related issues

Out of scope

  • No finding of this class is in a filter that has a V&V report (0 findings excluded).

Activity

  1. changed the title [-]BUG: File parsing and output path checks in 22 filters[/-] [+]BUG: C18: File parsing and output path checks in 22 filters[/+] on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions