Skip to content

fix(preprocessor): always emit dc/g/Root StatVarGroup in Nodes.__init__ - #796

Merged
clincoln8 merged 3 commits into
datacommonsorg:masterfrom
clincoln8:fix/always-emit-root-svg
Oct 19, 2026
Merged

clincoln8 merged 3 commits into
datacommonsorg:masterfrom
clincoln8:fix/always-emit-root-svg

Conversation

@clincoln8

@clincoln8 clincoln8 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Goal

Ensure the global root StatVarGroup (dc/g/Root, "Data Commons Variables") is always emitted by the preprocessor into _global triples so that manually defined StatVarGroup hierarchies in MCF have a valid root Node row in Spanner without requiring database seeding.

Key Aspects of the Approach

  • Unconditionally call self._ensure_root_group() in Nodes.__init__() (simple/stats/nodes.py).
  • Keep c/g/Root ("Custom Variables") gated behind if config.generate_hierarchy(): self.group("") (or explicit nodes.group("") calls) so unused custom root groups are not created for manually defined StatVarGroup hierarchies.

Implementation Details

  • Previously, self._ensure_root_group() (introduced in [DCP Init DB] Add the dc/g/Root node in preprocessor, exactly like c/g/Root. Allows us to remove Seeding entirely #780) was only called inside Nodes._default_custom_group() and Nodes.group(), which are not reached in standard dcpbridge imports using variablePerRow CSVs (VariablePerRowImporter) and manually defined StatVarGroup nodes in MCF (McfImporter) when groupStatVarsByProperty is not enabled. As a result, dc/g/Root was absent from the Spanner Node table, causing Mixer's BulkVariableGroupInfo (FilterNodesByTypes) to reject dc/g/Root and return an empty StatVar Explorer hierarchy.
  • Updated test_root_stat_var_group_created in simple/tests/stats/nodes_test.py to verify that dc/g/Root is initialized immediately on Nodes(CONFIG) construction while c/g/Root is not created unless requested.
  • Updated expected golden triples/JSON-LD shards in entities_importer and runner tests to include dc/g/Root.
  • Pinned --python 3.12 on uv run pytest for ingestion-helper in run_test.sh so uv does not resolve requires-python = ">=3.12" to CPython 3.15 in Cloud Build and time out building wheels from source.

Edge Cases, Errors & Logging

  • _ensure_root_group() is idempotent (if sc.ROOT_SVG_ID not in self.groups:), so subsequent calls to nodes.group() or _default_custom_group() reuse the existing dc/g/Root entry without duplication.

Testing & E2E Verification

  • Unit/Integration Tests: Ran the full simple test suite (pytest tests/ kg_util/, 127 passed) and verified yapf/isort formatting.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric Results
Complexity 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that the global root StatVarGroup (dc/g/Root) is always created by calling _ensure_root_group() during the initialization of the Nodes class. This guarantees that manually defined StatVarGroup hierarchies in MCF have a root Node in Spanner. The corresponding tests and expected test data files (CSV and JSON-LD) have been updated to reflect the presence of this root group. There are no review comments to address, and the changes look solid.

@clincoln8
clincoln8 requested a review from gmechali October 16, 2026 00:54
@clincoln8
clincoln8 merged commit b667970 into datacommonsorg:master Oct 19, 2026
11 checks passed
@clincoln8
clincoln8 deleted the fix/always-emit-root-svg branch October 19, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants