[gmechali]

Overview

Major performance and scalability refactor for the Simple Ingestion Preprocessor. Large observation CSV files are now chunked across worker processes using parallel byte ranges, eliminating straggler cores on high-CPU machines.

Key Changes

• Parallel Byte-Range CSV Chunking: Files $\ge 8\text{ MB}$ are split into $8\text{ MB}$ tasks ($N = \lceil\text{size} / 8\text{ MB}\rceil$), seeking to row boundaries. MCF files are prioritized first to declare sources and provenances upfront.
• Fast Cross-Part Collision Detection: Workers track uint64 observation hashes; the master process aggregates and detects cross-part ID collisions in C via np.unique.
• Concurrent Header Validation: Upfront CSV header validation scaled across 128 threads to fail fast with aggregated diagnostics.
• GCS & I/O Optimization: Worker GCS client pooling, cached bucket checks, elimination of redundant directory marker writes, and 1-entry consecutive metadata caching.
• Modular Code Cleanup: Decomposed runner.py, jsonld_stream_writer.py, and variable_per_row_importer.py into clean, testable helper functions.

Performance (UN / WHO: 2,403 files on 64 cores)

• Total VM runtime: Slashed from ~10+ min down to 143 s (active compute 127 s).
• Largest file (90.2 MB): Execution dropped from 75–80 s down to 29 s (split across 11 chunks).

Testing

• 116/116 unit tests passing (added tests for task planning, concurrent header validation, and end-to-end multi-process execution with splitting).
• Golden JSON-LD output deep comparisons confirm 0 diffs.
• Synchronized across import/simple and website/import/simple.

[codacy-production]

Not up to standards ⛔

🔴 Issues 3 high · 7 medium · 51 minor

Alerts:
⚠ 61 issues (≤ 0 issues of at least minor severity)

Results:
61 new issues

Category	Results
Documentation	21 minor
ErrorProne	2 high
Security	2 medium 1 high
CodeStyle	30 minor
Complexity	5 medium

View in Codacy

🟢 Metrics 166 complexity

Metric	Results
Complexity	166

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request introduces significant performance optimizations for the ingestion pipeline, including parallel processing of large observation CSVs by splitting them into chunks, compact JSON-LD serialization, and optimized GCS file listing and streaming with ranged reads. A critical issue was identified in the runner where progress, final status, and unresolved entities from worker processes are not merged back into the master process's reporter, which would result in an incomplete or incorrect final report.json.

[gmechali]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces parallel processing and performance optimizations for importing large datasets, including splitting large observation CSVs into chunks for multi-process execution, optimizing GCS file listing and connection pooling, and implementing caching for serialization. The review feedback highlights several critical correctness and concurrency issues: a potential silent data loss bug in the byte-range splitting logic when a split falls exactly on a line boundary, race conditions in concurrent cache modifications and GCS storage initialization, race conditions in parallel MCF/CSV task ordering, and a potential crash when parsing the HEADER_VALIDATION_MAX_WORKERS environment variable.