[Krishnam24maheshwari]

Summary

Updates the CRDC Education School Harassment or Bullying (CRDC_EducationSchool_HarassmentOrBullying) import pipeline to partition import_inputs by survey year (2012, 2014, 2018, 2021, 2022, and 2024Onwards), add automated data validation rules, and improve parallel processing in run.sh.

Key Changes

1. manifest.json

• Partitioned import_inputs by survey year (2012, 2014, 2018, 2021, 2022, and 2024Onwards) with explicit template_mcf, cleaned_csv, and node_mcf wildcards (processed_output/crdc_harassment_or_bullying_*.mcf).
• Added the 2024Onwards partition to support 2024 and future biennial survey releases without requiring manifest updates on each release.
• Added processed_output/*_counters.txt and validation_config.json to source_files and linked validation_config_file.
• Updated resource_limits (cpu: 16, memory: 128, disk: 200).

2. run.sh

• Added dynamic aggregation and processing of 2024+ CSV files into the crdc_harassment_or_bullying_2024Onwards output partition.
• Sorted historical survey-year CSVs (2012, 2014, 2018, 2021, 2022) by file size descending so the largest jobs start first.
• Passed --output_counters to stat_var_processor.py for each job to emit pipeline telemetry counters.
• Tracked spawned background job PIDs (PIDS) and waited on each job to capture non-zero exit codes (FAILED_JOBS).

3. validation_config.json

Added validation rules tailored to the biennial CRDC release cadence:

• check_deleted_records_percent (DELETED_RECORDS_PERCENT, threshold: 0.1): Allows up to 0.1% deleted records across refreshes to account for natural school closures, mergers, or boundary changes between biennial survey releases.
• check_max_date_consistent (MAX_DATE_CONSISTENT): Ensures all StatVars within each survey-year partition share a consistent latest observation date.
• check_latest_date_freshness (SQL_VALIDATOR): Verifies that the 2024Onwards partition contains observations for year 2024 or later (CAST(MaxDate AS INT) >= 2024), scoped via WHERE CAST(MinDate AS INT) >= 2024 so historical year partitions (input0–input4) are not falsely flagged.

---

Validation & Test Artifacts

• GCS Bucket: 2026_10_22T12_56_32_191837_07_00
• Validation Outputs (validation_output.csv):
  • input0 (2012): validation_output.csv
  • input1 (2014): validation_output.csv
  • input2 (2018): validation_output.csv
  • input3 (2021): validation_output.csv
  • input4 (2022): validation_output.csv
  • input5 (2024Onwards): validation_output.csv

[gemini-code-assist]

Code Review

This pull request updates the CRDC harassment or bullying import configuration to include historical data partitions (2012–2021) and a new 2024Onwards partition, while optimizing the parallel processing script (run.sh) to sort inputs by size and dynamically scale concurrency. It also introduces a validation configuration file (validation_config.json). Feedback focuses on critical issues in the shell script's background job management, where reaped jobs will cause incorrect failure counts and exit statuses are discarded, as well as a potential crash in the SQL validator if date fields are not safely parsed before casting to integers.