[goutamabrol]

Summary

Refactors the US_UrbanSchool_Algebra1 (statvar_imports/school_algebra1) pipeline to separate raw data downloading from preprocessing, standardizes ncesid and YEAR generation across CRDC survey years (2009–2024), and updates the manifest and test fixtures.

PR Checklist doc - link

Changes

• download_script.py: Decoupled in-place file mutation from downloading so raw CRDC files (.csv and .xlsx) in input_files/ remain intact; added dynamic survey year range detection, fail-fast checks for known survey years (2009–2023), HEAD Content-Type validation for speculative future years, and skipping for existing downloads.
• preprocess_data.py & preprocess_data_test.py: Added a dedicated preprocessing script (with unit tests) that reads raw files from input_files/, constructs zero-padded 12-digit ncesid values (LEAID + SCHID or COMBOKEY), extracts the 4-digit survey completion YEAR, masks negative CRDC reserve codes (< 0), filters to columns mapped in Algebra1_pvmap.csv, and writes clean CSVs to processed_files/.
• Algebra1_pvmap.csv: Aligned property-values (schoolEnrollment: EnrolledInSchool, abilityToSpeakEnglish: LimitedEnglishProficiency, ethnicity: Hispanic, race: AmericanIndianOrAlaskaNativeAlone, race: NativeHawaiianOrOtherPacificIslanderAlone) with canonical CRDC StatVars in us_urbaneducation_school_stat_vars.mcf.
• Algebra1_metadata.csv: Removed unused unit and scalingFactor columns from output_columns (observationDate,observationAbout,variableMeasured,value) and enabled generate_statvar_name,True.
• manifest.json & validation_config.json:
  • Added preprocess_data.py to scripts and updated stat_var_processor.py to read processed_files/*.csv and output counters to counters/algebra1_counters.csv.
  • Added counters/*.csv and validation_config.json to source_files, and "node_mcf": "output/*.mcf" to import_inputs.
  • Updated resource_limits (cpu: 8, memory: 32, disk: 100) and configured validation_config.json (DELETED_RECORDS_PERCENT, LINT_ERROR_COUNT, and SQL_VALIDATOR freshness checks for active multi-grade and historical Grade 7/8 StatVars).
• test_data/ & README.md: Regenerated input_test_data.csv and sample_output.csv to match the zero-padded ncesid, 4-column output schema, and canonical CRDC StatVars; updated README.md to document the 3-step workflow, 2009–2024 coverage, and excluded CRDC columns.

Cloud Batch job (test) - link

Validation - link

CRA = link

[gemini-code-assist]

Code Review

This pull request refactors the school algebra 1 data import workflow by introducing a dedicated preprocessing step (preprocess_data.py), updating the download script to skip already downloaded files, and updating the manifest, metadata, and test data accordingly. Feedback on the changes suggests optimizing the survey year generation in download_script.py to use a step of 2 for biennial years, which avoids unnecessary download attempts for non-existent odd years.