[shvngisingh]

Overview

Adds the automated import and validation configuration for Commerce_EDA_Poverty (statvar_imports/commerce_eda_poverty), which ingests county-level and island-territory poverty percentage rates from the U.S. Economic Development Administration (EDA) Persistent Poverty Counties (PPCs) dataset (EDA_FY23_PPCs.xlsx).

Key Changes

1. Download Pipeline (download_poverty.py & download_poverty_test.py):

   • Downloads the official EDA_FY23_PPCs.xlsx workbook from eda.gov (with automatic fallback to the Wayback Machine mirror if bot protection blocks automated requests, or local file ingestion via --input_file) by delegating HTTP requests to util.download_util.request_url.
   • Verifies the ZIP/XLSX magic-byte header (PK\x03\x04) before atomically saving input_files/EDA_FY23_PPCs.xlsx and extracting the Underlying_Data worksheet into input_files/Poverty.csv.
   • Defaults raw_csv_path to None so an unused duplicate CSV is not written to output/Poverty_original.csv unless explicitly requested.

2. Preprocessing & PV Mapping (process_poverty.py, process_poverty_test.py, & poverty_pvmap.csv):

   • Dynamically locates the GEOID header row, standardizes 5-digit county and island-territory FIPS codes (filtering out state-level XX000 summary rows), and validates percentage bounds within [0.0, 100.0].
   • Reshapes wide historical Decennial Census columns (1990, 2000, and any future <YEAR> ... % in Poverty columns) and recent estimates (2020 Island Areas Decennial Census for AS, GU, MP, VI; 2021 SAIPE / ACS 5-Year for the 50 states, DC, and PR) into normalized (GEOID, year, poverty_rate) rows.
   • Deduplicates observations by ["GEOID", "year"] while preserving deterministic input row ordering.
   • Maps Decennial Census benchmark years (1990, 2000, 2010, 2020, 2030, 2040, 2050) to measurementMethod: dcs:USDecennialCensus and SAIPE/ACS years (2021 and default) to measurementMethod: dcs:EDA_Estimate in poverty_pvmap.csv.

3. Manifest & Validation Configuration (manifest.json & validation_config.json):

   • Configures node_mcf: "output/*.mcf" in manifest.json and restricts source_files to raw inputs, counters, and validation_config.json.
   • Configures 3 validation rules in validation_config.json (in addition to the 3 built-in runner checks):
     • check_date_span_sql (SQL_VALIDATOR): Verifies CAST(MinDate AS INTEGER) <= 1990 AND CAST(MaxDate AS INTEGER) >= 2021 AND total_svs > 0.
     • check_max_date_consistent (MAX_DATE_CONSISTENT): Verifies uniform MaxDate across all statistical variables (2021).
     • check_deleted_records_percent (DELETED_RECORDS_PERCENT): Enforces threshold: 0.1 (0.1% maximum deleted records).

---

Test Environment Verification (datcom-infosys-dev)

• Batch Job ID: commerce-eda-poverty-shvngisingh-20261022-072318 (SUCCEEDED, ImportStatus.STAGING, 9,689 rows processed, 0 lint errors)
• Validation Result: 6/6 rules PASSED (check_deleted_records_percent, check_empty_import, check_missing_refs_count, check_lint_error_count, check_date_span_sql, check_max_date_consistent)
• Batch Job Execution Logs: link
• GCS Test Output Folder (2026_10_22T00_27_19_036019_07_00): link
• Validation Output (validation_output.csv): link
• Differ Summary (differ_summary.json): link
• GenMCF Summary Report (summary_report.csv):link
• GenMCF Report JSON (report.json): link

---

Local Unit Tests

python3 -m unittest discover -s statvar_imports/commerce_eda_poverty -p "*test*.py" -v
# Ran 30 tests in 0.244s — OK

[gemini-code-assist]

Code Review

This pull request introduces a new data importer for the Commerce EDA Poverty dataset, which tracks Persistent Poverty Counties. It includes metadata, mapping files, generated outputs, and a preprocessing script (process_poverty.py). The review feedback identifies two key issues: a hardcoded absolute project path in the preprocessing script that will cause failures in other environments, and an incorrect place type (AdministrativeArea1 instead of County) in the commented-out metadata configuration.

[saanikaaa]

If this is new import, we should have a CL as well to bring textproto. Mcf and dataset info about this import