diff --git a/statvar_imports/commerce_eda_poverty/README.md b/statvar_imports/commerce_eda_poverty/README.md
new file mode 100644
index 0000000000..8d1b64df21
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/README.md
@@ -0,0 +1,148 @@
+# Commerce EDA Poverty: Persistent Poverty County Status and Rates
+
+Author: Shivangi Singh
+Date: *September 2026*
+
+| Parameter | Details |
+| :--- | :--- |
+| Import Type | **Automated Import** (raw dataset downloaded directly from source website or local input file, and processed locally) |
+| Link to dataset preview or raw data | [EDA Persistent Poverty Counties](https://www.eda.gov/performance/resources/persistent-poverty-counties) |
+| Direct Workbook URL | [`EDA_FY23_PPCs.xlsx`](https://www.eda.gov/sites/default/files/2023-03/EDA_FY23_PPCs.xlsx) |
+| Archive Mirror URL | [`EDA_FY23_PPCs.xlsx` (Wayback Machine)](https://web.archive.org/web/20250308204521if_/https://www.eda.gov/sites/default/files/2023-03/EDA_FY23_PPCs.xlsx) |
+| Place types covered | U.S. Counties, County Equivalents, and Island Territories (`County` / `AdministrativeArea1`) |
+| Place ID resolution | `country/USA` FIPS (5-digit county and county-equivalent FIPS codes, `geoId/XXXXX`) |
+| Date range covered | 1990, 2000, 2020, 2021 (1990 Decennial Census, 2000 Decennial Census, 2020 Island Area Decennial Census, 2017–2021 ACS 5-Year) |
+| Statistical Variables | `Count_Person_BelowPovertyLevelInThePast12Months_AsFractionOf_Count_Person` |
+| Unit / Scaling | `Percent` / `100` |
+| Refresh Cycle | Annual automated check (`30 05 1 1 *`) aligned with EDA/Census releases |
+
+---
+
+## Overview
+
+This automated dataset import fetches and processes historical and recent county-level poverty percentage rates published by the U.S. Economic Development Administration (EDA) (`https://www.eda.gov/performance/resources/persistent-poverty-counties`) for Persistent Poverty Counties (PPCs) and all benchmarked U.S. counties.
+
+The download script (`download_poverty.py`) downloads the official `EDA_FY23_PPCs.xlsx` workbook directly from EDA (with automatic fallback to the Wayback Machine archive mirror if Cloudflare bot detection blocks automated requests, or from a local file via `--input_file`) into `input_files/EDA_FY23_PPCs.xlsx`. It extracts the `Underlying_Data` sheet (3,241 county rows) into `input_files/Poverty.csv`.
+
+The preprocessing script (`process_poverty.py`) reads the downloaded source file locally, cleans and standardizes 5-digit FIPS codes and poverty percentages into normalized `(GEOID, year, poverty_rate)` records in `output/Poverty_cleaned.csv`, and feeds the cleaned data into `stat_var_processor.py`. Neither script relies on Google Cloud Storage (GCS) staging.
+
+The dataset benchmarks poverty rates across statutory periods:
+- **1990**: 1990 Decennial Census (`year=1990`)
+- **2000**: 2000 Decennial Census (`year=2000`)
+- **2020**: 2020 Island Areas Decennial Census (`year=2020` for 11 territory county equivalents across FIPS prefixes `60`, `66`, `69`, `78`)
+- **2021**: 2021 Small Area Income and Poverty Estimates (`"SAIPE, 2021"`, `year=2021` for 3,143 counties across all 50 states and DC) and 2017–2021 ACS 5-Year Estimates (`"ACS 5-YEAR, 2017-2021"`, `year=2021` for all 78 Puerto Rico municipios; `3,143 + 78 = 3,221`)
+
+It covers 3,232 U.S. counties, county equivalents, and island territories (`3,143` state/DC counties + `78` Puerto Rico municipios + `11` island territory county equivalents). Island territories in the EDA dataset use 5-digit county-equivalent FIPS codes (e.g., 60010 for Eastern District, AS; 66010 for Guam; 69085 for Northern Islands, MP; 78010 for St. Croix, VI).
+
+> [!NOTE]
+> Direct automated HTTP requests to `eda.gov` may encounter Cloudflare bot protection (HTTP 403 Forbidden). `download_poverty.py` automatically falls back to an archive mirror of the official FY23 workbook. For manual/semi-automated refresh when upstream releases a new workbook, operators can download via a browser and provide it locally via `--input_file`.
+>
+> **Legacy `Commerce_EDA` Migration Note:** This standalone `Commerce_EDA_Poverty` import (`dc/base/Commerce_EDA_Poverty`) splits the Persistent Poverty Counties table out of the legacy combined `Commerce_EDA` import (`statvar_imports/commerce_eda`, alongside `Commerce_EDA_USAspending`). In accordance with Data Commons migration practice, the legacy `Commerce_EDA` import is retained temporarily until all split imports are merged and verified in production, and will be retired in an immediate follow-up cleanup PR/CL tracked under `b/548896854`.
+
+### Why Preprocessing is Required (Why PV Map Alone is Insufficient)
+
+The raw EDA dataset cannot be mapped directly into Data Commons observations using `stat_var_processor.py` / `poverty_pvmap.csv` alone because:
+1. **Format Reshaping (Wide to Long)**: The raw sheet contains one row per county with multiple poverty percentage columns corresponding to different time periods (`1990 Decennial Census, % in Poverty`, `2000 Decennial Census, % in Poverty`, and `Most Recent Estimate, % in Poverty*`). PV mapping requires normalized long-format records with explicit observation dates.
+2. **Dynamic & Heterogeneous Observation Dates**: The "Most Recent Estimate" is not uniform across places:
+ - For all 50 U.S. states and DC (3,143 counties), the benchmark is the 2021 Small Area Income and Poverty Estimates (`"SAIPE, 2021"` -> `year=2021`).
+ - For Puerto Rico (78 municipios; `3,143 + 78 = 3,221`), because the Island Areas Census does not cover Puerto Rico and SAIPE data is unavailable, EDA uses the 2017–2021 American Community Survey 5-Year Estimates (`"ACS 5-YEAR, 2017-2021"` -> end year `year=2021`).
+ - For Island Territories (11 county equivalents across American Samoa, Guam, Northern Mariana Islands, and U.S. Virgin Islands: FIPS prefixes `60`, `66`, `69`, `78`; `3,221 + 11 = 3,232`), SAIPE does not produce estimates; the benchmark is the 2020 Island Areas Decennial Census (`"Decennial Census, 2020"` -> `year=2020`).
+ - The survey year is parsed dynamically from the `Data Source―Most Recent Estimate` column (extracting the final 4-digit year so multi-year ranges like `"ACS 5-YEAR, 2017-2021"` resolve to `2021`, or falling back to territory FIPS prefix). Note that `poverty_pvmap.csv` intentionally sets `observationPeriod: P1Y` across all 3,232 places (including the 78 Puerto Rico municipios) to maintain a uniform facet across the entire EDA Persistent Poverty benchmark series. PV map CSV configurations cannot conditionally branch date assignment based on state FIPS code or evaluate regex over free-text notes.
+3. **Data Cleaning & Filtering**:
+ - Upstream Excel/CSV files contain non-tabular preamble headers, subtitles, and footnote rows that must be stripped.
+ - Raw county FIPS codes in Excel often drop leading zeros (e.g. `1001` instead of `01001`) and include state-level summary rollups (e.g. `01000` for Alabama statewide) that must be filtered out so they are not ingested as county entities.
+
+---
+
+## Statistical Variable
+
+```mcf
+Node: dcid:Count_Person_BelowPovertyLevelInThePast12Months_AsFractionOf_Count_Person
+typeOf: dcid:StatisticalVariable
+name: "Population: Below Poverty Level in The Past 12 Months (Per Capita)"
+populationType: dcid:Person
+measuredProperty: dcid:count
+statType: dcid:measuredValue
+measurementDenominator: dcid:Count_Person
+povertyStatus: dcid:BelowPovertyLevelInThePast12Months
+```
+
+---
+
+## Working Directory Context
+
+Commands in this workflow depend on the working directory:
+- **Module directory (`statvar_imports/commerce_eda_poverty/`)**: Execute download, preprocessing, and pipeline scripts (`download_poverty.py`, `process_poverty.py`, `stat_var_processor.py`).
+- **Repository root (`data/`)**: Execute validation runner, test scripts (`./run_tests.sh`), and unittest module invocations.
+
+---
+
+## Pipeline Execution (Automated Import)
+
+### Prerequisites
+Ensure Python dependencies are available:
+```bash
+pip install pandas openpyxl requests absl-py duckdb
+```
+
+For running `stat_var_processor.py` locally, authenticate Google Cloud Application Default
+Credentials to allow reading the central schema definitions from Google Cloud Storage:
+```bash
+gcloud auth application-default login
+```
+*(Required to read `--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf`).*
+
+### 1. Download Source Dataset (`download_poverty.py`)
+Run from `statvar_imports/commerce_eda_poverty/`. Downloads the official `EDA_FY23_PPCs.xlsx` workbook directly from EDA (or archive mirror / local input file) and extracts the `Underlying_Data` sheet to `input_files/Poverty.csv`:
+```bash
+cd statvar_imports/commerce_eda_poverty
+python3 download_poverty.py
+```
+
+To ingest a local copy directly without downloading:
+```bash
+python3 download_poverty.py --input_file=input_files/EDA_FY23_PPCs.xlsx
+```
+
+### 2. Preprocess Dataset (`process_poverty.py`)
+Run from `statvar_imports/commerce_eda_poverty/`. Ingests the downloaded source file locally, standardizes 5-digit county and county-equivalent island territory FIPS codes, partitions the recent rates into 2020 (island territories) and 2021 (states, DC, PR), validates survey year bounds [2020..current year], enforces percentage value bounds $[0.0, 100.0]$, and atomically outputs `output/Poverty_cleaned.csv`:
+```bash
+python3 process_poverty.py
+```
+
+### 3. Generate Data Commons Observations (`stat_var_processor.py`)
+Run from `statvar_imports/commerce_eda_poverty/`. Matches the scripts configured in `manifest.json`:
+```bash
+python3 ../../tools/statvar_importer/stat_var_processor.py \
+ --input_data=output/Poverty_cleaned.csv \
+ --pv_map=poverty_pvmap.csv \
+ --config_file=poverty_metadata.csv \
+ --output_path=output/Poverty_output \
+ --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
+ --output_counters=counters/Poverty_counters.csv
+```
+
+---
+
+## Validation
+
+After running the Data Commons `import_tool` (`genmcf` and differ steps, which produce `summary_report.csv`, `report.json`, and `differ_summary.json`), validate the generated outputs using the Import Validation Framework and `validation_config.json`. Run from the repository root `data/`:
+```bash
+python3 -m tools.import_validation.runner \
+ --validation_config=statvar_imports/commerce_eda_poverty/validation_config.json \
+ --stats_summary=statvar_imports/commerce_eda_poverty/output/summary_report.csv \
+ --differ_output=statvar_imports/commerce_eda_poverty/output \
+ --lint_report=statvar_imports/commerce_eda_poverty/output/report.json \
+ --validation_output=statvar_imports/commerce_eda_poverty/output/validation_report.json
+```
+
+## Testing
+
+Run unit tests verifying workbook download and mirror failover, local file ingestion, GEOID standardization, and value sanitation from the repository root `data/`:
+```bash
+python3 -m unittest discover -s statvar_imports/commerce_eda_poverty -p "*test*.py"
+```
+Or via the test runner script:
+```bash
+./run_tests.sh -p statvar_imports/commerce_eda_poverty
+```
diff --git a/statvar_imports/commerce_eda_poverty/download_poverty.py b/statvar_imports/commerce_eda_poverty/download_poverty.py
new file mode 100644
index 0000000000..ce16b79b98
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/download_poverty.py
@@ -0,0 +1,415 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Download script for Commerce EDA Persistent Poverty Counties (PPC) dataset.
+
+This script fetches the official Persistent Poverty Counties dataset from the
+U.S. Economic Development Administration (EDA) / Department of Commerce:
+ https://www.eda.gov/performance/resources/persistent-poverty-counties (EDA_FY23_PPCs.xlsx)
+It downloads the official Excel workbook (with automatic mirror failover),
+supports ingesting directly from an existing input file, extracts the underlying
+county-level poverty data table (3,232 places across 1990, 2000, and 2020/2021),
+and stages the workbook and extracted CSV under input_files/ for preprocessing
+(and optionally writes a raw CSV copy when --raw_csv_path is explicitly provided).
+"""
+
+import csv
+import io
+import os
+import shutil
+import sys
+import tempfile
+
+from absl import app, flags, logging
+import openpyxl
+
+MODULE_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.abspath(os.path.join(MODULE_DIR, "..", ".."))
+if PROJECT_ROOT not in sys.path:
+ sys.path.insert(0, PROJECT_ROOT)
+
+from util.download_util import request_url
+
+EDA_PPC_XLSX_URL = (
+ "https://www.eda.gov/sites/default/files/2023-03/EDA_FY23_PPCs.xlsx"
+)
+EDA_PPC_MIRROR_URL = (
+ "https://web.archive.org/web/20250308204521if_/"
+ "https://www.eda.gov/sites/default/files/2023-03/EDA_FY23_PPCs.xlsx"
+)
+
+DEFAULT_OUTPUT_DIR = os.path.join(MODULE_DIR, "input_files")
+DEFAULT_OUTPUT_XLSX = os.path.join(DEFAULT_OUTPUT_DIR, "EDA_FY23_PPCs.xlsx")
+DEFAULT_INPUT_CSV = os.path.join(DEFAULT_OUTPUT_DIR, "Poverty.csv")
+DEFAULT_RAW_CSV_FILE = None
+
+HTTP_HEADERS = {
+ "User-Agent": (
+ "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+ "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+ ),
+ "Accept": "*/*",
+}
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string(
+ "source_url",
+ EDA_PPC_XLSX_URL,
+ "Primary URL to download the Persistent Poverty Counties Excel workbook.",
+)
+flags.DEFINE_string(
+ "mirror_url",
+ EDA_PPC_MIRROR_URL,
+ "Fallback mirror URL to download the Persistent Poverty Counties Excel "
+ "workbook.",
+)
+flags.DEFINE_string(
+ "input_file",
+ None,
+ "Optional path to a local input file (.xlsx or .csv) to use instead of "
+ "downloading.",
+)
+flags.DEFINE_string(
+ "output_xlsx_path",
+ DEFAULT_OUTPUT_XLSX,
+ "Destination path to save the downloaded source Excel workbook.",
+)
+flags.DEFINE_string(
+ "output_csv_path",
+ DEFAULT_INPUT_CSV,
+ "Destination path to save the extracted input CSV file.",
+)
+flags.DEFINE_string(
+ "raw_csv_path",
+ DEFAULT_RAW_CSV_FILE,
+ "Optional path to save an extracted raw CSV copy of the workbook.",
+)
+flags.DEFINE_integer(
+ "max_retries",
+ 3,
+ "Maximum number of download retry attempts per URL.",
+)
+flags.DEFINE_integer(
+ "timeout",
+ 60,
+ "HTTP request timeout in seconds.",
+)
+
+
+def download_file(
+ download_url,
+ output_path,
+ session=None,
+ max_retries=3,
+ timeout=60,
+ require_zip_signature=False,
+):
+ """Downloads a file from download_url and saves it atomically to output_path."""
+ logging.info("Downloading file from: %s", download_url)
+ dst_dir = os.path.dirname(os.path.abspath(output_path))
+ os.makedirs(dst_dir, exist_ok=True)
+
+ tmp_path = None
+ try:
+ if session is not None:
+ response = session.get(
+ download_url, headers=HTTP_HEADERS, timeout=timeout
+ )
+ if hasattr(response, "raise_for_status"):
+ response.raise_for_status()
+ content = response.content
+ else:
+ content = request_url(
+ url=download_url,
+ headers=HTTP_HEADERS,
+ output="bytes",
+ timeout=timeout,
+ retries=max_retries,
+ )
+ if not content:
+ logging.error("Empty response body received from %s", download_url)
+ raise RuntimeError(
+ f"Empty response body received from {download_url}"
+ )
+ if require_zip_signature and not content.startswith(b"PK\x03\x04"):
+ logging.error(
+ "Downloaded content from %s is not a valid ZIP/XLSX archive.",
+ download_url,
+ )
+ raise ValueError(
+ f"Downloaded content from {download_url} is not a valid "
+ "ZIP/XLSX archive."
+ )
+
+ with tempfile.NamedTemporaryFile(
+ "wb", dir=dst_dir, delete=False, suffix=".tmp"
+ ) as tmp:
+ tmp_path = tmp.name
+ tmp.write(content)
+
+ if not os.path.exists(tmp_path) or os.path.getsize(tmp_path) == 0:
+ logging.error(
+ "Temporary download file is missing or empty: %s", tmp_path
+ )
+ raise RuntimeError(
+ f"Temporary download file is missing or empty: {tmp_path}"
+ )
+
+ os.replace(tmp_path, output_path)
+ tmp_path = None
+ logging.info(
+ "Download completed successfully. Saved %d bytes to %s",
+ len(content),
+ output_path,
+ )
+ return content
+ finally:
+ if tmp_path and os.path.exists(tmp_path):
+ try:
+ os.unlink(tmp_path)
+ except OSError:
+ pass
+
+
+
+def extract_sheet_to_csv(
+ excel_source,
+ csv_output_path,
+ target_sheet_name="Underlying_Data",
+):
+ """Extracts the underlying data worksheet from Excel workbook to a raw CSV."""
+ if isinstance(excel_source, bytes):
+ wb = openpyxl.load_workbook(io.BytesIO(excel_source), data_only=True)
+ else:
+ wb = openpyxl.load_workbook(excel_source, data_only=True)
+
+ try:
+ sheet_names = wb.sheetnames
+ selected_sheet = None
+ if target_sheet_name in sheet_names:
+ selected_sheet = target_sheet_name
+ else:
+ for name in sheet_names:
+ if any(
+ k in name.lower()
+ for k in ["underlying", "poverty", "data", "ppc"]
+ ):
+ selected_sheet = name
+ break
+ if not selected_sheet:
+ selected_sheet = sheet_names[0]
+ logging.warning(
+ "Worksheet '%s' not found in %s; falling back to '%s'.",
+ target_sheet_name,
+ sheet_names,
+ selected_sheet,
+ )
+
+ logging.info(
+ "Extracting sheet '%s' from Excel workbook...", selected_sheet
+ )
+ ws = wb[selected_sheet]
+
+ dst_dir = os.path.dirname(os.path.abspath(csv_output_path))
+ os.makedirs(dst_dir, exist_ok=True)
+
+ row_count = 0
+ tmp_path = None
+ try:
+ with tempfile.NamedTemporaryFile(
+ "w",
+ dir=dst_dir,
+ delete=False,
+ suffix=".tmp",
+ encoding="utf-8",
+ newline="",
+ ) as tmp:
+ tmp_path = tmp.name
+ writer = csv.writer(tmp)
+ for row in ws.iter_rows(values_only=True):
+ if not any(row):
+ continue
+ writer.writerow(
+ [("" if c is None else str(c)) for c in row]
+ )
+ row_count += 1
+
+ os.replace(tmp_path, csv_output_path)
+ tmp_path = None
+ finally:
+ if tmp_path and os.path.exists(tmp_path):
+ try:
+ os.unlink(tmp_path)
+ except OSError:
+ pass
+
+ logging.info(
+ "Extracted %d rows from sheet '%s' to %s",
+ row_count,
+ selected_sheet,
+ csv_output_path,
+ )
+ return csv_output_path
+ finally:
+ wb.close()
+
+
+def copy_file_atomically(src_path, dst_path):
+ """Copies src_path to dst_path atomically."""
+ dst_dir = os.path.dirname(os.path.abspath(dst_path))
+ os.makedirs(dst_dir, exist_ok=True)
+ tmp_path = None
+ try:
+ with tempfile.NamedTemporaryFile(
+ "wb", dir=dst_dir, delete=False, suffix=".tmp"
+ ) as tmp:
+ tmp_path = tmp.name
+ with open(src_path, "rb") as fsrc:
+ shutil.copyfileobj(fsrc, tmp)
+ os.replace(tmp_path, dst_path)
+ tmp_path = None
+ finally:
+ if tmp_path and os.path.exists(tmp_path):
+ try:
+ os.unlink(tmp_path)
+ except OSError:
+ pass
+ logging.info("Copied %s to %s", src_path, dst_path)
+
+
+def download_poverty_dataset(
+ source_url=EDA_PPC_XLSX_URL,
+ mirror_url=EDA_PPC_MIRROR_URL,
+ input_file=None,
+ output_xlsx_path=DEFAULT_OUTPUT_XLSX,
+ output_csv_path=DEFAULT_INPUT_CSV,
+ raw_csv_path=DEFAULT_RAW_CSV_FILE,
+ max_retries=3,
+ timeout=60,
+):
+ """Main workflow to download official EDA PPC workbook or ingest input file."""
+ # Case 1: Local input file specified
+ if input_file:
+ if not os.path.exists(input_file) or os.path.getsize(input_file) == 0:
+ raise FileNotFoundError(
+ f"Input file not found or empty: {input_file}"
+ )
+ logging.info("Using provided local input file: %s", input_file)
+ if input_file.lower().endswith((".xlsx", ".xls")):
+ copy_file_atomically(input_file, output_xlsx_path)
+ extract_sheet_to_csv(input_file, output_csv_path)
+ else:
+ copy_file_atomically(input_file, output_csv_path)
+ if raw_csv_path:
+ copy_file_atomically(output_csv_path, raw_csv_path)
+ return output_csv_path
+
+ # Clean up existing target files before a fresh download to avoid stale files
+ for path_to_clean in [output_xlsx_path, output_csv_path, raw_csv_path]:
+ if path_to_clean and os.path.exists(path_to_clean):
+ try:
+ os.remove(path_to_clean)
+ logging.info(
+ "Cleaned up existing target file before fresh download: %s",
+ path_to_clean,
+ )
+ except OSError as e:
+ logging.warning(
+ "Could not remove existing file %s: %s", path_to_clean, e
+ )
+
+ # Case 2: Download from web with primary and mirror fallback
+ content = None
+ urls_to_try = []
+ if source_url:
+ urls_to_try.append(source_url)
+ if mirror_url and mirror_url != source_url:
+ urls_to_try.append(mirror_url)
+
+ last_err = None
+ for url in urls_to_try:
+ try:
+ content = download_file(
+ download_url=url,
+ output_path=output_xlsx_path,
+ max_retries=max_retries,
+ timeout=timeout,
+ require_zip_signature=True,
+ )
+ # Extract Underlying_Data sheet to output_csv_path
+ extract_sheet_to_csv(content, output_csv_path)
+ logging.info(
+ "Successfully downloaded workbook and extracted sheet "
+ "from %s",
+ url,
+ )
+ break
+ except Exception as e:
+ last_err = e
+ logging.warning(
+ "Failed to download or process from %s: %s", url, e
+ )
+ if os.path.exists(output_xlsx_path):
+ try:
+ os.remove(output_xlsx_path)
+ except OSError:
+ pass
+ if os.path.exists(output_csv_path):
+ try:
+ os.remove(output_csv_path)
+ except OSError:
+ pass
+ content = None
+
+ if not content:
+ logging.error(
+ "Failed to acquire dataset from all URLs: %s. Last error: %s",
+ urls_to_try,
+ last_err,
+ )
+ raise RuntimeError(
+ f"Failed to acquire dataset from all URLs: {urls_to_try}. "
+ f"Last error: {last_err}"
+ ) from last_err
+
+ if raw_csv_path:
+ copy_file_atomically(output_csv_path, raw_csv_path)
+
+ return output_csv_path
+
+
+def main(argv):
+ """Main entrypoint for downloading the poverty dataset."""
+ del argv # Unused
+ try:
+ download_poverty_dataset(
+ source_url=FLAGS.source_url,
+ mirror_url=FLAGS.mirror_url,
+ input_file=FLAGS.input_file,
+ output_xlsx_path=FLAGS.output_xlsx_path,
+ output_csv_path=FLAGS.output_csv_path,
+ raw_csv_path=FLAGS.raw_csv_path,
+ max_retries=FLAGS.max_retries,
+ timeout=FLAGS.timeout,
+ )
+ except Exception as e:
+ logging.fatal(
+ "Failed to download Commerce EDA Poverty dataset: %s",
+ e,
+ exc_info=True,
+ )
+
+
+if __name__ == "__main__":
+ app.run(main)
diff --git a/statvar_imports/commerce_eda_poverty/download_poverty_test.py b/statvar_imports/commerce_eda_poverty/download_poverty_test.py
new file mode 100644
index 0000000000..977474cc14
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/download_poverty_test.py
@@ -0,0 +1,428 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for Commerce EDA Persistent Poverty Counties download script."""
+
+import io
+import os
+import sys
+import tempfile
+import unittest
+from unittest import mock
+from urllib import parse
+
+import openpyxl
+import pandas as pd
+
+MODULE_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.abspath(os.path.join(MODULE_DIR, "..", ".."))
+sys.path.insert(0, PROJECT_ROOT)
+
+from statvar_imports.commerce_eda_poverty.download_poverty import (
+ EDA_PPC_MIRROR_URL,
+ EDA_PPC_XLSX_URL,
+ HTTP_HEADERS,
+ download_file,
+ download_poverty_dataset,
+ extract_sheet_to_csv,
+)
+
+
+def _create_mock_eda_workbook(filepath=None):
+ """Creates a mock Excel workbook containing the Underlying_Data worksheet."""
+ wb = openpyxl.Workbook()
+ try:
+ # Sheet 1: Readme
+ ws_readme = wb.active
+ ws_readme.title = "EDA Read Me"
+ ws_readme.append(["FY2023 PERSISTENT POVERTY COUNTIES (PPCs)", ""])
+
+ # Sheet 2: Underlying_Data
+ ws_data = wb.create_sheet(title="Underlying_Data")
+ ws_data.append([
+ "Table. FY2023 Persistent Poverty County Status - as of Data Year 2021",
+ "", "", "", "", "", "", ""
+ ])
+ ws_data.append([
+ "Identifing Information", "", "Census Bureau Data", "", "", "",
+ "FY23 Persistent Poverty", "Census GEO PPC Code"
+ ])
+ ws_data.append([
+ "Name",
+ "GEOID",
+ "1990 Decennial Census, % in Poverty",
+ "2000 Decennial Census, % in Poverty",
+ "Most Recent Estimate, % in Poverty* ",
+ "Data Source―Most Recent Estimate",
+ "",
+ "",
+ ])
+ ws_data.append(
+ [
+ "Autauga County, AL",
+ "01001",
+ 15.7,
+ 10.9,
+ 13.3,
+ "SAIPE, 2021",
+ "No",
+ 1,
+ ]
+ )
+ ws_data.append(
+ [
+ "Barbour County, AL",
+ "01005",
+ 25.2,
+ 26.8,
+ 29.0,
+ "SAIPE, 2021",
+ "Yes",
+ 2,
+ ]
+ )
+ ws_data.append([
+ "Eastern District, AS",
+ "60010",
+ 56.0,
+ 58.6,
+ 52.2,
+ "Decennial Census, 2020",
+ "Yes",
+ 2,
+ ])
+
+ if filepath:
+ wb.save(filepath)
+ return filepath
+
+ bio = io.BytesIO()
+ wb.save(bio)
+ return bio.getvalue()
+ finally:
+ wb.close()
+
+
+class TestDownloadPoverty(unittest.TestCase):
+
+ def test_download_file_success(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ out_file = os.path.join(tmpdir, "test.xlsx")
+ test_content = b"PK\x03\x04test_content"
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ return_value=test_content,
+ ) as mock_request_url:
+ content = download_file(
+ "https://example.gov/EDA_FY23_PPCs.xlsx",
+ out_file,
+ max_retries=1,
+ )
+
+ self.assertEqual(content, test_content)
+ mock_request_url.assert_called_once_with(
+ url="https://example.gov/EDA_FY23_PPCs.xlsx",
+ headers=HTTP_HEADERS,
+ output="bytes",
+ timeout=60,
+ retries=1,
+ )
+ self.assertTrue(os.path.exists(out_file))
+ with open(out_file, "rb") as f:
+ self.assertEqual(f.read(), test_content)
+
+ def test_download_file_passes_headers_and_timeout(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ out_file = os.path.join(tmpdir, "params.xlsx")
+ test_content = b"PK\x03\x04param_test"
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ return_value=test_content,
+ ) as mock_request_url:
+ content = download_file(
+ "https://example.gov/EDA_FY23_PPCs.xlsx",
+ out_file,
+ max_retries=5,
+ timeout=45,
+ require_zip_signature=True,
+ )
+
+ self.assertEqual(content, test_content)
+ mock_request_url.assert_called_once_with(
+ url="https://example.gov/EDA_FY23_PPCs.xlsx",
+ headers=HTTP_HEADERS,
+ output="bytes",
+ timeout=45,
+ retries=5,
+ )
+
+ def test_download_file_fails(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ out_file = os.path.join(tmpdir, "fail.xlsx")
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ return_value=None,
+ ) as mock_request_url:
+ with self.assertRaises(RuntimeError):
+ download_file(
+ "https://example.gov/EDA_FY23_PPCs.xlsx",
+ out_file,
+ max_retries=2,
+ )
+
+ mock_request_url.assert_called_once()
+ self.assertFalse(os.path.exists(out_file))
+
+ def test_download_file_empty_body_raises(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ out_file = os.path.join(tmpdir, "empty.xlsx")
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ return_value=b"",
+ ) as mock_request_url:
+ with self.assertRaises(RuntimeError):
+ download_file(
+ "https://example.gov/EDA_FY23_PPCs.xlsx",
+ out_file,
+ max_retries=1,
+ )
+
+ mock_request_url.assert_called_once()
+ self.assertFalse(os.path.exists(out_file))
+
+ def test_download_file_session_without_mount_supported(self):
+ class DuckSession:
+
+ def __init__(self):
+ self.call_count = 0
+
+ def get(self, url, headers=None, timeout=None, **kwargs):
+ self.call_count += 1
+ resp = mock.MagicMock()
+ resp.__enter__.return_value = resp
+ resp.status_code = 200
+ resp.content = b"duck_data"
+ return resp
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ out_file = os.path.join(tmpdir, "duck.xlsx")
+ duck_session = DuckSession()
+ content = download_file(
+ "https://example.gov/duck.xlsx",
+ out_file,
+ session=duck_session,
+ )
+
+ self.assertEqual(content, b"duck_data")
+ self.assertEqual(duck_session.call_count, 1)
+ self.assertTrue(os.path.exists(out_file))
+ with open(out_file, "rb") as f:
+ self.assertEqual(f.read(), b"duck_data")
+
+ def test_extract_sheet_to_csv_from_bytes(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ csv_path = os.path.join(tmpdir, "extracted.csv")
+ excel_bytes = _create_mock_eda_workbook()
+
+ extract_sheet_to_csv(
+ excel_bytes, csv_path, target_sheet_name="Underlying_Data"
+ )
+ self.assertTrue(os.path.exists(csv_path))
+
+ df = pd.read_csv(csv_path, skiprows=2, dtype=str)
+ self.assertEqual(len(df), 3)
+ self.assertIn("GEOID", df.columns)
+ self.assertEqual(list(df["GEOID"]), ["01001", "01005", "60010"])
+
+ def test_extract_sheet_to_csv_from_file_path(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ xlsx_path = os.path.join(tmpdir, "EDA_FY23_PPCs.xlsx")
+ csv_path = os.path.join(tmpdir, "Poverty.csv")
+ _create_mock_eda_workbook(xlsx_path)
+
+ extract_sheet_to_csv(xlsx_path, csv_path)
+ self.assertTrue(os.path.exists(csv_path))
+
+ df = pd.read_csv(csv_path, skiprows=2, dtype=str)
+ self.assertEqual(len(df), 3)
+ self.assertIn("GEOID", df.columns)
+
+ def test_download_poverty_dataset_with_input_file_csv(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ src_csv = os.path.join(tmpdir, "source_input.csv")
+ dst_xlsx = os.path.join(tmpdir, "out.xlsx")
+ dst_csv = os.path.join(tmpdir, "out.csv")
+ raw_csv = os.path.join(tmpdir, "raw.csv")
+
+ with open(src_csv, "w", encoding="utf-8") as f:
+ f.write("Line 1\nLine 2\nName,GEOID\nCounty A,01001\n")
+
+ res = download_poverty_dataset(
+ input_file=src_csv,
+ output_xlsx_path=dst_xlsx,
+ output_csv_path=dst_csv,
+ raw_csv_path=raw_csv,
+ )
+
+ self.assertEqual(res, dst_csv)
+ self.assertTrue(os.path.exists(dst_csv))
+ self.assertTrue(os.path.exists(raw_csv))
+ with open(dst_csv, "r", encoding="utf-8") as f:
+ content = f.read()
+ self.assertIn("County A,01001", content)
+
+ def test_download_poverty_dataset_with_input_file_xlsx(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ src_xlsx = os.path.join(tmpdir, "source_input.xlsx")
+ dst_xlsx = os.path.join(tmpdir, "EDA_FY23_PPCs.xlsx")
+ dst_csv = os.path.join(tmpdir, "Poverty.csv")
+ raw_csv = os.path.join(tmpdir, "Poverty_original.csv")
+
+ _create_mock_eda_workbook(src_xlsx)
+
+ res = download_poverty_dataset(
+ input_file=src_xlsx,
+ output_xlsx_path=dst_xlsx,
+ output_csv_path=dst_csv,
+ raw_csv_path=raw_csv,
+ )
+
+ self.assertEqual(res, dst_csv)
+ self.assertTrue(os.path.exists(dst_xlsx))
+ self.assertTrue(os.path.exists(dst_csv))
+ self.assertTrue(os.path.exists(raw_csv))
+
+ df = pd.read_csv(dst_csv, skiprows=2, dtype=str)
+ self.assertEqual(len(df), 3)
+
+ def test_download_poverty_dataset_primary_fail_mirror_succeed(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ dst_xlsx = os.path.join(tmpdir, "EDA_FY23_PPCs.xlsx")
+ dst_csv = os.path.join(tmpdir, "Poverty.csv")
+ raw_csv = os.path.join(tmpdir, "Poverty_original.csv")
+
+ excel_bytes = _create_mock_eda_workbook()
+
+ def mock_request(url, **kwargs):
+ if parse.urlparse(url).netloc == "www.eda.gov":
+ raise RuntimeError("HTTP 403")
+ return excel_bytes
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ side_effect=mock_request,
+ ):
+ res = download_poverty_dataset(
+ source_url=EDA_PPC_XLSX_URL,
+ mirror_url=EDA_PPC_MIRROR_URL,
+ output_xlsx_path=dst_xlsx,
+ output_csv_path=dst_csv,
+ raw_csv_path=raw_csv,
+ max_retries=1,
+ )
+
+ self.assertEqual(res, dst_csv)
+ self.assertTrue(os.path.exists(dst_xlsx))
+ self.assertTrue(os.path.exists(dst_csv))
+ df = pd.read_csv(dst_csv, skiprows=2, dtype=str)
+ self.assertEqual(len(df), 3)
+
+ def test_download_poverty_dataset_missing_input_file_raises(self):
+ with self.assertRaises(FileNotFoundError):
+ download_poverty_dataset(input_file="/nonexistent/path/Poverty.csv")
+
+ def test_download_poverty_dataset_all_urls_fail_raises_and_cleans_stale_files(
+ self,
+ ):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ dst_xlsx = os.path.join(tmpdir, "EDA_FY23_PPCs.xlsx")
+ dst_csv = os.path.join(tmpdir, "Poverty.csv")
+ dst_raw_csv = os.path.join(tmpdir, "Poverty_original.csv")
+ for p in [dst_xlsx, dst_csv, dst_raw_csv]:
+ with open(p, "w", encoding="utf-8") as f:
+ f.write("stale_data")
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ side_effect=RuntimeError("HTTP 503"),
+ ):
+ with self.assertRaises(RuntimeError) as ctx:
+ download_poverty_dataset(
+ source_url="https://example.gov/p1.xlsx",
+ mirror_url="https://example.gov/m1.xlsx",
+ output_xlsx_path=dst_xlsx,
+ output_csv_path=dst_csv,
+ raw_csv_path=dst_raw_csv,
+ max_retries=1,
+ )
+
+ self.assertIn(
+ "Failed to acquire dataset from all URLs", str(ctx.exception)
+ )
+ self.assertFalse(os.path.exists(dst_xlsx))
+ self.assertFalse(os.path.exists(dst_csv))
+ self.assertFalse(os.path.exists(dst_raw_csv))
+
+ def test_download_poverty_dataset_primary_html_challenge_mirror_succeed(
+ self,
+ ):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ dst_xlsx = os.path.join(tmpdir, "EDA_FY23_PPCs.xlsx")
+ dst_csv = os.path.join(tmpdir, "Poverty.csv")
+ raw_csv = os.path.join(tmpdir, "Poverty_original.csv")
+
+ excel_bytes = _create_mock_eda_workbook()
+
+ def mock_request(url, **kwargs):
+ if parse.urlparse(url).netloc == "www.eda.gov":
+ return (
+ b"Just a moment..."
+ b""
+ )
+ return excel_bytes
+
+ with mock.patch(
+ "statvar_imports.commerce_eda_poverty.download_poverty"
+ ".request_url",
+ side_effect=mock_request,
+ ):
+ res = download_poverty_dataset(
+ source_url=EDA_PPC_XLSX_URL,
+ mirror_url=EDA_PPC_MIRROR_URL,
+ output_xlsx_path=dst_xlsx,
+ output_csv_path=dst_csv,
+ raw_csv_path=raw_csv,
+ max_retries=1,
+ )
+
+ self.assertEqual(res, dst_csv)
+ self.assertTrue(os.path.exists(dst_xlsx))
+ self.assertTrue(os.path.exists(dst_csv))
+ df = pd.read_csv(dst_csv, skiprows=2, dtype=str)
+ self.assertEqual(len(df), 3)
+
+
+if __name__ == "__main__":
+ unittest.main()
+
diff --git a/statvar_imports/commerce_eda_poverty/manifest.json b/statvar_imports/commerce_eda_poverty/manifest.json
new file mode 100644
index 0000000000..333f9c83ae
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/manifest.json
@@ -0,0 +1,31 @@
+{
+ "import_specifications": [
+ {
+ "import_name": "Commerce_EDA_Poverty",
+ "curator_emails": [
+ "support@datacommons.org"
+ ],
+ "provenance_url": "https://www.eda.gov/performance/resources/persistent-poverty-counties",
+ "provenance_description": "U.S. Economic Development Administration (EDA) - Persistent Poverty Counties.",
+ "cron_schedule": "30 05 1 1 *",
+ "scripts": [
+ "download_poverty.py",
+ "process_poverty.py",
+ "../../tools/statvar_importer/stat_var_processor.py --input_data=output/Poverty_cleaned.csv --pv_map=poverty_pvmap.csv --config_file=poverty_metadata.csv --output_path=output/Poverty_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_counters=counters/Poverty_counters.csv"
+ ],
+ "import_inputs": [
+ {
+ "template_mcf": "output/Poverty_output.tmcf",
+ "cleaned_csv": "output/Poverty_output.csv",
+ "node_mcf": "output/*.mcf"
+ }
+ ],
+ "validation_config_file": "validation_config.json",
+ "source_files": [
+ "input_files/*",
+ "counters/*",
+ "validation_config.json"
+ ]
+ }
+ ]
+}
diff --git a/statvar_imports/commerce_eda_poverty/poverty_metadata.csv b/statvar_imports/commerce_eda_poverty/poverty_metadata.csv
new file mode 100644
index 0000000000..d6696c12df
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/poverty_metadata.csv
@@ -0,0 +1,13 @@
+parameter,value
+url,https://www.eda.gov/performance/resources/persistent-poverty-counties
+description,Persistent Poverty County Status (sourced from U.S. Economic Development Administration)
+#place_type,County
+#places_within,country/USA
+#start_date,1990
+#end_date,2021
+release_frequency,P1Y
+process,
+comments,
+output_columns,"observationAbout,observationDate,variableMeasured,value,unit,scalingFactor,observationPeriod,measurementMethod"
+header_rows,1
+output_only_new_statvars,True
diff --git a/statvar_imports/commerce_eda_poverty/poverty_pvmap.csv b/statvar_imports/commerce_eda_poverty/poverty_pvmap.csv
new file mode 100644
index 0000000000..ef3f68fdc2
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/poverty_pvmap.csv
@@ -0,0 +1,12 @@
+key,prop,value,p1,v1,p2,v2,p3,v3,p4,v4
+GEOID,observationAbout,geoId/{Data},,,,,,,,
+year,observationDate,{Data},measurementMethod,dcs:EDA_Estimate,,,,,,
+year:1990,observationDate,1990,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2000,observationDate,2000,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2010,observationDate,2010,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2020,observationDate,2020,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2030,observationDate,2030,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2040,observationDate,2040,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2050,observationDate,2050,measurementMethod,dcs:USDecennialCensus,,,,,,
+year:2021,observationDate,2021,measurementMethod,dcs:EDA_Estimate,,,,,,
+poverty_rate,unit,Percent,scalingFactor,100,observationPeriod,P1Y,value,{Number},variableMeasured,dcs:Count_Person_BelowPovertyLevelInThePast12Months_AsFractionOf_Count_Person
diff --git a/statvar_imports/commerce_eda_poverty/process_poverty.py b/statvar_imports/commerce_eda_poverty/process_poverty.py
new file mode 100644
index 0000000000..1b9e9c8532
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/process_poverty.py
@@ -0,0 +1,484 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Preprocessing script for Commerce EDA Persistent Poverty Counties dataset.
+
+This script ingests the raw Persistent Poverty Counties dataset (downloaded
+from U.S. Economic Development Administration (EDA) / Department of Commerce),
+standardizes geographic identifiers (5-digit county FIPS codes across 50 US
+states, DC, and Puerto Rico, and 5-digit island territory codes), validates
+poverty percentage rates across 1990, 2000, 2020, and 2021, and generates the
+normalized cleaned CSV for stat_var_processor.py.
+
+Why Preprocessing is Required (Why PV Map Alone is Insufficient):
+1. Format Reshaping (Wide-to-Long): The raw EDA dataset is wide (one row per county
+ with separate columns for 1990, 2000, and recent estimate). PV mapping requires
+ a normalized long format where each row represents a single observation (GEOID,
+ date, poverty_rate).
+2. Dynamic & Heterogeneous Observation Dates: The "Most Recent Estimate" is not
+ from a single survey year:
+ - For the 50 US states and DC (3,143 counties), the benchmark is the 2021
+ Small Area Income and Poverty Estimates (SAIPE, "SAIPE, 2021" -> 2021).
+ - For Puerto Rico (78 municipios; 3,143 + 78 = 3,221), SAIPE and the Island
+ Areas Census are unavailable, so EDA uses the 2017-2021 American Community
+ Survey 5-year estimates ("ACS 5-YEAR, 2017-2021" -> end year 2021).
+ - For the 4 Island Territories (11 county equivalents across American Samoa,
+ Guam, Northern Mariana Islands, and US Virgin Islands; 3,221 + 11 = 3,232),
+ estimates come from the 2020 Island Areas Decennial Census ("Decennial
+ Census, 2020" -> 2020).
+ Preprocessing parses the survey year dynamically from the 'Data Source' note
+ (extracting the final 4-digit year so multi-year ranges like "2017-2021" map
+ to "2021", or falling back to territory FIPS prefix) to assign 2020 vs 2021
+ per record. Note that poverty_pvmap.csv intentionally sets a uniform
+ observationPeriod: P1Y across all 3,232 places so that all jurisdictions in
+ the EDA Persistent Poverty benchmark series share a single cohesive facet.
+ PV Map CSV configurations cannot conditionally extract and branch dates based
+ on FIPS prefixes or free-text regex matching.
+3. Metadata and Footnote Cleaning: Upstream Excel/CSV files contain non-tabular
+ header titles, subtitle rows, and footnote rows at the bottom that must be skipped.
+4. FIPS Code Standardization & Filtering: GEOIDs in raw files may lack leading zeros
+ (e.g., 1001 -> 01001) or represent state-level rollups (e.g. 01000) that must be
+ filtered out from county-level statistical variables.
+"""
+
+import csv
+import datetime
+import os
+import re
+import tempfile
+
+from absl import app, flags, logging
+import openpyxl
+import pandas as pd
+
+MODULE_DIR = os.path.dirname(os.path.abspath(__file__))
+
+DEFAULT_SOURCE_CSV = os.path.join(MODULE_DIR, "input_files", "Poverty.csv")
+DEFAULT_SOURCE_XLSX = os.path.join(MODULE_DIR, "input_files", "EDA_FY23_PPCs.xlsx")
+DEFAULT_RAW_CSV = os.path.join(MODULE_DIR, "output", "Poverty_original.csv")
+CLEANED_CSV = os.path.join(MODULE_DIR, "output", "Poverty_cleaned.csv")
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string(
+ "source_path",
+ None,
+ "Path to source file (.csv or .xlsx). If not specified, automatically "
+ "resolves from input_files/Poverty.csv, input_files/EDA_FY23_PPCs.xlsx, "
+ "or output/Poverty_original.csv.",
+)
+flags.DEFINE_string(
+ "cleaned_csv_path", CLEANED_CSV, "Path to save cleaned output CSV."
+)
+flags.DEFINE_integer(
+ "min_county_count", 3000, "Minimum number of valid counties expected."
+)
+flags.DEFINE_integer(
+ "min_survey_year", 2020, "Minimum valid survey year for most recent estimate."
+)
+flags.DEFINE_integer(
+ "max_survey_year",
+ None,
+ "Maximum valid survey year (defaults to current year).",
+)
+
+# Valid 2-digit US State and Territory FIPS codes
+VALID_STATE_FIPS = {
+ # 50 States + DC
+ "01", "02", "04", "05", "06", "08", "09", "10", "11", "12", "13", "15",
+ "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27",
+ "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39",
+ "40", "41", "42", "44", "45", "46", "47", "48", "49", "50", "51", "53",
+ "54", "55", "56",
+ # Territories: AS, GU, MP, PR, VI
+ "60", "66", "69", "72", "78",
+}
+
+# Island territories where most recent estimate is from 2020 Decennial Census
+ISLAND_TERRITORY_FIPS = {"60", "66", "69", "78"}
+
+COLUMN_RENAME_MAP = {
+ "GEOID": "GEOID",
+ "1990 Decennial Census, % in Poverty": "poverty_rate_1990",
+ "2000 Decennial Census, % in Poverty": "poverty_rate_2000",
+ "Most Recent Estimate, % in Poverty*": "poverty_rate_recent",
+ "Most Recent Estimate, % in Poverty": "poverty_rate_recent",
+}
+
+
+def clean_geoid(val):
+ """Standardizes GEOIDs to 5 digits and validates against US state FIPS.
+
+ Rejects state summary entries of the form XX000.
+ """
+ if pd.isna(val):
+ return None
+ s = str(val).strip()
+ match = re.match(r"^(\d+)(?:\.0+)?$", s)
+ if not match:
+ return None
+ digits = match.group(1)
+ if len(digits) == 4:
+ digits = digits.zfill(5)
+ if len(digits) == 5 and digits[:2] in VALID_STATE_FIPS and digits[2:] != "000":
+ return digits
+ return None
+
+
+def _extract_dataframe_from_excel(excel_path):
+ """Extracts Underlying_Data sheet from an Excel workbook into a DataFrame."""
+ wb = openpyxl.load_workbook(excel_path, data_only=True)
+ try:
+ sheet_names = wb.sheetnames
+ target_sheet = None
+ if "Underlying_Data" in sheet_names:
+ target_sheet = "Underlying_Data"
+ else:
+ for name in sheet_names:
+ if any(
+ k in name.lower()
+ for k in ["underlying", "poverty", "data", "ppc"]
+ ):
+ target_sheet = name
+ break
+ if not target_sheet:
+ target_sheet = sheet_names[0]
+ logging.warning(
+ "Worksheet 'Underlying_Data' not found in %s; falling back "
+ "to '%s'.",
+ sheet_names,
+ target_sheet,
+ )
+ ws = wb[target_sheet]
+ rows = list(ws.iter_rows(values_only=True))
+ if not rows:
+ raise ValueError(f"Excel sheet '{target_sheet}' is empty.")
+
+ # Find row with GEOID header dynamically across all rows
+ header_idx = None
+ for idx, r in enumerate(rows):
+ row_str = [str(c).strip() for c in r if c is not None]
+ if any(c.upper() == "GEOID" for c in row_str):
+ header_idx = idx
+ break
+
+ if header_idx is None:
+ raise ValueError(
+ f"Could not find header row containing 'GEOID' in sheet "
+ f"'{target_sheet}' of {excel_path}"
+ )
+
+ headers = [
+ ("" if c is None else str(c).strip()) for c in rows[header_idx]
+ ]
+ data_rows = []
+ for r in rows[header_idx + 1:]:
+ if not any(r):
+ continue
+ data_rows.append(
+ [
+ ("" if c is None else str(c).strip())
+ for c in r[: len(headers)]
+ ]
+ )
+
+ return pd.DataFrame(data_rows, columns=headers)
+ finally:
+ wb.close()
+
+
+def resolve_source_file_path(requested_path=None):
+ """Finds the raw source file, raising FileNotFoundError if missing."""
+ if requested_path:
+ if os.path.exists(requested_path) and os.path.getsize(requested_path) > 0:
+ return requested_path
+ raise FileNotFoundError(
+ f"Specified source file not found or empty: {requested_path}"
+ )
+
+ candidates = [
+ DEFAULT_SOURCE_CSV,
+ DEFAULT_SOURCE_XLSX,
+ DEFAULT_RAW_CSV,
+ ]
+ for candidate in candidates:
+ if os.path.exists(candidate) and os.path.getsize(candidate) > 0:
+ return candidate
+
+ raise FileNotFoundError(
+ "No downloaded source file found. Please run download_poverty.py first "
+ f"to fetch the dataset, or specify --source_path. Checked: {candidates}"
+ )
+
+
+def preprocess_poverty(
+ src_path=DEFAULT_SOURCE_CSV,
+ dst_path=CLEANED_CSV,
+ min_county_count=3000,
+ min_survey_year=2020,
+ max_survey_year=None,
+):
+ """Preprocesses the raw Poverty dataset into normalized cleaned CSV."""
+ logging.info("Preprocessing source Poverty dataset from %s...", src_path)
+ if max_survey_year is None:
+ max_survey_year = datetime.date.today().year
+ if not os.path.exists(src_path) or os.path.getsize(src_path) == 0:
+ raise ValueError(f"Source file does not exist or is empty: {src_path}")
+
+ if src_path.lower().endswith((".xlsx", ".xls")):
+ df = _extract_dataframe_from_excel(src_path)
+ else:
+ # Parse CSV rows via csv.reader so multi-line preamble cells never
+ # desynchronize logical record indices from physical line numbers.
+ with open(
+ src_path, "r", encoding="utf-8", errors="ignore", newline=""
+ ) as f:
+ rows = list(csv.reader(f))
+ if not rows:
+ raise ValueError(f"CSV file is empty: {src_path}")
+
+ header_idx = None
+ for idx, row in enumerate(rows):
+ if any(cell.strip().upper() == "GEOID" for cell in row):
+ header_idx = idx
+ break
+ if header_idx is None:
+ raise ValueError(
+ "Could not find header row containing 'GEOID' in CSV file: "
+ f"{src_path}"
+ )
+
+ headers = [cell.strip() for cell in rows[header_idx]]
+ data_rows = []
+ for r in rows[header_idx + 1 :]:
+ if not any(cell.strip() for cell in r):
+ continue
+ padded = [cell.strip() for cell in r[: len(headers)]]
+ if len(padded) < len(headers):
+ padded.extend([""] * (len(headers) - len(padded)))
+ data_rows.append(padded)
+ df = pd.DataFrame(data_rows, columns=headers)
+
+ if df.empty:
+ raise ValueError(f"File not read properly into dataframe: {src_path}")
+
+ # Strip column headers to avoid fragile whitespace issues
+ df.columns = df.columns.str.strip()
+
+ # Check if at least GEOID and the poverty columns are found
+ if "GEOID" not in df.columns:
+ raise ValueError("Missing required column 'GEOID' in source dataset")
+
+ # Rename columns to standard names (including future Decennial cols)
+ rename_dict = {}
+ for col in df.columns:
+ clean_col = col.rstrip("*").strip()
+ matched = False
+ for k, v in COLUMN_RENAME_MAP.items():
+ if col == k or clean_col == k.rstrip("*").strip():
+ rename_dict[col] = v
+ matched = True
+ break
+ if not matched:
+ hist_match = re.match(
+ r"^(\d{4})\b.*%\s*in\s*Poverty", clean_col, flags=re.IGNORECASE
+ )
+ if hist_match:
+ rename_dict[col] = f"poverty_rate_{hist_match.group(1)}"
+
+ # Guard against silent year corruption if Data Source column is present
+ data_source_cols = [c for c in df.columns if "data source" in c.lower()]
+ df = df.rename(columns=rename_dict)
+
+ required_cols = [
+ "GEOID",
+ "poverty_rate_1990",
+ "poverty_rate_2000",
+ "poverty_rate_recent",
+ ]
+ missing = [c for c in required_cols if c not in df.columns]
+ if missing:
+ raise ValueError(
+ f"Missing required columns in source dataset: {missing}"
+ )
+
+ # Standardize and validate GEOIDs
+ df["GEOID"] = df["GEOID"].apply(clean_geoid)
+ df = df.dropna(subset=["GEOID"])
+
+ is_territory = df["GEOID"].str[:2].isin(ISLAND_TERRITORY_FIPS)
+ default_years = pd.Series("2021", index=df.index).where(
+ ~is_territory, "2020"
+ )
+
+ if data_source_cols:
+ ds_col = data_source_cols[0]
+ ds_vals = df[ds_col].fillna("").astype(str).str.strip()
+ numeric_recent = pd.to_numeric(
+ df["poverty_rate_recent"].astype(str).str.strip(), errors="coerce"
+ )
+ recent_rate_valid = (
+ numeric_recent.notna()
+ & (numeric_recent >= 0.0)
+ & (numeric_recent <= 100.0)
+ )
+ has_ds = (
+ ~ds_vals.str.lower().isin({"", "nan", "na", "n/a", "none"})
+ & recent_rate_valid
+ )
+ extracted_years = ds_vals.str.findall(r"\b(?:19|20)\d{2}\b").str[-1]
+ unrecognized_mask = has_ds & extracted_years.isna()
+ if unrecognized_mask.any():
+ bad_idx = unrecognized_mask.idxmax()
+ raise ValueError(
+ f"Unrecognized survey year in {ds_col} "
+ f"('{ds_vals.loc[bad_idx]}') for GEOID "
+ f"{df.loc[bad_idx, 'GEOID']}"
+ )
+ yr_numeric = pd.to_numeric(extracted_years, errors="coerce")
+ out_of_range_mask = has_ds & (
+ (yr_numeric < min_survey_year) | (yr_numeric > max_survey_year)
+ )
+ if out_of_range_mask.any():
+ bad_idx = out_of_range_mask.idxmax()
+ bad_yr = int(yr_numeric.loc[bad_idx])
+ raise ValueError(
+ f"Unexpected survey year {bad_yr} in {ds_col} for GEOID "
+ f"{df.loc[bad_idx, 'GEOID']} (expected between "
+ f"{min_survey_year} and {max_survey_year})"
+ )
+ recent_years = default_years.where(~has_ds, extracted_years)
+ else:
+ recent_years = default_years
+
+ # Identify all historical year columns dynamically
+ historical_year_cols = sorted(
+ [
+ (re.match(r"^poverty_rate_(\d{4})$", c).group(1), c)
+ for c in df.columns
+ if re.match(r"^poverty_rate_(\d{4})$", c)
+ ],
+ key=lambda x: int(x[0]),
+ )
+
+ # Coerce and validate poverty values within [0.0, 100.0]
+ raw_poverty_cols = [c for _, c in historical_year_cols] + [
+ "poverty_rate_recent"
+ ]
+ for col in raw_poverty_cols:
+ df[col] = pd.to_numeric(df[col].astype(str).str.strip(), errors="coerce")
+ invalid_mask = df[col].notna() & ((df[col] < 0.0) | (df[col] > 100.0))
+ if invalid_mask.any():
+ logging.warning(
+ "Found %d out-of-bounds values in %s; setting to NaN",
+ invalid_mask.sum(),
+ col,
+ )
+ df.loc[invalid_mask, col] = None
+
+ # Keep counties that have at least one valid poverty rate observation
+ df = df.dropna(subset=raw_poverty_cols, how="all")
+
+ # Verify sanity threshold on valid county count
+ if len(df) < min_county_count:
+ raise ValueError(
+ f"Sanity check failed: Expected at least {min_county_count} "
+ f"counties, but found {len(df)}."
+ )
+
+ # Reshape wide-to-long using vectorized pd.melt() and pd.concat()
+ df = df.copy()
+ df["_row_order"] = range(len(df))
+ hist_rename = {col: yr for yr, col in historical_year_cols}
+ hist_df = df[["_row_order", "GEOID"] + list(hist_rename.keys())].rename(
+ columns=hist_rename
+ )
+ melted_hist = pd.melt(
+ hist_df,
+ id_vars=["_row_order", "GEOID"],
+ value_vars=list(hist_rename.values()),
+ var_name="year",
+ value_name="poverty_rate",
+ )
+ recent_df = pd.DataFrame(
+ {
+ "_row_order": df["_row_order"],
+ "GEOID": df["GEOID"],
+ "year": recent_years.loc[df.index].astype(str),
+ "poverty_rate": df["poverty_rate_recent"],
+ }
+ )
+ combined = pd.concat([melted_hist, recent_df], ignore_index=True)
+ combined = combined.dropna(subset=["poverty_rate"])
+ combined["poverty_rate"] = combined["poverty_rate"].astype(float)
+ combined = combined.drop_duplicates(
+ subset=["GEOID", "year"], keep="last"
+ )
+ combined["_year_num"] = combined["year"].astype(int)
+ combined = combined.sort_values(
+ by=["_row_order", "_year_num"], kind="mergesort"
+ )
+ df = combined[["GEOID", "year", "poverty_rate"]].reset_index(drop=True)
+
+ # Atomic write to destination file
+ dst_dir = os.path.dirname(os.path.abspath(dst_path))
+ os.makedirs(dst_dir, exist_ok=True)
+ temp_path = None
+ try:
+ with tempfile.NamedTemporaryFile(
+ "w",
+ dir=dst_dir,
+ delete=False,
+ suffix=".tmp",
+ encoding="utf-8",
+ newline="",
+ ) as tmp_file:
+ temp_path = tmp_file.name
+ df.to_csv(tmp_file, index=False)
+
+ os.replace(temp_path, dst_path)
+ temp_path = None
+ finally:
+ if temp_path and os.path.exists(temp_path):
+ try:
+ os.unlink(temp_path)
+ except OSError:
+ pass
+
+ logging.info("Poverty dataset cleaned and saved successfully to %s!", dst_path)
+ logging.info("Shape: %s", df.shape)
+ return dst_path
+
+
+def main(argv):
+ """Main entrypoint for preprocessing the poverty dataset."""
+ del argv # Unused
+ try:
+ source_path = resolve_source_file_path(FLAGS.source_path)
+ preprocess_poverty(
+ src_path=source_path,
+ dst_path=FLAGS.cleaned_csv_path,
+ min_county_count=FLAGS.min_county_count,
+ min_survey_year=FLAGS.min_survey_year,
+ max_survey_year=FLAGS.max_survey_year,
+ )
+ except Exception as e:
+ logging.fatal(
+ "Failed to preprocess Commerce EDA Poverty dataset: %s",
+ e,
+ exc_info=True,
+ )
+
+
+if __name__ == "__main__":
+ app.run(main)
diff --git a/statvar_imports/commerce_eda_poverty/process_poverty_test.py b/statvar_imports/commerce_eda_poverty/process_poverty_test.py
new file mode 100644
index 0000000000..5b4cc3396d
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/process_poverty_test.py
@@ -0,0 +1,504 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for process_poverty.py."""
+
+import os
+import sys
+import tempfile
+import unittest
+
+import openpyxl
+import pandas as pd
+
+MODULE_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.abspath(os.path.join(MODULE_DIR, "..", ".."))
+sys.path.insert(0, PROJECT_ROOT)
+
+from statvar_imports.commerce_eda_poverty.process_poverty import (
+ clean_geoid,
+ preprocess_poverty,
+ resolve_source_file_path,
+)
+
+
+def _create_mock_excel_file(filepath, rows, description_row=True):
+ """Creates a .xlsx workbook mimicking the official EDA PPC workbook."""
+ wb = openpyxl.Workbook()
+ ws = wb.active
+ try:
+ ws.title = "Underlying_Data"
+ if description_row:
+ ws.append([
+ "Table. FY2023 Persistent Poverty County Status - as of Data "
+ "Year 2021",
+ "", "", "", "", ""
+ ])
+ ws.append(
+ ["Identifing Information", "", "Census Bureau Data", "", "", ""]
+ )
+ ws.append([
+ "Name",
+ "GEOID",
+ "1990 Decennial Census, % in Poverty",
+ "2000 Decennial Census, % in Poverty",
+ "Most Recent Estimate, % in Poverty* ",
+ "Data Source―Most Recent Estimate",
+ ])
+ for r in rows:
+ ws.append(r)
+ wb.save(filepath)
+ finally:
+ wb.close()
+
+
+class TestProcessPoverty(unittest.TestCase):
+
+ def test_clean_geoid(self):
+ # 5-digit strings
+ self.assertEqual(clean_geoid("01001"), "01001")
+ self.assertEqual(clean_geoid("72143"), "72143")
+ self.assertEqual(clean_geoid("78010"), "78010")
+
+ # 4-digit zero-padding (e.g. Alabama 01005, Alaska 02090)
+ self.assertEqual(clean_geoid("1005"), "01005")
+ self.assertEqual(clean_geoid("2090"), "02090")
+
+ # Float strings
+ self.assertEqual(clean_geoid("1001.0"), "01001")
+ self.assertEqual(clean_geoid("01001.0"), "01001")
+ self.assertEqual(clean_geoid("1001.00"), "01001")
+
+ # Numeric floats and ints
+ self.assertEqual(clean_geoid(1001.0), "01001")
+ self.assertEqual(clean_geoid(2090), "02090")
+
+ # State summary FIPS ending in 000 must return None
+ self.assertIsNone(clean_geoid("01000"))
+ self.assertIsNone(clean_geoid("1000"))
+ self.assertIsNone(clean_geoid("72000"))
+
+ # Invalid cases returning None
+ self.assertIsNone(clean_geoid("1001.5"))
+ self.assertIsNone(clean_geoid("-1001.0"))
+ self.assertIsNone(clean_geoid(-1001.0))
+ self.assertIsNone(clean_geoid("abc"))
+ self.assertIsNone(clean_geoid(""))
+ self.assertIsNone(clean_geoid(None))
+ self.assertIsNone(clean_geoid("00100"))
+ self.assertIsNone(clean_geoid("0100"))
+ self.assertIsNone(clean_geoid("99001"))
+ self.assertIsNone(clean_geoid("123456"))
+
+ def test_resolve_source_file_path_specified(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ sample_file = os.path.join(tmpdir, "custom.csv")
+ with open(sample_file, "w") as f:
+ f.write("a,b\n1,2\n")
+ self.assertEqual(resolve_source_file_path(sample_file), sample_file)
+
+ def test_resolve_source_file_path_missing_raises(self):
+ with self.assertRaises(FileNotFoundError):
+ resolve_source_file_path("/nonexistent/file.csv")
+
+ def test_preprocess_poverty_missing_file(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ missing_path = os.path.join(tmpdir, "nonexistent.csv")
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError):
+ preprocess_poverty(src_path=missing_path, dst_path=dst_path)
+
+ def test_preprocess_poverty_empty_file(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ empty_path = os.path.join(tmpdir, "empty.csv")
+ with open(empty_path, "w") as f:
+ pass
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError):
+ preprocess_poverty(src_path=empty_path, dst_path=dst_path)
+
+ def test_preprocess_poverty_header_only_empty(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ header_only = os.path.join(tmpdir, "header_only.csv")
+ with open(header_only, "w") as f:
+ f.write(
+ "Header 1\nHeader 2\n"
+ "Name,GEOID,\"1990 Decennial Census, % in Poverty\","
+ "\"2000 Decennial Census, % in Poverty\","
+ "\"Most Recent Estimate, % in Poverty*\"\n"
+ )
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError) as ctx:
+ preprocess_poverty(src_path=header_only, dst_path=dst_path)
+ self.assertIn(
+ "File not read properly into dataframe", str(ctx.exception)
+ )
+
+ def test_preprocess_poverty_no_geoid_header_raises(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ no_geoid_csv = os.path.join(tmpdir, "no_geoid.csv")
+ with open(no_geoid_csv, "w") as f:
+ f.write("Line 1\nLine 2\nCol1,Col2\nVal1,Val2\n")
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError) as ctx:
+ preprocess_poverty(src_path=no_geoid_csv, dst_path=dst_path)
+ self.assertIn(
+ "Could not find header row containing 'GEOID'",
+ str(ctx.exception),
+ )
+
+ def test_preprocess_poverty_missing_columns(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ bad_csv = os.path.join(tmpdir, "bad.csv")
+ with open(bad_csv, "w") as f:
+ f.write("Line 1\nLine 2\nGEOID,OtherCol\n01001,10.0\n")
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError):
+ preprocess_poverty(src_path=bad_csv, dst_path=dst_path)
+
+ def test_preprocess_poverty_unexpected_survey_year(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ for bad_source in ["SAIPE, 1999", "SAIPE, 2099"]:
+ future_csv = os.path.join(tmpdir, "future.csv")
+ with open(future_csv, "w") as f:
+ f.write(
+ "Header 1\nHeader 2\n"
+ "Name,GEOID,\"1990 Decennial Census, % in Poverty\","
+ "\"2000 Decennial Census, % in Poverty\","
+ "\"Most Recent Estimate, % in Poverty*\","
+ "\"Data Source―Most Recent Estimate\"\n"
+ f'"Autauga County, AL",01001,15.7,10.9,13.3,"{bad_source}"\n'
+ )
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError) as ctx:
+ preprocess_poverty(
+ src_path=future_csv, dst_path=dst_path, min_county_count=1
+ )
+ self.assertIn("Unexpected survey year", str(ctx.exception))
+
+ malformed_csv = os.path.join(tmpdir, "malformed.csv")
+ with open(malformed_csv, "w") as f:
+ f.write(
+ "Header 1\nHeader 2\n"
+ "Name,GEOID,\"1990 Decennial Census, % in Poverty\","
+ "\"2000 Decennial Census, % in Poverty\","
+ "\"Most Recent Estimate, % in Poverty*\","
+ "\"Data Source―Most Recent Estimate\"\n"
+ '"Autauga County, AL",01001,15.7,10.9,13.3,"SAIPE FY23"\n'
+ )
+ with self.assertRaises(ValueError) as ctx:
+ preprocess_poverty(
+ src_path=malformed_csv, dst_path=dst_path, min_county_count=1
+ )
+ self.assertIn("Unrecognized survey year", str(ctx.exception))
+
+ def test_preprocess_poverty_valid_survey_year_range(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ sample_csv = os.path.join(tmpdir, "valid_year.csv")
+ with open(sample_csv, "w") as f:
+ f.write(
+ "Header 1\nHeader 2\n"
+ "Name,GEOID,\"1990 Decennial Census, % in Poverty\","
+ "\"2000 Decennial Census, % in Poverty\","
+ "\"Most Recent Estimate, % in Poverty*\","
+ "\"Data Source―Most Recent Estimate\"\n"
+ '"Autauga County, AL",01001,15.7,10.9,13.3,"SAIPE, 2023"\n'
+ )
+ dst_path = os.path.join(tmpdir, "output.csv")
+ preprocess_poverty(
+ src_path=sample_csv, dst_path=dst_path, min_county_count=1
+ )
+ self.assertTrue(os.path.exists(dst_path))
+ df = pd.read_csv(dst_path, dtype={"GEOID": str, "year": str})
+ self.assertEqual(
+ list(df.columns), ["GEOID", "year", "poverty_rate"]
+ )
+ row_2023 = df[(df["GEOID"] == "01001") & (df["year"] == "2023")]
+ self.assertEqual(len(row_2023), 1)
+ self.assertEqual(row_2023["poverty_rate"].iloc[0], 13.3)
+
+ def test_preprocess_poverty_min_county_count_failure(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ fixture_path = os.path.join(
+ MODULE_DIR, "test_data", "Poverty_input.csv"
+ )
+ dst_path = os.path.join(tmpdir, "output.csv")
+ with self.assertRaises(ValueError):
+ preprocess_poverty(
+ src_path=fixture_path,
+ dst_path=dst_path,
+ min_county_count=3000,
+ )
+
+ def test_preprocess_poverty_with_test_data(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ fixture_path = os.path.join(
+ MODULE_DIR, "test_data", "Poverty_input.csv"
+ )
+ expected_path = os.path.join(
+ MODULE_DIR, "test_data", "Poverty_expected_output.csv"
+ )
+ actual_csv = os.path.join(tmpdir, "Poverty_cleaned.csv")
+
+ preprocess_poverty(
+ src_path=fixture_path, dst_path=actual_csv, min_county_count=20
+ )
+
+ self.assertTrue(os.path.exists(actual_csv))
+ df_actual = pd.read_csv(
+ actual_csv, dtype={"GEOID": str, "year": str}
+ )
+ df_expected = pd.read_csv(
+ expected_path, dtype={"GEOID": str, "year": str}
+ )
+ pd.testing.assert_frame_equal(df_actual, df_expected)
+
+ # 30 representative counties/territories across 1990, 2000, 2020/2021
+ self.assertEqual(df_actual["GEOID"].nunique(), 30)
+ self.assertEqual(
+ list(df_actual.columns), ["GEOID", "year", "poverty_rate"]
+ )
+
+ # Verify 6 island territories (MP, VI) map recent estimate to year 2020
+ territory_rows = df_actual[
+ df_actual["GEOID"].str[:2].isin({"60", "66", "69", "78"})
+ ]
+ self.assertEqual(territory_rows["GEOID"].nunique(), 6)
+ self.assertIn("2020", set(territory_rows["year"]))
+ self.assertNotIn("2021", set(territory_rows["year"]))
+
+ # Verify state counties (24 AL counties) map recent estimate to 2021
+ state_rows = df_actual[
+ ~df_actual["GEOID"].str[:2].isin({"60", "66", "69", "78"})
+ ]
+ self.assertEqual(state_rows["GEOID"].nunique(), 24)
+ self.assertIn("2021", set(state_rows["year"]))
+ self.assertNotIn("2020", set(state_rows["year"]))
+
+ def test_preprocess_poverty_edge_cases(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ raw_csv = os.path.join(tmpdir, "raw.csv")
+ actual_csv = os.path.join(tmpdir, "cleaned.csv")
+
+ raw_content = (
+ "Header 1\n"
+ "Header 2\n"
+ "Name,GEOID,\"1990 Decennial Census, % in Poverty\","
+ "\"2000 Decennial Census, % in Poverty\","
+ "\"Most Recent Estimate, % in Poverty* \"\n"
+ '"Autauga County, AL",01001,15.7,10.9,13.3\n'
+ '"Alabama State Summary",01000,18.0,16.0,15.0\n'
+ '"Yukon-Koyukuk, AK",2090,7.6,7.8,9.6\n'
+ '"Eastern District, AS",60010,25.0,28.0,30.0\n'
+ '"Barbour County, AL",01005.0,25.2,26.8,29.0\n'
+ '"Bibb County, AL",1007.0,21.2,20.6,24.9\n'
+ '"Blount County, AL",01009,-5.0,150.0,14.5\n'
+ '"Bullock County, AL",01011,-10.0,120.0,999.0\n'
+ '"Invalid 1",99001,15.0,15.0,15.0\n'
+ '"Invalid 2",abc,10.0,10.0,10.0\n'
+ '"Invalid 3",0100,5.0,4.2,3.1\n'
+ '"Source footnote",,,,\n'
+ )
+ with open(raw_csv, "w") as f:
+ f.write(raw_content)
+
+ preprocess_poverty(
+ src_path=raw_csv, dst_path=actual_csv, min_county_count=1
+ )
+ df_actual = pd.read_csv(
+ actual_csv, dtype={"GEOID": str, "year": str}
+ )
+
+ expected_data = {
+ "GEOID": [
+ "01001", "01001", "01001",
+ "02090", "02090", "02090",
+ "60010", "60010", "60010",
+ "01005", "01005", "01005",
+ "01007", "01007", "01007",
+ "01009",
+ ],
+ "year": [
+ "1990", "2000", "2021",
+ "1990", "2000", "2021",
+ "1990", "2000", "2020",
+ "1990", "2000", "2021",
+ "1990", "2000", "2021",
+ "2021",
+ ],
+ "poverty_rate": [
+ 15.7, 10.9, 13.3,
+ 7.6, 7.8, 9.6,
+ 25.0, 28.0, 30.0,
+ 25.2, 26.8, 29.0,
+ 21.2, 20.6, 24.9,
+ 14.5,
+ ],
+ }
+ df_expected = pd.DataFrame(expected_data)
+ pd.testing.assert_frame_equal(df_actual, df_expected)
+
+ def test_preprocess_poverty_from_excel(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ excel_path = os.path.join(tmpdir, "mock.xlsx")
+ cleaned_csv = os.path.join(tmpdir, "cleaned.csv")
+
+ mock_rows = [
+ [
+ "Autauga County, AL",
+ "01001",
+ 15.7,
+ 10.9,
+ 13.3,
+ "SAIPE, 2021",
+ ],
+ [
+ "Eastern District, AS",
+ "60010",
+ 25.0,
+ 28.0,
+ 30.0,
+ "Decennial Census, 2020",
+ ],
+ ]
+ _create_mock_excel_file(excel_path, mock_rows)
+
+ preprocess_poverty(
+ src_path=excel_path, dst_path=cleaned_csv, min_county_count=1
+ )
+ self.assertTrue(os.path.exists(cleaned_csv))
+ df = pd.read_csv(cleaned_csv, dtype={"GEOID": str, "year": str})
+ self.assertEqual(len(df), 6)
+ self.assertEqual(
+ df.loc[
+ (df["GEOID"] == "01001") & (df["year"] == "2021"),
+ "poverty_rate",
+ ].iloc[0],
+ 13.3,
+ )
+ self.assertEqual(
+ df.loc[
+ (df["GEOID"] == "60010") & (df["year"] == "2020"),
+ "poverty_rate",
+ ].iloc[0],
+ 30.0,
+ )
+
+
+ def test_preprocess_poverty_header_detection_with_1990_title_row(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ csv_path = os.path.join(tmpdir, "title_with_1990.csv")
+ cleaned_csv = os.path.join(tmpdir, "cleaned.csv")
+ headers = (
+ 'Name,GEOID,"1990 Decennial Census, % in Poverty",'
+ '"2000 Decennial Census, % in Poverty",'
+ '"Most Recent Estimate, % in Poverty*",'
+ 'Data Source―Most Recent Estimate\n'
+ )
+ with open(csv_path, "w", encoding="utf-8") as f:
+ f.write(
+ '"Table 1. EDA PPC Report (1990-2021)\n'
+ 'Summary by Census GEOID"\n'
+ )
+ f.write(headers)
+ f.write(
+ '"Autauga County, AL",01001,15.7,10.9,13.3,"SAIPE, 2021"\n'
+ )
+
+ preprocess_poverty(
+ src_path=csv_path, dst_path=cleaned_csv, min_county_count=1
+ )
+ self.assertTrue(os.path.exists(cleaned_csv))
+ df = pd.read_csv(cleaned_csv, dtype={"GEOID": str, "year": str})
+ self.assertEqual(len(df), 3)
+ self.assertEqual(df["GEOID"].iloc[0], "01001")
+
+ def test_preprocess_poverty_survey_year_with_footnotes_and_citations(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ csv_path = os.path.join(tmpdir, "footnotes.csv")
+ cleaned_csv = os.path.join(tmpdir, "cleaned.csv")
+ headers = (
+ 'Name,GEOID,"1990 Decennial Census, % in Poverty",'
+ '"2000 Decennial Census, % in Poverty",'
+ '"Most Recent Estimate, % in Poverty*",'
+ 'Data Source―Most Recent Estimate\n'
+ )
+ with open(csv_path, "w", encoding="utf-8") as f:
+ f.write(headers)
+ f.write(
+ '"Autauga County, AL",01001,15.7,10.9,13.3,"SAIPE, 2021*"\n'
+ )
+ f.write(
+ '"Barbour County, AL",01005,25.2,26.8,29.0,'
+ '"SAIPE, 2021 [1]"\n'
+ )
+ f.write(
+ '"Adjuntas Municipio, PR",72001,63.3,57.5,58.8,'
+ '"ACS 5-YEAR, 2017-2021"\n'
+ )
+ f.write(
+ '"Historical County, AL",01007,21.2,20.6,NA,"NA"\n'
+ )
+ f.write(
+ '"Sentinel County, AL",01009,18.4,17.2,-999,"Not Available"\n'
+ )
+
+ preprocess_poverty(
+ src_path=csv_path, dst_path=cleaned_csv, min_county_count=1
+ )
+ self.assertTrue(os.path.exists(cleaned_csv))
+ df = pd.read_csv(cleaned_csv, dtype={"GEOID": str, "year": str})
+ self.assertEqual(len(df), 13)
+ recent_years = set(
+ df.loc[df["year"] != "1990"].loc[df["year"] != "2000", "year"]
+ )
+ self.assertEqual(recent_years, {"2021"})
+ pr_rows = df[df["GEOID"] == "72001"]
+ self.assertEqual(list(pr_rows["year"]), ["1990", "2000", "2021"])
+ self.assertEqual(list(pr_rows["poverty_rate"]), [63.3, 57.5, 58.8])
+ na_rows = df[df["GEOID"] == "01007"]
+ self.assertEqual(list(na_rows["year"]), ["1990", "2000"])
+ sentinel_rows = df[df["GEOID"] == "01009"]
+ self.assertEqual(list(sentinel_rows["year"]), ["1990", "2000"])
+
+ def test_preprocess_poverty_deduplicates_duplicate_geoids(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ csv_path = os.path.join(tmpdir, "dup_geoid.csv")
+ cleaned_csv = os.path.join(tmpdir, "cleaned.csv")
+ headers = (
+ 'Name,GEOID,"1990 Decennial Census, % in Poverty",'
+ '"2000 Decennial Census, % in Poverty",'
+ '"Most Recent Estimate, % in Poverty*",'
+ 'Data Source―Most Recent Estimate\n'
+ )
+ with open(csv_path, "w", encoding="utf-8") as f:
+ f.write(headers)
+ f.write(
+ '"Autauga County, AL",1001,15.0,10.0,12.0,"SAIPE, 2021"\n'
+ )
+ f.write(
+ '"Autauga County, AL",01001,15.7,10.9,13.3,"SAIPE, 2021"\n'
+ )
+
+ preprocess_poverty(
+ src_path=csv_path, dst_path=cleaned_csv, min_county_count=1
+ )
+ df = pd.read_csv(cleaned_csv, dtype={"GEOID": str, "year": str})
+ self.assertEqual(len(df), 3)
+ self.assertEqual(list(df["year"]), ["1990", "2000", "2021"])
+ self.assertEqual(list(df["poverty_rate"]), [15.7, 10.9, 13.3])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/statvar_imports/commerce_eda_poverty/test_data/Poverty_expected_output.csv b/statvar_imports/commerce_eda_poverty/test_data/Poverty_expected_output.csv
new file mode 100644
index 0000000000..aef0c9893c
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/test_data/Poverty_expected_output.csv
@@ -0,0 +1,91 @@
+GEOID,year,poverty_rate
+01001,1990,15.7
+01001,2000,10.9
+01001,2021,13.3
+01003,1990,14.3
+01003,2000,10.1
+01003,2021,12.5
+01005,1990,25.2
+01005,2000,26.8
+01005,2021,29.0
+01007,1990,21.2
+01007,2000,20.6
+01007,2021,24.9
+01009,1990,15.3
+01009,2000,11.7
+01009,2021,14.5
+01011,1990,36.5
+01011,2000,33.5
+01011,2021,39.1
+01013,1990,31.5
+01013,2000,24.6
+01013,2021,27.2
+01015,1990,15.7
+01015,2000,16.1
+01015,2021,21.8
+01017,1990,18.8
+01017,2000,17.0
+01017,2021,23.7
+01019,1990,17.6
+01019,2000,15.6
+01019,2021,21.6
+01021,1990,17.1
+01021,2000,15.7
+01021,2021,18.7
+01023,1990,30.2
+01023,2000,24.5
+01023,2021,28.1
+01025,1990,25.9
+01025,2000,22.6
+01025,2021,23.7
+01027,1990,17.4
+01027,2000,17.1
+01027,2021,21.8
+01029,1990,15.3
+01029,2000,13.9
+01029,2021,18.4
+01031,1990,15.5
+01031,2000,14.7
+01031,2021,18.1
+01033,1990,14.6
+01033,2000,14.0
+01033,2021,19.3
+01035,1990,29.7
+01035,2000,26.6
+01035,2021,28.0
+01037,1990,18.2
+01037,2000,14.9
+01037,2021,21.5
+01039,1990,22.0
+01039,2000,18.4
+01039,2021,23.1
+01041,1990,24.3
+01041,2000,22.1
+01041,2021,21.7
+01043,1990,15.3
+01043,2000,13.0
+01043,2021,16.0
+01045,1990,14.8
+01045,2000,15.1
+01045,2021,18.7
+01047,1990,36.2
+01047,2000,31.1
+01047,2021,35.5
+69100,1990,51.7
+69100,2000,34.2
+69100,2020,36.4
+69110,1990,51.4
+69110,2000,46.9
+69110,2020,38.2
+69120,1990,51.3
+69120,2000,41.2
+69120,2020,35.5
+78010,1990,33.7
+78010,2000,38.7
+78010,2020,24.9
+78020,1990,15.0
+78020,2000,18.5
+78020,2020,18.9
+78030,1990,21.3
+78030,2000,27.2
+78030,2020,21.2
diff --git a/statvar_imports/commerce_eda_poverty/test_data/Poverty_input.csv b/statvar_imports/commerce_eda_poverty/test_data/Poverty_input.csv
new file mode 100644
index 0000000000..8ba17bec71
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/test_data/Poverty_input.csv
@@ -0,0 +1,39 @@
+Table. FY2023 Persistent Poverty County Status - as of Data Year 2021,,,,,,,,,
+Identifing Information,,Census Bureau Data,,,,FY23 Persistent Poverty,Census GEO PPC Code,Notes,
+Name,GEOID,"1990 Decennial Census, % in Poverty","2000 Decennial Census, % in Poverty","Most Recent Estimate, % in Poverty* ",Data Source―Most Recent Estimate,,,,
+"Autauga County, AL",01001,15.7,10.9,13.3,"SAIPE, 2021",No,1,,
+"Baldwin County, AL",01003,14.3,10.1,12.5,"SAIPE, 2021",No,1,,
+"Barbour County, AL",01005,25.2,26.8,29,"SAIPE, 2021",Yes,2,,
+"Bibb County, AL",01007,21.2,20.6,24.9,"SAIPE, 2021",Yes,2,,
+"Blount County, AL",01009,15.3,11.7,14.5,"SAIPE, 2021",No,1,,
+"Bullock County, AL",01011,36.5,33.5,39.1,"SAIPE, 2021",Yes,2,,
+"Butler County, AL",01013,31.5,24.6,27.2,"SAIPE, 2021",Yes,2,,
+"Calhoun County, AL",01015,15.7,16.1,21.8,"SAIPE, 2021",No,1,,
+"Chambers County, AL",01017,18.8,17,23.7,"SAIPE, 2021",No,1,,
+"Cherokee County, AL",01019,17.6,15.6,21.6,"SAIPE, 2021",No,1,,
+"Chilton County, AL",01021,17.1,15.7,18.7,"SAIPE, 2021",No,1,,
+"Choctaw County, AL",01023,30.2,24.5,28.1,"SAIPE, 2021",Yes,2,,
+"Clarke County, AL",01025,25.9,22.6,23.7,"SAIPE, 2021",Yes,2,,
+"Clay County, AL",01027,17.4,17.1,21.8,"SAIPE, 2021",No,1,,
+"Cleburne County, AL",01029,15.3,13.9,18.4,"SAIPE, 2021",No,1,,
+"Coffee County, AL",01031,15.5,14.7,18.1,"SAIPE, 2021",No,1,,
+"Colbert County, AL",01033,14.6,14,19.3,"SAIPE, 2021",No,1,,
+"Conecuh County, AL",01035,29.7,26.6,28,"SAIPE, 2021",Yes,2,,
+"Coosa County, AL",01037,18.2,14.9,21.5,"SAIPE, 2021",No,1,,
+"Covington County, AL",01039,22,18.4,23.1,"SAIPE, 2021",No,1,,
+"Crenshaw County, AL",01041,24.3,22.1,21.7,"SAIPE, 2021",Yes,2,,
+"Cullman County, AL",01043,15.3,13,16,"SAIPE, 2021",No,1,,
+"Dale County, AL",01045,14.8,15.1,18.7,"SAIPE, 2021",No,1,,
+"Dallas County, AL",01047,36.2,31.1,35.5,"SAIPE, 2021",Yes,2,,
+"Rota Municipality, MP",69100,51.7,34.2,36.4,"Decennial Census, 2020",Yes,2,,
+"Saipan Municipality, MP",69110,51.4,46.9,38.2,"Decennial Census, 2020",Yes,2,,
+"Tinian Municipality, MP",69120,51.3,41.2,35.5,"Decennial Census, 2020",Yes,2,,
+"St. Croix Island, VI",78010,33.7,38.7,24.9,"Decennial Census, 2020",Yes,2,,
+"St. John Island, VI",78020,15,18.5,18.9,"Decennial Census, 2020",No,1,,
+"St. Thomas Island, VI",78030,21.3,27.2,21.2,"Decennial Census, 2020",Yes,2,,
+"Source: 1990, Decennial Census, Public Use; 2000, Decennial Census, Public Use; 2021 Small Area Income and Poverty Estimates, Public Use; 2017-2021 5-Year American Community Survey, Public Use; 2020 Decennial Census, Public Use",,,,,,,,,
+"*Most recent estimate, % in poverty -- For continental U.S., it's 2021 SAIPE, which uses the upper bound of the confidence interval (for statistical uncertainty). For Puerto Rico, it's 2021 ACS 5-year, which uses the percentage estimate+MOE. For Island Areas, it's 2020 decennial number.",,,,,,,,,
+"Note: All three data points must have a valid numerical percent in poverty >= 19.5 percent to be considered in persistent poverty. We are using 19.5 because values for percent in poverty are reflected to one decimal point and EDA rounds up to whole integers. Counties without a numerical percent in poverty are coded as NA. These included areas that were not in existence, experienced a boundary change or are uninhabited. See Technical Documentation for more details.",,,,,,,,,
+"For more information on the decennial census, see the technical documentation here: https://www.census.gov/programs-surveys/decennial-census/technical-documentation/complete-technical-documents.html",,,,,,,,,
+"For more information on the American Community Survey and the margins of error, see the technical documentation here: https://www.census.gov/programs-surveys/acs/technical-documentation.html",,,,,,,,,
+"For more information on SAIPE and the estimation of the 90-percent confidence intervals, see the technical documentation here: https://www.census.gov/programs-surveys/saipe/technical-documentation/methodology/counties-states/county-level.html",,,,,,,,,
diff --git a/statvar_imports/commerce_eda_poverty/validation_config.json b/statvar_imports/commerce_eda_poverty/validation_config.json
new file mode 100644
index 0000000000..a8d76da460
--- /dev/null
+++ b/statvar_imports/commerce_eda_poverty/validation_config.json
@@ -0,0 +1,37 @@
+{
+ "schema_version": "1.0",
+ "rules": [
+ {
+ "rule_id": "check_date_span_sql",
+ "description": "Verify that every active StatVar covers historical 1990 benchmarks through at least 2021.",
+ "validator": "SQL_VALIDATOR",
+ "scope": {
+ "data_source": "stats"
+ },
+ "params": {
+ "query": "SELECT StatVar, MinDate, MaxDate, COUNT(*) OVER () AS total_svs FROM stats",
+ "condition": "CAST(MinDate AS INTEGER) <= 1990 AND CAST(MaxDate AS INTEGER) >= 2021 AND total_svs > 0"
+ }
+ },
+ {
+ "rule_id": "check_max_date_consistent",
+ "description": "Ensures that all statistical variables share a consistent maximum observation date.",
+ "validator": "MAX_DATE_CONSISTENT",
+ "scope": {
+ "data_source": "stats"
+ },
+ "params": {}
+ },
+ {
+ "rule_id": "check_deleted_records_percent",
+ "description": "Checks that the percentage of deleted records is within 0.1% threshold.",
+ "validator": "DELETED_RECORDS_PERCENT",
+ "scope": {
+ "data_source": "differ_summary"
+ },
+ "params": {
+ "threshold": 0.1
+ }
+ }
+ ]
+}