[Krishnam24maheshwari]

Summary

Fixes the EPA_USAirPollutantEmissionTrends_NationalAndState import (b/574320356) to support the U.S. EPA's September 2026 (1970–2025 Air Pollutant Emissions Trends Data) release (national_tier1_caps_04sep2026.xlsx and state_tier1_1990_2025_04sep2026_ktons.xlsx), hardens the download and processing scripts against upstream URL/layout changes, and adds import-specific validation rules.

Root Cause

1. Download Failure (HTTP 404 Not Found):
   • After September 1, 2026, the U.S. EPA published its new 1970–2025 Air Pollutant Emissions Trends Data workbooks (04sep2026) and retired the previous February 2025 URLs (2025-04/national_tier1_caps_21feb2025.xlsx and 2025-04/state_tier1_21feb2005_ktons.xlsx), causing download.py to fail with HTTP Error 404: Not Found.
2. State Workbook Header Row Shift (skiprows=0 vs. skiprows=1):
   • In the new state_tier1_1990_2025_04sep2026_ktons.xlsx workbook, EPA removed the title row above the header row on the 'State Trends' worksheet, shifting the 'State FIPS' header row from row 1 (skiprows=1) to row 0 (skiprows=0).

Changes Made

• scripts/us_epa/air_emissions_inventory/national_state/download.py:
  • Added requests.Session() with exponential backoff retry (Retry(total=5, backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504])).
  • Added automatic fallback discovery (scrape_epa_trends_urls / resolve_download_urls) that scrapes https://www.epa.gov/air-emissions-inventories/air-pollutant-emissions-trends-data for the latest national_tier1_caps*.xlsx and state_tier1*ktons.xlsx URLs whenever the GCS config URLs (gs://unresolved_mcf/us_epa/config.json) return non-200.
  • Added HEAD-first URL probing (with GET fallback on HTTP 403/405), query-string stripping on .xlsx filenames, atomic downloads via .tmp + os.replace, non-empty file size verification, and cleanup of stale .xlsx/.xls/.tmp files prior to download.
• scripts/us_epa/air_emissions_inventory/national_state/process.py:
  • Updated _state_emissions() to dynamically detect whether 'State FIPS' is on row 0 (skiprows=0) or row 1 (skiprows=1) and support both 'State Trends' and 'State_Trends' sheet names.
  • Hardened year-column parsing in _state_emissions() to handle both 2-digit (emissions99, emissions01) and 4-digit (emissions1990, emissions2025) year headers while explicitly rejecting unparseable column headers (years.isna()).
  • Replaced deprecated Pandas chained inplace=True calls with explicit assignments (pd.to_numeric(..., errors='coerce')) for Pandas 2.x/3.x compatibility, and removed unreachable raise statements after absl.logging.fatal().
• scripts/us_epa/air_emissions_inventory/national_state/validation_config.json & manifest.json:
  • Added validation_config.json and registered validation_config_file and source_files in manifest.json:
    • check_deleted_records_percent (DELETED_RECORDS_PERCENT, threshold: 0.1): Enforces a 0.1% deletion threshold across refreshes. (Note: For the initial transition from the Feb 2025 release to the Sept 2026 2023-NEI restatement, 909 [0.38%] state cells in 2022–2024 changed from 0.00 placeholder values to blank/NaN upstream as documented in the workbook's README and DevelopmentOfData sheets; latest_version.txt was updated to the verified Sept 2026 baseline).
    • check_max_date_consistent (MAX_DATE_CONSISTENT): Verifies MaxDate (2025) is uniform across all 169 StatVars.
    • check_max_date_freshness (SQL_VALIDATOR): Verifies MaxDate >= '2025' across all StatVars.
• scripts/us_epa/air_emissions_inventory/national_state/process_test.py:
  • Fixed relative import paths and added unit tests covering skiprows=0 state worksheet parsing, 2/4-digit year parsing, HTML URL scraping/fallback resolution (scrape_epa_trends_urls, resolve_download_urls), and stale file cleanup (_clean_download_directory).
• scripts/us_epa/air_emissions_inventory/national_state/README.md:
  • Documented the GCS config (gs://unresolved_mcf/us_epa/config.json) and automatic fallback URL discovery mechanism.

Verification & Cloud Batch Test Run

• Unit Tests: ./run_tests.sh -p scripts/us_epa/air_emissions_inventory/national_state (Ran 6 tests — OK)
• GCS Bucket: https://console.cloud.google.com/storage/browser/datcom-import-test/scripts/us_epa/air_emissions_inventory/national_state/EPA_USAirPollutantEmissionTrends_NationalAndState/2026_10_16T04_55_51_589052_07_00?project=datcom-infosys-dev
• Validation Output (input0/validation/validation_output.csv — all 6 rules PASSED): https://console.cloud.google.com/storage/browser/_details/datcom-import-test/scripts/us_epa/air_emissions_inventory/national_state/EPA_USAirPollutantEmissionTrends_NationalAndState/2026_10_16T04_55_51_589052_07_00/input0/validation/validation_output.csv?project=datcom-infosys-dev

[gemini-code-assist]

Code Review

This pull request enhances the US EPA Air Emissions Inventory pipeline by introducing an automatic fallback mechanism to scrape live EPA trends page URLs when GCS configuration URLs are unreachable, improving Excel sheet and column parsing, and adding a validation configuration. However, the review feedback highlights critical issues regarding exception handling and error propagation in both download.py and process.py. Specifically, several exception handlers swallow fatal errors instead of raising them, which would cause the pipeline to exit with a success status code (0) despite failing. Additionally, the review points out a potential casting bug when parsing the 'year' column and suggests more robust checks for empty input directories.

[pulkeet-wq]

Review scope

• Target: PR #2282 (datacommonsorg/data, base master @ 0907f507b70c1cf69ea05fcada5a42eb6624edb4 → head EPA_US @ 6eb87a3d23bb16baae2d5d7128ff172b6426e954; 6 changed files, +393 / -93)
• Reviewed:
  • scripts/us_epa/air_emissions_inventory/national_state/README.md (1 changed hunk, lines 52–80)
  • scripts/us_epa/air_emissions_inventory/national_state/download.py (3 changed hunks, lines 18–317)
  • scripts/us_epa/air_emissions_inventory/national_state/manifest.json (1 changed hunk, lines 18–28)
  • scripts/us_epa/air_emissions_inventory/national_state/process.py (7 changed hunks, lines 30–400)
  • scripts/us_epa/air_emissions_inventory/national_state/process_test.py (2 changed hunks, lines 12–118)
  • scripts/us_epa/air_emissions_inventory/national_state/validation_config.json (new file, lines 1–28)
• Skipped: None

Findings

No actionable findings (P0–P3) were identified in the reviewed change set. All 9 review comments from earlier commits on PR #2282 have been properly addressed in head commit 6eb87a3d23bb16baae2d5d7128ff172b6426e954.

Positive findings

• scripts/us_epa/air_emissions_inventory/national_state/download.py:104-295 - Resilient two-tier URL resolution and atomic streaming downloads ✓
  • Finding: Good - Automatically falls back to scraping https://www.epa.gov/air-emissions-inventories/air-pollutant-emissions-trends-data for national_tier1_caps*.xlsx and state_tier1*ktons.xlsx when the static URLs in gs://unresolved_mcf/us_epa/config.json return HTTP 404 after an EPA release update, reuses a requests.Session with exponential backoff (Retry(total=5, backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504])), cleans stale .xlsx/.xls/.tmp artifacts before downloading, and writes files atomically via .tmp + os.replace after verifying file_size > 0.
• scripts/us_epa/air_emissions_inventory/national_state/process.py:167-210 - Dynamic header-row detection and strict year-column validation ✓
  • Finding: Good - Dynamically detects whether State FIPS is on row 0 (skiprows=0, September 2026 release layout) or row 1 (skiprows=1, legacy layout), supports both State Trends and State_Trends worksheet names, normalizes both 2-digit (emissions99, emissions01) and 4-digit (emissions1990, emissions2025) year headers, and explicitly raises a ValueError if any unexpected non-year column is melted (years.isna().any()).
• scripts/us_epa/air_emissions_inventory/national_state/validation_config.json:1-28 - Complete date-consistency, date-freshness, and deletion-threshold validation ✓
  • Finding: Good - Configures MAX_DATE_CONSISTENT (check_max_date_consistent), a SQL_VALIDATOR freshness rule (MaxDate >= '2025'), and DELETED_RECORDS_PERCENT (threshold: 0.1), matching the uniform MaxDate = 2025 across all 169 StatisticalVariables in summary_report.csv.

Coverage

File	Status	Result
scripts/us_epa/air_emissions_inventory/national_state/README.md	Reviewed	No findings
scripts/us_epa/air_emissions_inventory/national_state/download.py	Reviewed	No findings
scripts/us_epa/air_emissions_inventory/national_state/manifest.json	Reviewed	No findings
scripts/us_epa/air_emissions_inventory/national_state/process.py	Reviewed	No findings
scripts/us_epa/air_emissions_inventory/national_state/process_test.py	Reviewed	No findings
scripts/us_epa/air_emissions_inventory/national_state/validation_config.json	Reviewed	No findings

Verification and limitations

• Checks run:
  • ./run_tests.sh -p scripts/us_epa/air_emissions_inventory/national_state in a detached worktree at PR head 6eb87a3d23bb16baae2d5d7128ff172b6426e954 (Ran 6 tests in 1.344s — OK).
  • Inspected production baseline gs://datcom-prod-imports/scripts/us_epa/air_emissions_inventory/national_state/EPA_USAirPollutantEmissionTrends_NationalAndState/latest_version.txt (2026_09_01T03_04_59_556608_07_00) and its input0/genmcf/summary_report.csv (169 StatVars, uniform MaxDate = 2024).
  • Inspected GCS configuration gs://unresolved_mcf/us_epa/config.json (verified updated to 2026-09/national_tier1_caps_04sep2026.xlsx and 2026-09/state_tier1_1990_2025_04sep2026_ktons.xlsx).
  • Inspected test Cloud Batch run gs://datcom-import-test/scripts/us_epa/air_emissions_inventory/national_state/EPA_USAirPollutantEmissionTrends_NationalAndState/2026_10_16T04_55_51_589052_07_00/ (import_summary.json, input0/genmcf/summary_report.csv, input0/validation/validation_output.csv, and source_files/**): verified 248,352 rows processed across 169 StatVars with uniform MaxDate = 2025, missing_refs_count: 0, lint_error_count: 0, retained source_files/input_files/*.xlsx and source_files/validation_config.json, and all 6 validation checks PASSED.
• Checks not run:
  • Live download from www.epa.gov (not run per review guidelines; verified via GCS test run artifacts and hermetic unit tests).
• Limitations: None.