Repository navigation
CDCWonderNatality : Automation - #2233
Conversation
Code fix unenergy
…nd pipeline scripts - Implement automated download script download.sh from GCS repository - Implement unified process.py pipeline for country, state, and county tables - Add unit tests in process_test.py and ensure deterministic sorting in preprocess.py - Add manifest.json for Cloud Run/Batch automation (annual cadence, 3 import inputs, no node_mcf) - Add validation_config.json with empty import, lint error, and missing refs checks - Update README.md with comprehensive documentation and usage instructions
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an automated end-to-end data processing pipeline for CDC Wonder Natality data, including new scripts for downloading, processing, and testing, along with a manifest and validation configuration. The review feedback highlights several improvement opportunities: making the geographic classification logic in process.py more robust, casting aggregated count statistics to integers (Int64) to prevent float representation, correcting a fallback test data copy that mixes state and county resolutions, ensuring output verification checks for non-empty files, and replacing the file command in download.sh with a more portable unzip -t check.
There was a problem hiding this comment.
Code Review
This pull request introduces automation scripts, configurations, and tests for the CDC Wonder Natality import, including an end-to-end processing pipeline (process.py) and a download script (download.sh). Feedback on the new pipeline highlights critical issues in process.py: the geographic classification logic for sample_geo contains flaws regarding DCID lengths and empty values; converting the Quantity column to numeric can result in float representations instead of integers; and copying state-level data to county-level output as a fallback introduces mismatched geographic associations.
- Use distinct output TMCF paths (country.tmcf, state.tmcf, county.tmcf) in manifest.json to prevent Cloud Batch GCS basename collision. - Remove dangling validation_config_file from manifest.json and README.md. - Ensure chronological sorting and filename-based routing in process.py. - Use regex Geo validation (geoId/XX vs geoId/XXXXX) and cast aggregated quantities to Int64. - Tighten error handling in download.sh: require gcloud, remove silent failure suppression, and verify all geographic resolution levels. - Track Buganizer issue b/555054352.
- Fix off-by-one check in --use_test_data county DCID synthesis using regex - Ensure Int64 cast and keep='last' in aggregate.py and process.py - Hardened download.sh to clean input dir and select latest dated version folder - Restore validation_config.json with SQL_VALIDATOR date freshness check - Add multi-bracket sorting, deduplication, and DCID assertions in process_test.py - Replace RuntimeError with logging.fatal in process.py - Update README.md with manual preprocessing/staging workflow and clean dependencies - Track Buganizer issue b/555057010.
- Fix yapf code formatting across scripts/us_cdc/natality to pass CI checks - Remove .astype(str) on nullable Int64 in process.py to prevent <NA> serialization - Handle pipefail with || true in download.sh and explicitly check dated snapshot dirs - Add 1995-2002 chronological bracket overlap test in process_test.py - Remove redundant sys.exit(1) calls after logging.fatal() in process.py - Track Buganizer issue b/555057010.
- Vectorize numeric parsing and integer formatting on non-null rows - Avoid setting Int64 dtype directly into string[pyarrow] columns - Add assertions in process_test.py ensuring no <NA> strings or float notations - b/555057010
|
All 5 findings have been resolved in CRA :
Additionally, the Cloud Batch automation job was re-run and succeeded end-to-end:
|
SandeepTuniki
left a comment
SandeepTuniki
left a comment
There was a problem hiding this comment.
Thanks for the changes! I left a few comments and a clarifying question below.
…WONDER - Add download.py with headless Chrome to scrape live CDC WONDER exports and retrieve historical baseline from GCS. - Refactor process.py to merge historical 1995-2022 baseline with live 2016-2024 data, preventing historical data loss. - Normalize gestational age, weight, and interval units to match Data Commons schema (Week, Gram, Month). - Clean up redundant legacy scripts, zip files, and bracket JSON configs. - Update manifest.json, validation_config.json, README.md, and unit test suite. - Resolves b/555057010.
- Fix silent data loss on TSV rows with notes/footnotes by retaining header and columns directly. - Sort historical brackets chronologically by year range and remove unused _merge_csv_files. - Terminate download pipeline via logging.fatal on baseline failure to prevent historical data loss. - Always aggregate national totals from final_state_df to ensure multi-decade coverage in country.csv. - Fix canonical unit mapping from Years to Year for Average Mother's Age. - Filter invalid place DCIDs (geoId/99, geoId/99999). - Update validation_config.json with lag_years <= 2 and scoped consistency rule across 9 active StatVars. - Normalize Count_* quantities to clean integer strings across all geographic levels in memory. - Increase download timeout to 180s and robustly filter candidate download files. - Add manifest.json to source_files and document prerequisites in README.md. - Expand unit test suite in process_test.py and download_test.py.
Hi @SandeepTuniki, All findings and review comments have been addressed in the latest commits:
Verification
Ready for re-review! |
- download.py: Raise RuntimeError on missing year listbox; add exponential backoff in retry loop. - manifest.json: Add explicit container resource limits (2 CPU, 8 GB RAM). - process.py: Make county Geo filtering null-safe. - validation_config.json: Bound latest date freshness query with MAX, IS NOT NULL, and lag between 0 and 3 years. - process_test.py: Add unit tests verifying freshness validation fails on missing StatVar and future dates.
…eshold rule (0.1%)
SandeepTuniki
left a comment
SandeepTuniki
left a comment
There was a problem hiding this comment.
Thanks for making the updates—the year-bracket tuple sorting (_extract_year_bracket) and .tmcf output generation/verification look good!
However, after checking the latest commits and Cloud Batch run (2026_10_12T01_56_58_821447_07_00), a few critical items are still missing or were reverted in the final commits (refreshing the demographic StatVars, restoring validation_config.json in manifest.json along with the date freshness/consistency checks, and avoiding the fabricated 0 counts from the historical country_cleaned_*.csv files).
Could you please review and address the inline comments? Thanks!
…on config & SQL rules, filter zero placeholders
|
Hi @SandeepTuniki, Thank you for the thorough review! All 6 inline comments have been addressed in commit 1. Summary of Fixes
2. Cloud Batch Verification (
|
SandeepTuniki
left a comment
SandeepTuniki
left a comment
There was a problem hiding this comment.
Thanks for addressing all the review comments. LGTM!
Summary
This PR implements automated pipeline support for CDC WONDER Natality data across Country, State, and County levels, addressing all code review findings.
Key Changes
download.py:--headless=new,--no-sandbox).B_3 = *None*) and all 6 active 2016+ demographic Group-By dimensions (Age of Mother 13=D149.V39,Mother's Single Race 15=D149.V49,Mother's Education=D149.V5,Mother's Hispanic Origin=D149.V43,Marital Status=D149.V27, andMother's Nativity=D149.V48) for State and County levels (selecting post-2022 years on demographic slices to stay below CDC WONDER's 75,000-row export limit).RuntimeErrorif the year selection listbox (V_D149.V20) or any optional measure checkbox (_OPTIONAL_MEASURES) is missing from the CDC WONDER form, and uses exponential backoff (5 * (2 ** (attempt - 1)), guarded byif attempt < max_retries) across retry attempts.gs://unresolved_mcf/cdc/wonder/natality/) with demographic breakdowns to ensure zero historical data loss.process.py:(start_year, end_year)sorting andkeep='last'precedence for overlapping years._DEMOGRAPHIC_CODE_MAP) and validates all 9 expected_MEASURE_MAPcolumns exist in the TSV header.Years->Year,Weeks->Week,Grams->Gram,Months->Month.geoId/99,geoId/00,geoId/99999,geoId/00000,geoId/*999).0.0placeholder rows on unknown/unstated"Not Available"slices (_MotherMaritalStatusUnknownOrNotStatedand_MotherRaceUnknownOrNotStatedforYear >= 2016) and computescountry.csvdirectly viaaggregate_state_to_country(final_state_df).manifest.json:country,state, andcounty,"validation_config_file": "validation_config.json", annual schedule ("0 0 1 6 *"), and containerresource_limits(cpu: 2,memory: 8).validation_config.json:check_latest_date_freshness(lag_years <= 3),check_scoped_date_consistency_active_statvars(distinct_dates = 1 AND active_statvar_count IN (1, 9, 46, 401)across allMaxDate >= 2022StatVars), andcheck_deleted_records_percent(threshold: 2.5).process_test.py&download_test.py:Test Environment Verification (
2026_10_21T23_41_40_528726_07_00)cdcwondernatality-samnotra-20261022-061014(ImportStatus.STAGING)input0): 1,425 rows (46 active 2016+ StatVars atMaxDate = 2024, 0 at2022), 1.04% deleted records (14 fabricated0rows removed), 0 missing refs, 0 lint errors, all 6 validation checks PASSED.input1): 363,912 rows (401 active 2016+ StatVars atMaxDate = 2024, 0 at2022), 1.93% deleted records (6,4260.0placeholder rows removed), 0 missing refs, 0 lint errors, all 6 validation checks PASSED.input2): 3,316,862 rows (401 active 2016+ StatVars atMaxDate = 2024, 0 at2022), 2.36% deleted records (72,8280.0placeholder rows removed), 0 missing refs, 0 lint errors, all 6 validation checks PASSED.Artifacts (
2026_10_21T23_41_40_528726_07_00):input0): Differ | Validation | Summary Report | Report JSONinput1): Differ | Validation | Summary Report | Report JSONinput2): Differ | Validation | Summary Report | Report JSON20231215) vs. GCS Differ Analysis: Analysis Report | Country Differ | State Differ | County DifferBuganizer: b/555057010