Skip to content

CensusACS5YearSurvey_SubjectTables_S2303 - #2287

Merged
goutamabrol merged 9 commits into
datacommonsorg:masterfrom
goutamabrol:S2303_refresh
Oct 22, 2026
Merged

goutamabrol merged 9 commits into
datacommonsorg:masterfrom
goutamabrol:S2303_refresh

Conversation

@goutamabrol

@goutamabrol goutamabrol commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Updates the processing scripts and documentation for the US Census ACS 5-Year Subject Table S2303 (Work Status in the Past 12 Months) to support the 2010–2024 dataset refresh across all 11 supported geographic summary levels.

Key Changes

  • scripts/us_census/acs5yr/subject_tables/S2303/process.py:
    • Subclasses SubjectTableDataLoaderBase from ../common/data_loader.py (S2303SubjectTableDataLoader) to restrict percentage-to-count conversion (_convert_percent_to_numbers, round((Percent / 100) * Denominator, 3)) to 2010–2014 (percent_years = ['2014', '2013', '2012', '2011', '2010']), while preserving direct count observations reported in 2015–2024 across all 11 geographic summary levels.
    • Adds CLI entrypoint (--option, --table_prefix, --spec_path, --input_path, --output_dir, --has_percent, --debug) to run column-map generation and observation processing directly from S2303/.
  • scripts/us_census/acs5yr/subject_tables/S2303/process_test.py:
    • Enables both package and directory-scoped unittest discover execution.
  • scripts/us_census/acs5yr/subject_tables/S2303/README.md:
    • Documents the 2010–2024 coverage period, 11 supported geographic levels, census_api_data_downloader.py download commands, and 5 3-year batch processing commands (2010_2012 through 2022_2024).

Testing & Verification

  • Ran unit tests: ./run_tests.sh -p scripts/us_census/acs5yr/subject_tables/S2303 (Passed).
  • Verified code formatting with yapf --diff --style=google.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the US Census ACS5YR S2303 Subject Table import pipeline, introducing a generic data_loader.py module and refactoring process.py to use command-line flags. Feedback on these changes highlights several critical improvements: fixing a file extension comparison bug (comparing against 'zip' instead of '.zip'), preventing premature output generation during directory processing to avoid schema corruption, using 'with' statements for safe file handling, and ensuring pipeline scripts raise exceptions or exit with non-zero status codes on failures.

Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/data_loader.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/__init__.py Outdated
@balit-raibot
balit-raibot self-requested a review October 19, 2026 05:37

@SandeepTuniki SandeepTuniki left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! The 2010–2014 percentage-to-count conversion gating (percent_years) and the 3-year batch processing documentation in README.md look good.

I left a few inline comments on process.py regarding in-place column_map mutation in _process_dataframe, per-year unique geo/StatVar counter tracking, and scoping column_map.json by table_prefix for parallel batch runs.

Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py Outdated
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py
Comment thread scripts/us_census/acs5yr/subject_tables/S2303/process.py

@SandeepTuniki SandeepTuniki left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@goutamabrol
goutamabrol merged commit 7ef60af into datacommonsorg:master Oct 22, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants