Repository navigation
CensusACS5YearSurvey_SubjectTables_S2303 - #2287
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the US Census ACS5YR S2303 Subject Table import pipeline, introducing a generic data_loader.py module and refactoring process.py to use command-line flags. Feedback on these changes highlights several critical improvements: fixing a file extension comparison bug (comparing against 'zip' instead of '.zip'), preventing premature output generation during directory processing to avoid schema corruption, using 'with' statements for safe file handling, and ensuring pipeline scripts raise exceptions or exit with non-zero status codes on failures.
This reverts commit 1a0c4bd.
…fix test discovery
SandeepTuniki
left a comment
There was a problem hiding this comment.
Thanks for the changes! The 2010–2014 percentage-to-count conversion gating (percent_years) and the 3-year batch processing documentation in README.md look good.
I left a few inline comments on process.py regarding in-place column_map mutation in _process_dataframe, per-year unique geo/StatVar counter tracking, and scoping column_map.json by table_prefix for parallel batch runs.
Summary
Updates the processing scripts and documentation for the US Census ACS 5-Year Subject Table S2303 (Work Status in the Past 12 Months) to support the 2010–2024 dataset refresh across all 11 supported geographic summary levels.
Key Changes
scripts/us_census/acs5yr/subject_tables/S2303/process.py:SubjectTableDataLoaderBasefrom../common/data_loader.py(S2303SubjectTableDataLoader) to restrict percentage-to-count conversion (_convert_percent_to_numbers,round((Percent / 100) * Denominator, 3)) to2010–2014(percent_years = ['2014', '2013', '2012', '2011', '2010']), while preserving direct count observations reported in2015–2024across all 11 geographic summary levels.--option,--table_prefix,--spec_path,--input_path,--output_dir,--has_percent,--debug) to run column-map generation and observation processing directly fromS2303/.scripts/us_census/acs5yr/subject_tables/S2303/process_test.py:unittest discoverexecution.scripts/us_census/acs5yr/subject_tables/S2303/README.md:census_api_data_downloader.pydownload commands, and 5 3-year batch processing commands (2010_2012through2022_2024).Testing & Verification
./run_tests.sh -p scripts/us_census/acs5yr/subject_tables/S2303(Passed).yapf --diff --style=google.