Repository navigation
new import japan census - #2295
Open
goutamabrol wants to merge 12 commits into
Open
goutamabrol wants to merge 12 commits into
goutamabrol wants to merge 12 commits into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces an automated import pipeline for Japan Census and Demographic Statistics, including scripts for downloading data from the e-Stat API and a parallel processing wrapper to merge the resulting datasets. The feedback focuses on improving process.py by replacing ProcessPoolExecutor with ThreadPoolExecutor for I/O-bound subprocess tasks, adding explicit checks for empty input directories to raise a FileNotFoundError, and cleaning up standard library imports by moving tempfile to the top of the file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
Japan_Censusimport understatvar_imports/japan_census/for Japan Population Census, Employment, Education, Mortality, and Health Statistics from the Official Statistics of Japan (e-Stat).https://dashboard.e-stat.go.jp/api/1.0/Json/getData), covering National (country/JPN), 47 Prefectures (AdministrativeArea1), and 1,917 Municipalities (AdministrativeArea2) across census waves (1920–2025) and annual statistical releases (1948–2026).+27,760new2022–2026observations added,0deleted,0modified vs. previousJapan_Censusimport).download_script.py,process.py, & unit tests): Downloads all 9 dataset groups in parallel from the e-Stat JSON API, derives broad age-group totals (<15,15–64,65+,85+), merges pre-1975/archival historical records fromgs://unresolved_mcf/country/japan/deleted_historical_data.csvwith atomic file writes into 18 input CSVs, and processes shards concurrently viaprocess.py.japan_census_pv_map.csv) and metadata config (japan_census_metadata.csv) mappingyear,indicatorCd,regionCd, andvalueacross all 1,965 places (country/JPNandwikidataId/Q...), scaling labor force counts (#Multiply,10) to match existing scaling factors, and filtering non-target regional aggregate rows via#IgnoreRow.manifest.json): Configured for Cloud Batch execution withdownload_script.py,process.py(--existing_statvar_mcf,--output_counters,node_mcf), and automated monthly cron schedule (0 5 1 * *).validation_config.json): Enforces historical deletion threshold of 0.1% (DELETED_RECORDS_PERCENT) and date freshness/value bounds (SQL_VALIDATOR) across all 92 active recurring StatVars and 9 historical centenarian sub-brackets (100–104,105–109,110+).Cloud Batch Verified: End-to-end job completed with status
SUCCEEDED.PR Checklist: link
Cloud Batch Job (test) - link
Validation: link