Skip to content

new import japan census - #2295

Open
goutamabrol wants to merge 12 commits into
datacommonsorg:masterfrom
goutamabrol:japan_census
Open

goutamabrol wants to merge 12 commits into
datacommonsorg:masterfrom
goutamabrol:japan_census

Conversation

@goutamabrol

@goutamabrol goutamabrol commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Adds the Japan_Census import under statvar_imports/japan_census/ for Japan Population Census, Employment, Education, Mortality, and Health Statistics from the Official Statistics of Japan (e-Stat).

  • Data Source: Portal Site of Official Statistics of Japan (e-Stat Statistical Dashboard JSON API https://dashboard.e-stat.go.jp/api/1.0/Json/getData), covering National (country/JPN), 47 Prefectures (AdministrativeArea1), and 1,917 Municipalities (AdministrativeArea2) across census waves (1920–2025) and annual statistical releases (1948–2026).
  • Coverage: Maps 101 canonical StatVars across demographics (total population and 5-year/broad age brackets by gender), labor force (employment, unemployment, not in labor force, unemployment rate), education (students and teachers by school level), and vital/health statistics (deaths by gender and cause of death), generating 296,108 observations from 415,244 raw records (+27,760 new 2022–2026 observations added, 0 deleted, 0 modified vs. previous Japan_Census import).
  • Downloader & Processor (download_script.py, process.py, & unit tests): Downloads all 9 dataset groups in parallel from the e-Stat JSON API, derives broad age-group totals (<15, 15–64, 65+, 85+), merges pre-1975/archival historical records from gs://unresolved_mcf/country/japan/deleted_historical_data.csv with atomic file writes into 18 input CSVs, and processes shards concurrently via process.py.
  • Mappings: Unified PV map (japan_census_pv_map.csv) and metadata config (japan_census_metadata.csv) mapping year, indicatorCd, regionCd, and value across all 1,965 places (country/JPN and wikidataId/Q...), scaling labor force counts (#Multiply,10) to match existing scaling factors, and filtering non-target regional aggregate rows via #IgnoreRow.
  • Automation (manifest.json): Configured for Cloud Batch execution with download_script.py, process.py (--existing_statvar_mcf, --output_counters, node_mcf), and automated monthly cron schedule (0 5 1 * *).
  • Validation (validation_config.json): Enforces historical deletion threshold of 0.1% (DELETED_RECORDS_PERCENT) and date freshness/value bounds (SQL_VALIDATOR) across all 92 active recurring StatVars and 9 historical centenarian sub-brackets (100–104, 105–109, 110+).

Cloud Batch Verified: End-to-end job completed with status SUCCEEDED.

PR Checklist: link

Cloud Batch Job (test) - link

Validation: link

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an automated import pipeline for Japan Census and Demographic Statistics, including scripts for downloading data from the e-Stat API and a parallel processing wrapper to merge the resulting datasets. The feedback focuses on improving process.py by replacing ProcessPoolExecutor with ThreadPoolExecutor for I/O-bound subprocess tasks, adding explicit checks for empty input directories to raise a FileNotFoundError, and cleaning up standard library imports by moving tempfile to the top of the file.

Comment thread statvar_imports/japan_census/process.py Outdated
Comment thread statvar_imports/japan_census/process.py
Comment thread statvar_imports/japan_census/process.py
Comment thread statvar_imports/japan_census/process.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant