From 4ff65dbd60ddee80f8c0933fc9935eee78803891 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Thu, 22 Oct 2026 06:47:58 +0000 Subject: [PATCH 01/17] new import japan census --- statvar_imports/japan_census/README.md | 98 +++ .../japan_census/download_script.py | 617 ++++++++++++++++++ .../japan_census/download_script_test.py | 218 +++++++ statvar_imports/japan_census/manifest.json | 35 + statvar_imports/japan_census/process.py | 309 +++++++++ statvar_imports/japan_census/process_test.py | 147 +++++ .../test_data/sample_expected_output.csv | 19 + .../japan_census/test_data/sample_input.csv | 19 + .../japan_census/validation_config.json | 32 + 9 files changed, 1494 insertions(+) create mode 100644 statvar_imports/japan_census/README.md create mode 100644 statvar_imports/japan_census/download_script.py create mode 100644 statvar_imports/japan_census/download_script_test.py create mode 100644 statvar_imports/japan_census/manifest.json create mode 100644 statvar_imports/japan_census/process.py create mode 100644 statvar_imports/japan_census/process_test.py create mode 100644 statvar_imports/japan_census/test_data/sample_expected_output.csv create mode 100644 statvar_imports/japan_census/test_data/sample_input.csv create mode 100644 statvar_imports/japan_census/validation_config.json diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md new file mode 100644 index 0000000000..0ef888246d --- /dev/null +++ b/statvar_imports/japan_census/README.md @@ -0,0 +1,98 @@ +# Japan Census and Demographic Statistics Data + +## 1. Import Overview + +This project downloads, processes, and imports demographic, employment, education, health, and mortality statistics for Japan across National (`country/JPN`), Prefecture (`AdministrativeArea1`), and Municipality (`AdministrativeArea2`) geographic levels, provided by the Official Statistics of Japan (e-Stat Statistical Dashboard). + +* **Source URL**: [https://dashboard.e-stat.go.jp/en/](https://dashboard.e-stat.go.jp/en/) +* **API Endpoint**: `https://dashboard.e-stat.go.jp/api/1.0/Json/getData` +* **Import Type**: Automated +* **Source Data Availability**: + * **Population Census (National & Prefecture)**: 5-year census waves (`1920`–`2025`). + * **Population Census (Municipality)**: 5-year census waves (`1980`–`2025`). + * **Employment & Unemployment Rate (National)**: Annual (`1953`–`2025`). + * **Student Enrollment (National)**: Annual (`1948`–`2026`). + * **Teachers (National & Prefecture)**: Annual (`1975`–`2024`). + * **Deaths & Causes of Death (National & Prefecture)**: Annual (`1975`–`2025`). +* **Release Frequency**: Annual (vital, labor, and education statistics) and quinquennial (5-year Population Census). +* **Notes**: This dataset covers 101 StatisticalVariables across 1,965 places (`country/JPN`, 47 Prefectures, and 1,917 Municipalities). + +--- + +## 2. Preprocessing Steps + +The import pipeline queries the e-Stat Statistical Dashboard JSON API in parallel, merges static pre-1975/archival census records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv` so no historical observations are lost, and processes all 18 input files in parallel via `process.py`. + +* **Files**: + * `download_script.py`: Queries the e-Stat Statistical Dashboard API (`https://dashboard.e-stat.go.jp/api/1.0/Json/getData`), computes exact broad age-group sums (`<15`, `15–64`, `65+`, `85+`) from 5-year census brackets, merges historical records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv` (`--historical_gcs_path`), and writes 18 standardized CSV files into `input_files/`. + * `process.py`: Multi-process wrapper around `stat_var_processor.py` that processes all `input_files/*.csv` shards in parallel (`--parallelism=2`) and deterministically merges the resulting `.csv`, `.tmcf`, `.mcf`, and `counters.csv` outputs into `output_files/`. + * `japan_census_metadata.csv`: Configuration file for `stat_var_processor.py` specifying output columns, header rows, mapped columns, and provenance URL. + * `japan_census_pv_map.csv`: Property-value mapping file mapping e-Stat indicator codes (`indicatorCd`) and 5-digit region codes (`regionCd`) to Data Commons StatisticalVariables and place DCIDs (`country/JPN` and `wikidataId/Q...`). + * `validation_config.json`: Configuration defining historical deletion (`<= 0.1%`), value bounds, and date freshness validation rules across all 101 StatisticalVariables. + * `test_data/`: Sample input data (`sample_input.csv`) and expected output (`sample_expected_output.csv`) for testing. + +* **Transformation pipeline**: + 1. `download_script.py` fetches all 9 dataset groups from the e-Stat JSON API using concurrent workers and atomic file writes into `input_files/`. + 2. `process.py` invokes `stat_var_processor.py` in parallel across the 18 input CSVs against `japan_census_pv_map.csv` and `japan_census_metadata.csv`, referencing canonical schema `gs://unresolved_mcf/scripts/statvar/stat_vars.mcf`. + 3. The processor filters non-target regional aggregate rows via `#IgnoreRow`, scales labor force counts reported in units of 10,000 persons to thousands via `#Multiply,10`, resolves place DCIDs, and produces `output_files/japan_census_output.csv` and `output_files/japan_census_output.tmcf`. + 4. Aggregated processor counters are written to `counters/japan_census_counters.csv`. + +* **Data Quality Checks**: + * Automated validation via `tools/import_validation/runner.py` evaluates the output against `validation_config.json`, enforcing: + * `check_deleted_records_percent`: Historical deletion threshold `<= 0.1%` (`0` deleted records). + * `check_max_date_freshness`: Verifies `MaxDate`, `NumPlaces >= 1`, `MinValue >= 0`, and rate bounds (`MaxValue <= 100` for `UnemploymentRate_Person`) across all 92 active recurring indicators. + * `check_max_date_historical_indicators`: Verifies `MaxDate >= 2020` and `MinValue >= 0` across the 9 discontinued centenarian sub-brackets (`100–104`, `105–109`, `110+`). + +--- + +## 3. Automated Import + +This import is configured for automated monthly execution in `manifest.json`. + +### Automated Steps +1. The automated job triggers on cron schedule `0 5 1 * *` configured in `manifest.json`. +2. `download_script.py` queries the e-Stat Dashboard API and populates `input_files/*.csv`. +3. `process.py` runs `stat_var_processor.py` with `--parallelism=2` to generate `output_files/japan_census_output.csv` and `output_files/japan_census_output.tmcf`. +4. Validation rules in `validation_config.json` are evaluated before publication. + +--- + +## 4. Script Execution Details + +To run the import pipeline locally from `statvar_imports/japan_census/`: + +### Download the Data + +```bash +python3 download_script.py +``` + +### Process the Data + +```bash +python3 process.py \ + --input_data=input_files/*.csv \ + --pv_map=japan_census_pv_map.csv \ + --config_file=japan_census_metadata.csv \ + --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \ + --output_path=output_files/japan_census_output \ + --output_counters=counters/japan_census_counters.csv \ + --parallelism=2 +``` + +### Run Sample Integration Test + +```bash +python3 ../../tools/statvar_importer/stat_var_processor.py \ + --input_data=test_data/sample_input.csv \ + --pv_map=japan_census_pv_map.csv \ + --config_file=japan_census_metadata.csv \ + --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \ + --output_path=test_data/sample_expected_output +``` + +### Run Unit Tests + +```bash +python3 -m unittest discover -s . -p "*_test.py" -v +``` diff --git a/statvar_imports/japan_census/download_script.py b/statvar_imports/japan_census/download_script.py new file mode 100644 index 0000000000..bd1671e9b6 --- /dev/null +++ b/statvar_imports/japan_census/download_script.py @@ -0,0 +1,617 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Download script for Japan Census and socio-economic datasets from e-Stat Dashboard API.""" + +import concurrent.futures +import csv +import os +import sys +import tempfile +from typing import Any + +_MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(1, os.path.join(_MODULE_DIR, '../../util/')) + +from absl import app +from absl import flags +from absl import logging +import download_util +import file_util + +_FLAGS = flags.FLAGS +flags.DEFINE_string( + 'output_dir', + os.path.join(_MODULE_DIR, 'input_files'), + 'Directory path to save the downloaded CSV files.', +) +flags.DEFINE_integer( + 'max_workers', + 8, + 'Maximum number of parallel threads for e-Stat API requests.', +) +flags.DEFINE_string( + 'historical_gcs_path', + 'gs://unresolved_mcf/country/japan/deleted_historical_data.csv', + 'GCS path to the deleted historical data CSV file.', +) + +ESTAT_JSON_API_URL = 'https://dashboard.e-stat.go.jp/api/1.0/Json/getData' + +# 5-year age bracket suffixes: 010 (0-4 years) through 200 (95-99 years) +_AGE_5YR_SUFFIXES = [f'{i:03d}' for i in range(10, 210, 10)] + +# Leading 3 columns (year, indicatorCd, regionCd) are mapped_columns=3 for stat_var_processor. +CSV_HEADER = [ + 'year', + 'indicatorCd', + 'regionCd', + 'value', + 'indicatorName', + 'unitCd', + 'unitName', + 'statCd', + 'regionName', + 'timeCd', + 'cycle', + 'regionRank', + 'isSeasonal', + 'isProvisional', +] + + +def _build_population_5yr_indicators( + gender_digits: tuple[str, ...] = ('0', '1', '2'), + include_totals: bool = True, +) -> list[str]: + """Builds e-Stat indicator codes for population totals and 5-year age brackets.""" + indicators: list[str] = [] + if include_totals: + for g in gender_digits: + indicators.append(f'0201010{g}00000010000') + for g in gender_digits: + for suffix in _AGE_5YR_SUFFIXES: + indicators.append(f'0201010{g}20000010{suffix}') + indicators.append(f'0201010{g}30000011000') + return indicators + + +# Dataset configurations mapping output CSV filename to query parameters. +# RegionalRank: '2' = Country (Japan), '3' = Prefecture, '4' = Municipality. +# Cycle: '3' = Calendar Year. +DATASET_CONFIGS: dict[str, dict[str, Any]] = { + 'jpn_employement.csv': { + 'indicators': [ + '0301010000010010010', # Employed persons (Both sexes) + '0301010000020010010', # Unemployed persons (Both sexes) + '0301010000030010010', # Not in labour force (Both sexes) + ], + 'regional_ranks': ['2'], + 'batch_size': 3, + 'is_census': False, + }, + 'jpn_unemploymentrate.csv': { + 'indicators': [ + '0301010000020020010', # Unemployment rate (Both sexes) + ], + 'regional_ranks': ['2'], + 'batch_size': 1, + 'is_census': False, + }, + 'jpn_student.csv': { + 'indicators': [ + '1201050300000010000', # Elementary school students + '1201050400000010000', # Lower secondary school students + '1201050500000010000', # Upper secondary school students + '1201050800000010000', # University students and graduate school students + ], + 'regional_ranks': ['2'], + 'batch_size': 4, + 'is_census': False, + }, + 'jpn_aa1_teacher.csv': { + 'indicators': [ + '1201040100000010000', # Teachers of kindergartens + '1201040300000010000', # Teachers of elementary schools + '1201040500000010000', # Teachers of senior high schools (mapped to Count_Teacher_UniversityAndCollegeTeacher) + ], + 'regional_ranks': ['2', '3'], + 'batch_size': 3, + 'is_census': False, + }, + 'jpn_aa1_health.csv': { + 'indicators': [ + '1505130000000010030', # Deaths by causes (diabetes mellitus) + '1505130000000010050', # Deaths by causes (heart diseases excl. hypertensive) + '1505130000000010070', # Deaths by causes (suicide) + ], + 'regional_ranks': ['2', '3'], + 'batch_size': 3, + 'is_census': False, + }, + 'jpn_aa1_death.csv': { + 'indicators': [ + '0203020000000010000', # Deaths (Both sexes) + '0203020100000010000', # Deaths (Male) + '0203020200000010000', # Deaths (Female) + ], + 'regional_ranks': ['2', '3'], + 'batch_size': 3, + 'is_census': False, + }, + 'jpn_gender_age_total_population.csv': { + 'indicators': _build_population_5yr_indicators(), + 'regional_ranks': ['2'], + 'batch_size': 5, + 'is_census': True, + 'census_mode': 'national', + }, + 'jpn_aa1_gender_age_total_population.csv': { + 'indicators': _build_population_5yr_indicators(), + 'regional_ranks': ['3'], + 'batch_size': 5, + 'is_census': True, + 'census_mode': 'prefecture', + }, + 'jpn_aa1_aa2_age_gender.csv': { + 'indicators': _build_population_5yr_indicators(), + 'regional_ranks': ['4'], + 'batch_size': 3, + 'is_census': True, + 'census_mode': 'municipality', + }, +} + + +def _extract_class_lookup( + class_objs: list[dict[str, Any]] | dict[str, Any], target_id: str +) -> dict[str, str]: + """Extracts a code-to-name dictionary from e-Stat CLASS_INF.CLASS_OBJ.""" + if isinstance(class_objs, dict): + class_objs = [class_objs] + for obj in class_objs: + if obj.get('@id') == target_id: + classes = obj.get('CLASS', []) + if isinstance(classes, dict): + classes = [classes] + return { + str(c.get('@code', '')): str( + c.get('@name') if '@name' in c else c.get('$', '') + ) + for c in classes + if isinstance(c, dict) + } + return {} + + +def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: + """Parses an e-Stat Dashboard API JSON response and extracts data rows with metadata. + + Args: + data: Parsed JSON dictionary returned by the e-Stat Dashboard API. + + Returns: + List of row dictionaries keyed by CSV_HEADER fields. + + Raises: + RuntimeError: If the API returned an error status code or invalid payload. + """ + if not isinstance(data, dict) or 'GET_STATS' not in data: + raise RuntimeError('Invalid JSON response from e-Stat API: missing GET_STATS.') + + get_stats = data['GET_STATS'] + result = get_stats.get('RESULT', {}) + status_code = str(result.get('status', '')) + error_msg = str(result.get('errorMsg', '')) + + if status_code not in ('0', '100'): + raise RuntimeError( + f'e-Stat API error (status={status_code}): {error_msg}' + ) + + if status_code == '100': + return [] + + stat_data = get_stats.get('STATISTICAL_DATA', {}) + class_objs = stat_data.get('CLASS_INF', {}).get('CLASS_OBJ', []) + + indicator_map = _extract_class_lookup(class_objs, 'indicator') + unit_map = _extract_class_lookup(class_objs, 'unit') + region_map = _extract_class_lookup(class_objs, 'regionCode') + time_map = _extract_class_lookup(class_objs, 'time') + + data_objs = stat_data.get('DATA_INF', {}).get('DATA_OBJ', []) + if isinstance(data_objs, dict): + data_objs = [data_objs] + + parsed_records: list[dict[str, str]] = [] + for obj in data_objs: + val_obj = obj.get('VALUE', {}) + if not isinstance(val_obj, dict): + continue + ind_cd = str(val_obj.get('@indicator', '')) + unit_cd = str(val_obj.get('@unit', '')) + reg_cd = str(val_obj.get('@regionCode', '')) + time_cd = str(val_obj.get('@time', '')) + year_str = time_map.get( + time_cd, time_cd[:4] if len(time_cd) >= 4 else time_cd + ) + + parsed_records.append({ + 'year': year_str, + 'indicatorCd': ind_cd, + 'regionCd': reg_cd, + 'value': str(val_obj.get('$', '')), + 'indicatorName': indicator_map.get(ind_cd, ''), + 'unitCd': unit_cd, + 'unitName': unit_map.get(unit_cd, ''), + 'statCd': str(val_obj.get('@stat', '')), + 'regionName': region_map.get(reg_cd, ''), + 'timeCd': time_cd, + 'cycle': str(val_obj.get('@cycle', '')), + 'regionRank': str(val_obj.get('@regionRank', '')), + 'isSeasonal': str(val_obj.get('@isSeasonal', '')), + 'isProvisional': str(val_obj.get('@isProvisional', '')), + }) + + return parsed_records + + +def fetch_indicator_batch( + indicators: list[str], + regional_rank: str, + cycle: str = '3', + is_seasonal: str = '1', + time_from: str = '', + time_to: str = '', +) -> list[dict[str, str]]: + """Fetches a batch of indicators for a given regional rank from the e-Stat JSON API.""" + params: dict[str, str] = { + 'Lang': 'EN', + 'IndicatorCode': ','.join(indicators), + 'RegionalRank': regional_rank, + 'Cycle': cycle, + 'IsSeasonalAdjustment': is_seasonal, + 'MetaGetFlg': 'Y', + 'SectionHeaderFlg': '1', + } + if time_from: + params['TimeFrom'] = time_from + if time_to: + params['TimeTo'] = time_to + + logging.info( + 'Requesting e-Stat JSON API: RegionalRank=%s, Indicators=%d (%s...)', + regional_rank, + len(indicators), + indicators[0], + ) + response_json = download_util.request_url( + url=ESTAT_JSON_API_URL, + params=params, + method='GET', + output='json', + timeout=60, + retries=3, + retry_secs=5, + ) + if not response_json or not isinstance(response_json, dict): + raise RuntimeError( + f'Invalid or empty JSON response from {ESTAT_JSON_API_URL} for params={params}' + ) + return parse_estat_json_response(response_json) + + +def _derive_census_population_rows( + records: list[dict[str, str]], mode: str +) -> list[dict[str, str]]: + """Filters to 5-year census years and computes exact unrounded broad age sums. + + At National (RegionalRank=2) and Prefecture (RegionalRank=3) levels, direct + broad-age indicators (0201010*100000100*) report rounded annual Population + Estimates (in thousands), whereas summing the 5-year Census age brackets + (0201010*20000010* + 0201010*30000011000) yields the exact unrounded Population + Census counts for <15, 15-64, 65+, and 85+. + """ + census_records = [ + r for r in records if r['year'].isdigit() and int(r['year']) % 5 == 0 + ] + + by_reg_yr_ind: dict[tuple[str, str, str], dict[str, str]] = {} + reg_meta: dict[tuple[str, str], dict[str, str]] = {} + for r in census_records: + key = (r['regionCd'], r['year'], r['indicatorCd']) + by_reg_yr_ind[key] = r + reg_meta[(r['regionCd'], r['year'])] = r + + output_rows: list[dict[str, str]] = [] + # Keep total population (Both sexes, Male, Female) and 5-year brackets: + # National keeps 010..230 (0-4 through 110+); Prefecture keeps 010..170 (0-4 through 80-84); + # Municipality keeps only the 3 totals + 9 broad age sums (<15, 15-64, 65+ for Both/Male/Female = 12 StatVars). + if mode == 'municipality': + allowed_5yr: set[str] = set() + else: + max_5yr_suffix = 230 if mode == 'national' else 170 + allowed_5yr = { + f'0201010{g}20000010{i:03d}' + for g in ('0', '1', '2') + for i in range(10, max_5yr_suffix + 10, 10) + } + allowed_totals = { + '0201010000000010000', + '0201010100000010000', + '0201010200000010000', + } + + for r in census_records: + ind = r['indicatorCd'] + if ind in allowed_totals or ind in allowed_5yr: + output_rows.append(r) + + # Derive exact Census counts for <15, 15-64, 65+, and (for National/Prefecture) 85+ from 5-year brackets. + broad_Specs = [ + ('10000010010', 'Population (under 15)', [f'{i:03d}' for i in (10, 20, 30)], False), + ('10000010020', 'Population (15-64)', [f'{i:03d}' for i in range(40, 140, 10)], False), + ('10000010030', 'Population (65 and over)', [f'{i:03d}' for i in range(140, 210, 10)], True), + ] + if mode != 'municipality': + broad_Specs.append( + ('40000010070', 'Population (85 and over)', [f'{i:03d}' for i in (180, 190, 200)], True) + ) + + for reg_cd, yr in sorted(reg_meta.keys()): + base = reg_meta[(reg_cd, yr)] + for g in ('0', '1', '2'): + for target_tail, desc, suffixes, include_100plus in broad_Specs: + part_codes = [f'0201010{g}20000010{s}' for s in suffixes] + if include_100plus: + part_codes.append(f'0201010{g}30000011000') + vals: list[int] = [] + valid = True + for c in part_codes: + rec = by_reg_yr_ind.get((reg_cd, yr, c)) + if not rec or not rec['value'].lstrip('-').isdigit(): + valid = False + break + vals.append(int(rec['value'])) + if valid: + new_row = dict(base) + new_row['indicatorCd'] = f'0201010{g}{target_tail}' + new_row['indicatorName'] = desc + new_row['value'] = str(sum(vals)) + output_rows.append(new_row) + + return output_rows + + +_HISTORICAL_CACHE: dict[str, list[dict[str, str]]] = {} + + +def _load_historical_rows(historical_gcs_path: str) -> list[dict[str, str]]: + """Loads and caches historical rows from GCS or local CSV.""" + if not historical_gcs_path: + return [] + if historical_gcs_path in _HISTORICAL_CACHE: + return _HISTORICAL_CACHE[historical_gcs_path] + try: + logging.info( + 'Reading historical deleted data from GCS: %s', historical_gcs_path + ) + with file_util.FileIO(historical_gcs_path, 'r', encoding='utf-8') as f: + rows = list(csv.DictReader(f)) + _HISTORICAL_CACHE[historical_gcs_path] = rows + return rows + except Exception as e: + logging.warning( + 'Could not read historical deleted data from %s: %s. Proceeding with fresh data only.', + historical_gcs_path, + e, + ) + _HISTORICAL_CACHE[historical_gcs_path] = [] + return [] + + +def _apply_historical_supplement( + dataset_filename: str, + records: list[dict[str, str]], + historical_gcs_path: str = '', +) -> list[dict[str, str]]: + """Merges pre-1975 historical Census and exact archival rows from GCS CSV.""" + hist_rows = _load_historical_rows(historical_gcs_path) + if not hist_rows: + return records + + by_key: dict[tuple[str, str, str], dict[str, str]] = {} + for r in records: + by_key[(r['regionCd'], r['year'], r['indicatorCd'])] = r + + added_or_updated = 0 + for s in hist_rows: + if s['dataset'] != dataset_filename: + continue + key = (s['regionCd'], s['year'], s['indicatorCd']) + if key in by_key: + by_key[key]['value'] = s['value'] + else: + by_key[key] = { + 'year': s['year'], + 'indicatorCd': s['indicatorCd'], + 'regionCd': s['regionCd'], + 'value': s['value'], + 'indicatorName': '', + 'unitCd': '', + 'unitName': '', + 'statCd': '', + 'regionName': '', + 'timeCd': f"{s['year']}CY00", + 'cycle': '3', + 'regionRank': '2' if s['regionCd'] == '00000' else '3', + 'isSeasonal': '1', + 'isProvisional': '0', + } + added_or_updated += 1 + + if added_or_updated: + logging.info( + 'Applied %d historical supplement rows to %s', + added_or_updated, + dataset_filename, + ) + return list(by_key.values()) + + +def _write_csv_atomically(output_filepath: str, rows: list[dict[str, str]]) -> None: + """Writes rows atomically to output_filepath and verifies non-empty output.""" + output_dir = os.path.dirname(output_filepath) + os.makedirs(output_dir, exist_ok=True) + fd, tmp_path = tempfile.mkstemp( + prefix='.tmp_jpn_', suffix='.csv', dir=output_dir + ) + try: + with os.fdopen(fd, 'w', encoding='utf-8', newline='') as f: + writer = csv.DictWriter(f, fieldnames=CSV_HEADER, lineterminator='\n') + writer.writeheader() + writer.writerows(rows) + if not (os.path.exists(tmp_path) and os.path.getsize(tmp_path) > 0): + raise RuntimeError( + f'Atomic write failed: {tmp_path} is empty or missing.' + ) + os.replace(tmp_path, output_filepath) + except Exception: + if os.path.exists(tmp_path): + os.remove(tmp_path) + raise + + +def download_dataset( + output_filepath: str, + indicators: list[str], + regional_ranks: list[str], + batch_size: int, + is_census: bool = False, + census_mode: str = '', + max_workers: int = 1, + historical_gcs_path: str = '', +) -> int: + """Downloads all batches for a dataset and writes atomically to output_filepath.""" + tasks: list[tuple[list[str], str]] = [] + for rank in regional_ranks: + for i in range(0, len(indicators), batch_size): + tasks.append((indicators[i : i + batch_size], rank)) + + all_records: list[dict[str, str]] = [] + if max_workers > 1 and len(tasks) > 1: + with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: + futures = [ + executor.submit(fetch_indicator_batch, batch, rank) + for batch, rank in tasks + ] + for fut in futures: + all_records.extend(fut.result()) + else: + for batch, rank in tasks: + all_records.extend( + fetch_indicator_batch( + indicators=batch, + regional_rank=rank, + ) + ) + + if is_census: + all_records = _derive_census_population_rows(all_records, census_mode) + + all_records = _apply_historical_supplement( + os.path.basename(output_filepath), + all_records, + historical_gcs_path=historical_gcs_path, + ) + + if not all_records: + raise RuntimeError( + f'No records retrieved for dataset {os.path.basename(output_filepath)}' + ) + + if census_mode == 'municipality': + # Split municipality census by year (matching CNS jpn_aa1_aa2_age_gender_{year}.csv) + # so multi-core processing can run across all census years in parallel. + output_dir = os.path.dirname(output_filepath) + by_year: dict[str, list[dict[str, str]]] = {} + for r in all_records: + by_year.setdefault(r['year'], []).append(r) + # Remove any old unsplit jpn_aa1_aa2_age_gender.csv if present + if os.path.exists(output_filepath): + os.remove(output_filepath) + for yr, yr_rows in sorted(by_year.items()): + yr_filepath = os.path.join(output_dir, f'jpn_aa1_aa2_age_gender_{yr}.csv') + _write_csv_atomically(yr_filepath, yr_rows) + logging.info('Saved %d rows to %s', len(yr_rows), yr_filepath) + return len(all_records) + + _write_csv_atomically(output_filepath, all_records) + logging.info( + 'Saved %d rows to %s', len(all_records), output_filepath + ) + return len(all_records) + + +def download_all( + output_dir: str, + max_workers: int = 8, + historical_gcs_path: str = '', +) -> None: + """Downloads all configured Japan Census datasets into output_dir.""" + os.makedirs(output_dir, exist_ok=True) + total_rows = 0 + for filename, cfg in DATASET_CONFIGS.items(): + output_filepath = os.path.join(output_dir, filename) + logging.info('Downloading dataset: %s', filename) + count = download_dataset( + output_filepath=output_filepath, + indicators=cfg['indicators'], + regional_ranks=cfg['regional_ranks'], + batch_size=cfg['batch_size'], + is_census=cfg.get('is_census', False), + census_mode=cfg.get('census_mode', ''), + max_workers=max_workers, + historical_gcs_path=historical_gcs_path, + ) + total_rows += count + logging.info( + 'Successfully downloaded all %d datasets (%d total rows) to %s', + len(DATASET_CONFIGS), + total_rows, + output_dir, + ) + + +def main(argv: list[str]) -> None: + if len(argv) > 1: + raise app.UsageError('Too many command-line arguments.') + output_dir = ( + _FLAGS.output_dir + if os.path.isabs(_FLAGS.output_dir) + else os.path.join(_MODULE_DIR, _FLAGS.output_dir) + ) + try: + download_all( + output_dir, + max_workers=_FLAGS.max_workers, + historical_gcs_path=_FLAGS.historical_gcs_path, + ) + except Exception as e: + logging.fatal('%s', e, exc_info=True) + + +if __name__ == '__main__': + app.run(main) diff --git a/statvar_imports/japan_census/download_script_test.py b/statvar_imports/japan_census/download_script_test.py new file mode 100644 index 0000000000..1969475de6 --- /dev/null +++ b/statvar_imports/japan_census/download_script_test.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Unit tests for Japan Census download_script.py.""" + +import csv +import os +import sys +import tempfile +import unittest +from unittest import mock + +# Ensure repository root and module directory are in sys.path for local and root-level test runs +_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +_REPO_ROOT = os.path.abspath(os.path.join(_SCRIPT_DIR, '../..')) +for _p in (_SCRIPT_DIR, _REPO_ROOT): + if _p not in sys.path: + sys.path.insert(0, _p) + +import download_script + + +class DownloadScriptTest(unittest.TestCase): + + def setUp(self): + super().setUp() + self.sample_api_response = { + 'GET_STATS': { + 'RESULT': { + 'status': '0', + 'errorMsg': 'Success.', + }, + 'STATISTICAL_DATA': { + 'CLASS_INF': { + 'CLASS_OBJ': [ + { + '@id': 'indicator', + 'CLASS': { + '@code': '0301010000020020010', + '@name': 'Unemployment rate (Both sexes)', + }, + }, + { + '@id': 'unit', + 'CLASS': { + '@code': '001', + '$': '%', + }, + }, + { + '@id': 'regionCode', + 'CLASS': { + '@code': '00000', + '@name': 'Japan', + }, + }, + { + '@id': 'time', + 'CLASS': [ + {'@code': '1953CY00', '@name': '1953'}, + {'@code': '1954CY00', '@name': '1954'}, + ], + }, + ] + }, + 'DATA_INF': { + 'DATA_OBJ': [ + { + 'VALUE': { + '@indicator': '0301010000020020010', + '@unit': '001', + '@stat': '00200531', + '@regionCode': '00000', + '@time': '1953CY00', + '@cycle': '3', + '@regionRank': '2', + '@isSeasonal': '1', + '@isProvisional': '0', + '$': '1.9', + } + }, + { + 'VALUE': { + '@indicator': '0301010000020020010', + '@unit': '001', + '@stat': '00200531', + '@regionCode': '00000', + '@time': '1954CY00', + '@cycle': '3', + '@regionRank': '2', + '@isSeasonal': '1', + '@isProvisional': '0', + '$': '2.3', + } + }, + ] + }, + }, + } + } + + def test_parse_estat_json_response_success(self): + records = download_script.parse_estat_json_response( + self.sample_api_response + ) + self.assertEqual(len(records), 2) + self.assertEqual(records[0]['indicatorCd'], '0301010000020020010') + self.assertEqual( + records[0]['indicatorName'], 'Unemployment rate (Both sexes)' + ) + self.assertEqual(records[0]['unitName'], '%') + self.assertEqual(records[0]['regionCd'], '00000') + self.assertEqual(records[0]['regionName'], 'Japan') + self.assertEqual(records[0]['year'], '1953') + self.assertEqual(records[0]['value'], '1.9') + + def test_parse_estat_json_response_error_raises(self): + error_payload = { + 'GET_STATS': { + 'RESULT': { + 'status': '400', + 'errorMsg': 'Too many records.', + } + } + } + with self.assertRaises(RuntimeError): + download_script.parse_estat_json_response(error_payload) + + def test_write_csv_atomically_failure_cleanup(self): + with tempfile.TemporaryDirectory() as tmp_dir: + target_path = os.path.join(tmp_dir, 'test_output.csv') + with mock.patch.object(os.path, 'getsize', return_value=0): + with self.assertRaises(RuntimeError): + download_script._write_csv_atomically( + target_path, [{'year': '2020'}] + ) + self.assertFalse(os.path.exists(target_path)) + self.assertEqual(os.listdir(tmp_dir), []) + + @mock.patch.object(download_script.download_util, 'request_url') + def test_download_dataset_empty_error(self, mock_request_url): + mock_request_url.return_value = { + 'GET_STATS': { + 'RESULT': { + 'status': '100', + 'errorMsg': 'No data.', + } + } + } + with tempfile.TemporaryDirectory() as tmp_dir: + out_file = os.path.join(tmp_dir, 'jpn_unemploymentrate.csv') + with self.assertRaises(RuntimeError): + download_script.download_dataset( + output_filepath=out_file, + indicators=['0301010000020020010'], + regional_ranks=['2'], + batch_size=1, + ) + + @mock.patch.object(download_script.download_util, 'request_url') + def test_download_dataset_writes_csv_atomically(self, mock_request_url): + mock_request_url.return_value = self.sample_api_response + with tempfile.TemporaryDirectory() as tmp_dir: + out_file = os.path.join(tmp_dir, 'jpn_unemploymentrate.csv') + count = download_script.download_dataset( + output_filepath=out_file, + indicators=['0301010000020020010'], + regional_ranks=['2'], + batch_size=1, + ) + self.assertEqual(count, 2) + self.assertTrue(os.path.exists(out_file)) + with open(out_file, 'r', encoding='utf-8') as f: + reader = csv.reader(f) + header = next(reader) + self.assertEqual(header[:4], ['year', 'indicatorCd', 'regionCd', 'value']) + rows = list(reader) + self.assertEqual(len(rows), 2) + self.assertEqual(rows[1][0], '1954') + self.assertEqual(rows[1][3], '2.3') + + def test_apply_historical_supplement_from_path(self): + with tempfile.TemporaryDirectory() as tmp_dir: + hist_csv = os.path.join(tmp_dir, 'deleted_historical_data.csv') + with open(hist_csv, 'w', encoding='utf-8', newline='') as f: + writer = csv.writer(f) + writer.writerow(['dataset', 'year', 'indicatorCd', 'regionCd', 'value']) + writer.writerow([ + 'jpn_aa1_gender_age_total_population.csv', + '1920', + '0201010000000010000', + '01000', + '2359183', + ]) + merged = download_script._apply_historical_supplement( + 'jpn_aa1_gender_age_total_population.csv', + [], + historical_gcs_path=hist_csv, + ) + self.assertEqual(len(merged), 1) + self.assertEqual(merged[0]['year'], '1920') + self.assertEqual(merged[0]['regionCd'], '01000') + self.assertEqual(merged[0]['value'], '2359183') + + +if __name__ == '__main__': + unittest.main() diff --git a/statvar_imports/japan_census/manifest.json b/statvar_imports/japan_census/manifest.json new file mode 100644 index 0000000000..323b5acaeb --- /dev/null +++ b/statvar_imports/japan_census/manifest.json @@ -0,0 +1,35 @@ +{ + "import_specifications": [ + { + "import_name": "Japan_Census", + "curator_emails": [ + "support@datacommons.org" + ], + "provenance_url": "https://dashboard.e-stat.go.jp/en/", + "provenance_description": "Japan Population Census, Employment, Education, Mortality, and Health Statistics from the Portal Site of Official Statistics of Japan (e-Stat).", + "scripts": [ + "download_script.py", + "process.py --input_data=input_files/*.csv --pv_map=japan_census_pv_map.csv --config_file=japan_census_metadata.csv --output_path=output_files/japan_census_output --output_counters=counters/japan_census_counters.csv --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --parallelism=2" + ], + "import_inputs": [ + { + "template_mcf": "output_files/japan_census_output.tmcf", + "cleaned_csv": "output_files/japan_census_output.csv", + "node_mcf": "output_files/*.mcf" + } + ], + "source_files": [ + "input_files/*.csv", + "counters/*.csv", + "validation_config.json" + ], + "cron_schedule": "0 5 1 * *", + "validation_config_file": "validation_config.json", + "resource_limits": { + "cpu": 8, + "memory": 32, + "disk": 100 + } + } + ] +} \ No newline at end of file diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py new file mode 100644 index 0000000000..60766cd56a --- /dev/null +++ b/statvar_imports/japan_census/process.py @@ -0,0 +1,309 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Parallel wrapper for running stat_var_processor.py across multiple input CSVs. + +Spawns concurrent worker processes (one per input CSV file), merges all output +CSVs, MCFs, and counters into the unified output_path, and preserves full +stat_var_processor CLI flag compatibility. +""" + +from collections import defaultdict +import concurrent.futures +import glob +import os +import shutil +import subprocess +import sys + +from absl import app +from absl import flags +from absl import logging + +_MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) +_PROCESSOR_SCRIPT = os.path.abspath( + os.path.join(_MODULE_DIR, '../../tools/statvar_importer/stat_var_processor.py') +) + +_FLAGS = flags.FLAGS +flags.DEFINE_string( + 'input_data', + os.path.join(_MODULE_DIR, 'input_files/*.csv'), + 'Comma-separated list or glob pattern of input CSV files.', +) +flags.DEFINE_string( + 'pv_map', + os.path.join(_MODULE_DIR, 'japan_census_pv_map.csv'), + 'Path to the PV map CSV file.', +) +flags.DEFINE_string( + 'config_file', + os.path.join(_MODULE_DIR, 'japan_census_metadata.csv'), + 'Path to the metadata config CSV file.', +) +flags.DEFINE_string( + 'output_path', + os.path.join(_MODULE_DIR, 'output_files/japan_census_output'), + 'Output prefix path for merged CSV, TMCF, and MCF files.', +) +flags.DEFINE_string( + 'output_counters', + os.path.join(_MODULE_DIR, 'counters/japan_census_counters.csv'), + 'Output path for merged counters CSV file.', +) +flags.DEFINE_string( + 'existing_statvar_mcf', + 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', + 'Path to existing StatVar MCF file.', +) +flags.DEFINE_integer( + 'parallelism', + 2, + 'Maximum number of concurrent stat_var_processor worker processes.', +) + + +def _expand_input_files(input_data: str) -> list[str]: + """Expands comma-separated paths or glob patterns into sorted file paths.""" + files: list[str] = [] + for pattern in input_data.split(','): + pattern = pattern.strip() + if not pattern: + continue + matched = sorted(glob.glob(pattern)) + if matched: + files.extend(matched) + elif os.path.exists(pattern): + files.append(pattern) + # Sort largest files first so long-running shards start immediately. + return sorted(set(files), key=lambda p: -os.path.getsize(p)) + + +def _run_shard( + shard_idx: int, + input_file: str, + shard_output_prefix: str, + shard_counters_path: str, + pv_map: str, + config_file: str, + existing_statvar_mcf: str, +) -> tuple[int, str, str, str]: + """Runs stat_var_processor.py on a single input file in a subprocess.""" + cmd = [ + sys.executable, + _PROCESSOR_SCRIPT, + f'--input_data={input_file}', + f'--pv_map={pv_map}', + f'--config_file={config_file}', + f'--output_path={shard_output_prefix}', + f'--output_counters={shard_counters_path}', + f'--existing_statvar_mcf={existing_statvar_mcf}', + ] + logging.info( + '[Shard %02d] Starting %s', shard_idx, os.path.basename(input_file) + ) + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.returncode != 0: + raise RuntimeError( + f'Shard {shard_idx} ({input_file}) failed with exit code {proc.returncode}:\n' + f'{proc.stderr[-2000:]}' + ) + logging.info( + '[Shard %02d] Finished %s', shard_idx, os.path.basename(input_file) + ) + return shard_idx, input_file, shard_output_prefix, shard_counters_path + + +def _merge_csv_files(shard_prefixes: list[str], final_csv_path: str) -> int: + """Merges shard CSV files into final_csv_path, keeping a single header.""" + total_rows = 0 + wrote_header = False + with open(final_csv_path, 'w', encoding='utf-8', newline='') as out_f: + for prefix in shard_prefixes: + shard_csv = f'{prefix}.csv' + if not os.path.exists(shard_csv): + continue + with open(shard_csv, 'r', encoding='utf-8') as in_f: + header = in_f.readline() + if not header: + continue + if not wrote_header: + out_f.write(header) + wrote_header = True + for line in in_f: + out_f.write(line) + total_rows += 1 + return total_rows + + +def _merge_counters(shard_counter_files: list[str], final_counters_path: str) -> None: + """Aggregates numeric counters across all shards into final_counters_path.""" + if not final_counters_path: + return + os.makedirs(os.path.dirname(os.path.abspath(final_counters_path)), exist_ok=True) + totals: dict[str, float] = defaultdict(float) + is_int_map: dict[str, bool] = defaultdict(lambda: True) + + for cf in shard_counter_files: + if not os.path.exists(cf): + continue + with open(cf, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if not line or line.startswith('key,'): + continue + parts = line.rsplit(',', 1) + if len(parts) != 2: + continue + k, v_str = parts[0].strip('"'), parts[1].strip() + try: + if '.' in v_str: + totals[k] += float(v_str) + is_int_map[k] = False + else: + totals[k] += int(v_str) + except ValueError: + continue + + with open(final_counters_path, 'w', encoding='utf-8') as out_f: + out_f.write('key,value\n') + for k in sorted(totals.keys()): + val = totals[k] + if is_int_map[k]: + out_f.write(f'{k},{int(val)}\n') + else: + out_f.write(f'{k},{val:.2f}\n') + + +def _merge_mcf_files(shard_prefixes: list[str], suffix: str, final_mcf_path: str) -> None: + """Deduplicates MCF nodes by Node line across shards and writes final_mcf_path.""" + seen_nodes: dict[str, str] = {} + for prefix in shard_prefixes: + shard_mcf = f'{prefix}{suffix}' + if not os.path.exists(shard_mcf): + continue + with open(shard_mcf, 'r', encoding='utf-8') as f: + content = f.read() + for block in content.split('\n\n'): + block = block.strip() + if not block: + continue + lines = [l for l in block.splitlines() if not l.strip().startswith('#')] + if not lines: + continue + node_line = next((l for l in lines if l.startswith('Node:')), lines[0]) + if node_line not in seen_nodes: + seen_nodes[node_line] = '\n'.join(lines) + + if os.path.isdir(final_mcf_path): + shutil.rmtree(final_mcf_path) + elif os.path.exists(final_mcf_path): + os.remove(final_mcf_path) + + if not seen_nodes: + return + + with open(final_mcf_path, 'w', encoding='utf-8') as out_f: + for node_key in sorted(seen_nodes.keys()): + out_f.write(seen_nodes[node_key] + '\n\n') + + +def main(_) -> None: + input_files = _expand_input_files(_FLAGS.input_data) + if not input_files: + raise RuntimeError(f'No input files matched: {_FLAGS.input_data}') + + output_path = os.path.abspath(_FLAGS.output_path) + output_dir = os.path.dirname(output_path) + os.makedirs(output_dir, exist_ok=True) + + shards_tmp_dir = tempfile.mkdtemp(prefix='.shards_', dir=output_dir) + max_workers = max( + 1, min(_FLAGS.parallelism, len(input_files), os.cpu_count() or 1) + ) + logging.info( + 'Processing %d input files across %d parallel workers...', + len(input_files), + max_workers, + ) + + try: + shard_prefixes: list[str] = [] + shard_counters: list[str] = [] + futures = [] + + with concurrent.futures.ProcessPoolExecutor( + max_workers=max_workers + ) as executor: + for idx, input_file in enumerate(input_files): + base = os.path.splitext(os.path.basename(input_file))[0] + s_prefix = os.path.join(shards_tmp_dir, f'shard_{idx:02d}_{base}') + s_counters = os.path.join(shards_tmp_dir, f'counters_{idx:02d}_{base}.csv') + shard_prefixes.append(s_prefix) + shard_counters.append(s_counters) + futures.append( + executor.submit( + _run_shard, + idx, + os.path.abspath(input_file), + s_prefix, + s_counters, + os.path.abspath(_FLAGS.pv_map), + os.path.abspath(_FLAGS.config_file), + _FLAGS.existing_statvar_mcf, + ) + ) + + for fut in concurrent.futures.as_completed(futures): + fut.result() + + # 1. Merge CSV outputs + final_csv = f'{output_path}.csv' + total_rows = _merge_csv_files(shard_prefixes, final_csv) + logging.info('Merged %d SVObs rows into %s', total_rows, final_csv) + + # 2. Generate unified TMCF (normalizing table reference to japan_census_output) + final_tmcf = f'{output_path}.tmcf' + table_name = os.path.basename(output_path) + for prefix in shard_prefixes: + shard_tmcf = f'{prefix}.tmcf' + if os.path.exists(shard_tmcf): + shard_table = os.path.basename(prefix) + with open(shard_tmcf, 'r', encoding='utf-8') as f: + tmcf_text = f.read().replace(shard_table, table_name) + with open(final_tmcf, 'w', encoding='utf-8') as out_f: + out_f.write(tmcf_text) + break + + # 3. Merge StatVar MCF and Schema MCF files + _merge_mcf_files(shard_prefixes, '_stat_vars.mcf', f'{output_path}_stat_vars.mcf') + _merge_mcf_files( + shard_prefixes, + '_stat_vars_schema.mcf', + f'{output_path}_stat_vars_schema.mcf', + ) + + # 4. Merge Counters + if _FLAGS.output_counters: + _merge_counters(shard_counters, os.path.abspath(_FLAGS.output_counters)) + logging.info('Merged counters into %s', _FLAGS.output_counters) + + finally: + if os.path.exists(shards_tmp_dir): + shutil.rmtree(shards_tmp_dir) + + +if __name__ == '__main__': + import tempfile + app.run(main) diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py new file mode 100644 index 0000000000..02193c6566 --- /dev/null +++ b/statvar_imports/japan_census/process_test.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Unit tests for Japan Census process.py.""" + +import os +import sys +import tempfile +import unittest +from unittest import mock + +# Ensure repository root and module directory are in sys.path for local and root-level test runs +_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +_REPO_ROOT = os.path.abspath(os.path.join(_SCRIPT_DIR, '../..')) +for _p in (_SCRIPT_DIR, _REPO_ROOT): + if _p not in sys.path: + sys.path.insert(0, _p) + +import process + + +class ProcessTest(unittest.TestCase): + + def test_expand_input_files_sorts_largest_first(self): + with tempfile.TemporaryDirectory() as tmp_dir: + small_file = os.path.join(tmp_dir, 'a_small.csv') + large_file = os.path.join(tmp_dir, 'b_large.csv') + with open(small_file, 'w', encoding='utf-8') as f: + f.write('a,b\n1,2\n') + with open(large_file, 'w', encoding='utf-8') as f: + f.write('a,b\n' + '1,2\n' * 50) + + pattern = os.path.join(tmp_dir, '*.csv') + expanded = process._expand_input_files(pattern) + self.assertEqual(expanded, [large_file, small_file]) + + def test_merge_csv_files_keeps_single_header(self): + with tempfile.TemporaryDirectory() as tmp_dir: + p1 = os.path.join(tmp_dir, 'shard_00') + p2 = os.path.join(tmp_dir, 'shard_01') + with open(f'{p1}.csv', 'w', encoding='utf-8') as f: + f.write( + 'observationAbout,observationDate,value,variableMeasured\n' + 'dcid:country/JPN,2020,100,dcid:Count_Person\n' + ) + with open(f'{p2}.csv', 'w', encoding='utf-8') as f: + f.write( + 'observationAbout,observationDate,value,variableMeasured\n' + 'dcid:country/JPN,2025,95,dcid:Count_Person\n' + ) + + final_csv = os.path.join(tmp_dir, 'merged.csv') + total_rows = process._merge_csv_files([p1, p2], final_csv) + self.assertEqual(total_rows, 2) + with open(final_csv, 'r', encoding='utf-8') as f: + lines = f.read().splitlines() + self.assertEqual( + lines, + [ + 'observationAbout,observationDate,value,variableMeasured', + 'dcid:country/JPN,2020,100,dcid:Count_Person', + 'dcid:country/JPN,2025,95,dcid:Count_Person', + ], + ) + + def test_merge_counters_aggregates_int_and_float(self): + with tempfile.TemporaryDirectory() as tmp_dir: + c1 = os.path.join(tmp_dir, 'c1.csv') + c2 = os.path.join(tmp_dir, 'c2.csv') + with open(c1, 'w', encoding='utf-8') as f: + f.write('key,value\nsvobs-added,100\nelapsed_time,1.25\n') + with open(c2, 'w', encoding='utf-8') as f: + f.write('key,value\nsvobs-added,250\nelapsed_time,2.50\n') + + final_counters = os.path.join(tmp_dir, 'merged_counters.csv') + process._merge_counters([c1, c2], final_counters) + with open(final_counters, 'r', encoding='utf-8') as f: + lines = f.read().splitlines() + self.assertEqual( + lines, + [ + 'key,value', + 'elapsed_time,3.75', + 'svobs-added,350', + ], + ) + + def test_merge_mcf_files_deduplicates_and_skips_empty(self): + with tempfile.TemporaryDirectory() as tmp_dir: + p1 = os.path.join(tmp_dir, 'shard_00') + p2 = os.path.join(tmp_dir, 'shard_01') + final_mcf = os.path.join(tmp_dir, 'merged_stat_vars.mcf') + + # 1. Empty case: should not create final_mcf + process._merge_mcf_files([p1, p2], '_stat_vars.mcf', final_mcf) + self.assertFalse(os.path.exists(final_mcf)) + + # 2. Non-empty case with duplicate node across shards + node_block = ( + '# Auto generated comment\n\n' + 'Node: dcid:Count_Person\n' + 'typeOf: dcid:StatisticalVariable\n' + 'populationType: dcid:Person\n' + ) + with open(f'{p1}_stat_vars.mcf', 'w', encoding='utf-8') as f: + f.write(node_block) + with open(f'{p2}_stat_vars.mcf', 'w', encoding='utf-8') as f: + f.write(node_block) + + process._merge_mcf_files([p1, p2], '_stat_vars.mcf', final_mcf) + self.assertTrue(os.path.exists(final_mcf)) + with open(final_mcf, 'r', encoding='utf-8') as f: + content = f.read() + self.assertEqual(content.count('Node: dcid:Count_Person'), 1) + + @mock.patch.object(process.subprocess, 'run') + def test_run_shard_raises_on_nonzero_exit(self, mock_run): + mock_proc = mock.MagicMock() + mock_proc.returncode = 1 + mock_proc.stderr = 'Processor failure details' + mock_run.return_value = mock_proc + + with self.assertRaises(RuntimeError): + process._run_shard( + shard_idx=0, + input_file='input.csv', + shard_output_prefix='out_prefix', + shard_counters_path='counters.csv', + pv_map='pv.csv', + config_file='meta.csv', + existing_statvar_mcf='stat_vars.mcf', + ) + + +if __name__ == '__main__': + unittest.main() diff --git a/statvar_imports/japan_census/test_data/sample_expected_output.csv b/statvar_imports/japan_census/test_data/sample_expected_output.csv new file mode 100644 index 0000000000..cca6dbd693 --- /dev/null +++ b/statvar_imports/japan_census/test_data/sample_expected_output.csv @@ -0,0 +1,19 @@ +observationAbout,observationDate,value,variableMeasured +country/JPN,1920,55963053,dcid:Count_Person +country/JPN,1925,59736822,dcid:Count_Person +wikidataId/Q1037393,1975,5338206,dcid:Count_Person +wikidataId/Q71699,1975,1468646,dcid:Count_Person +wikidataId/Q37951,2025,1961996,dcid:Count_Person +wikidataId/Q1079635,2025,255523,dcid:Count_Person +country/JPN,1953,39130,dcid:Count_Person_Employed +country/JPN,1954,39630,dcid:Count_Person_Employed +country/JPN,1953,1.9,dcid:UnemploymentRate_Person +country/JPN,1954,2.3,dcid:UnemploymentRate_Person +country/JPN,1948,10774652,dcid:Count_Student_PrimarySchool +country/JPN,1949,10991927,dcid:Count_Student_PrimarySchool +country/JPN,1975,89683,dcid:Count_Teacher_Kindergarten +country/JPN,1976,93616,dcid:Count_Teacher_Kindergarten +country/JPN,1978,695821,dcid:Count_Death +country/JPN,1979,689664,dcid:Count_Death +country/JPN,1975,9032,dcid:Count_MortalityEvent_Diabetes +country/JPN,1976,9183,dcid:Count_MortalityEvent_Diabetes diff --git a/statvar_imports/japan_census/test_data/sample_input.csv b/statvar_imports/japan_census/test_data/sample_input.csv new file mode 100644 index 0000000000..ab2a55095f --- /dev/null +++ b/statvar_imports/japan_census/test_data/sample_input.csv @@ -0,0 +1,19 @@ +year,indicatorCd,regionCd,value,indicatorName,unitCd,unitName,statCd,regionName,timeCd,cycle,regionRank,isSeasonal,isProvisional +1920,0201010000000010000,00000,55963053,Total population (Both sexes),090,person,20020101,Japan,1920CY00,3,2,1,0 +1925,0201010000000010000,00000,59736822,Total population (Both sexes),090,person,20020101,Japan,1925CY00,3,2,1,0 +1975,0201010000000010000,01000,5338206,Total population (Both sexes),090,person,20020101,Hokkaido,1975CY00,3,3,1,0 +1975,0201010000000010000,02000,1468646,Total population (Both sexes),090,person,20020101,Aomori-ken,1975CY00,3,3,1,0 +2025,0201010000000010000,01100,1961996,Total population (Both sexes),090,person,00200521,Sapporo-shi,2025CY00,3,4,1,0 +2025,0201010000000010000,01101,255523,Total population (Both sexes),090,person,00200521,Sapporo-shi Chuo-ku,2025CY00,3,4,1,0 +1953,0301010000010010010,00000,3913,Employed persons (Both sexes),093,10 thousand persons,00200531,Japan,1953CY00,3,2,1,0 +1954,0301010000010010010,00000,3963,Employed persons (Both sexes),093,10 thousand persons,00200531,Japan,1954CY00,3,2,1,0 +1953,0301010000020020010,00000,1.9,Unemployment rate (Both sexes),001,%,00200531,Japan,1953CY00,3,2,1,0 +1954,0301010000020020010,00000,2.3,Unemployment rate (Both sexes),001,%,00200531,Japan,1954CY00,3,2,1,0 +1948,1201050300000010000,00000,10774652,Elementary school students,090,person,00400001,Japan,1948CY00,3,2,1,0 +1949,1201050300000010000,00000,10991927,Elementary school students,090,person,00400001,Japan,1949CY00,3,2,1,0 +1975,1201040100000010000,00000,89683,Number of kindergarten teachers (Total),090,person,00200502,Japan,1975CY00,3,2,1,0 +1976,1201040100000010000,00000,93616,Number of kindergarten teachers (Total),090,person,00200502,Japan,1976CY00,3,2,1,0 +1978,0203020000000010000,00000,695821,Deaths,090,person,00450011,Japan,1978CY00,3,2,1,0 +1979,0203020000000010000,00000,689664,Deaths,090,person,00450011,Japan,1979CY00,3,2,1,0 +1975,1505130000000010030,00000,9032,Deaths due to diabetes mellitus,090,person,00200502,Japan,1975CY00,3,2,1,0 +1976,1505130000000010030,00000,9183,Deaths due to diabetes mellitus,090,person,00200502,Japan,1976CY00,3,2,1,0 diff --git a/statvar_imports/japan_census/validation_config.json b/statvar_imports/japan_census/validation_config.json new file mode 100644 index 0000000000..925a8109fd --- /dev/null +++ b/statvar_imports/japan_census/validation_config.json @@ -0,0 +1,32 @@ +{ + "schema_version": "1.0", + "rules": [ + { + "rule_id": "check_deleted_records_percent", + "description": "Strictly enforce historical deletion average threshold of 0.1%", + "validator": "DELETED_RECORDS_PERCENT", + "enabled": false, + "params": { + "threshold": 0.1 + } + }, + { + "rule_id": "check_max_date_freshness", + "description": "Verify MaxDate is within allowable publication lag and values are non-negative (and within [0, 100] for rates) for active recurring indicators", + "validator": "SQL_VALIDATOR", + "params": { + "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, NumPlaces, COUNT(*) OVER () AS total_svs FROM stats WHERE StatVar NOT IN ('Count_Person_100To104Years', 'Count_Person_100To104Years_Male', 'Count_Person_100To104Years_Female', 'Count_Person_105To109Years', 'Count_Person_105To109Years_Male', 'Count_Person_105To109Years_Female', 'Count_Person_110OrMoreYears', 'Count_Person_110OrMoreYears_Male', 'Count_Person_110OrMoreYears_Female')", + "condition": "CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 5) AND total_svs >= 92 AND NumPlaces >= 1 AND MinValue >= 0 AND (StatVar != 'UnemploymentRate_Person' OR MaxValue <= 100)" + } + }, + { + "rule_id": "check_max_date_historical_indicators", + "description": "Verify MaxDate is at least the last published census wave (2020) and values are non-negative for discontinued historical centenarian sub-brackets", + "validator": "SQL_VALIDATOR", + "params": { + "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, COUNT(*) OVER () AS total_svs FROM stats WHERE StatVar IN ('Count_Person_100To104Years', 'Count_Person_100To104Years_Male', 'Count_Person_100To104Years_Female', 'Count_Person_105To109Years', 'Count_Person_105To109Years_Male', 'Count_Person_105To109Years_Female', 'Count_Person_110OrMoreYears', 'Count_Person_110OrMoreYears_Male', 'Count_Person_110OrMoreYears_Female')", + "condition": "CAST(MaxDate AS INTEGER) >= 2020 AND total_svs >= 9 AND MinValue >= 0" + } + } + ] +} \ No newline at end of file From 7561eb36061862e347584dcecab0b5e19bcf0763 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Thu, 22 Oct 2026 06:50:49 +0000 Subject: [PATCH 02/17] pvmap and metadata --- .../japan_census/japan_census_metadata.csv | 6 + .../japan_census/japan_census_pv_map.csv | 3976 +++++++++++++++++ 2 files changed, 3982 insertions(+) create mode 100644 statvar_imports/japan_census/japan_census_metadata.csv create mode 100644 statvar_imports/japan_census/japan_census_pv_map.csv diff --git a/statvar_imports/japan_census/japan_census_metadata.csv b/statvar_imports/japan_census/japan_census_metadata.csv new file mode 100644 index 0000000000..8d03aeeca1 --- /dev/null +++ b/statvar_imports/japan_census/japan_census_metadata.csv @@ -0,0 +1,6 @@ +parameter,value +url,https://dashboard.e-stat.go.jp/en/ +output_columns,"observationAbout,observationDate,value,variableMeasured" +header_rows,1 +mapped_columns,3 + diff --git a/statvar_imports/japan_census/japan_census_pv_map.csv b/statvar_imports/japan_census/japan_census_pv_map.csv new file mode 100644 index 0000000000..981a72f168 --- /dev/null +++ b/statvar_imports/japan_census/japan_census_pv_map.csv @@ -0,0 +1,3976 @@ +key,p1,v1,p2,v2,p3,v3,p4,v4,p5,v5,p6,v6,p7,v7 +year,observationDate,{@Data} +regionCd,observationAbout,{@Data} +value,value,{Number} +indicatorName,#ignore,indicator description redundant with indicatorCd,value,{@Data} +unitCd,#ignore,unit code metadata,value,{@Data} +unitName,#ignore,unit name metadata,value,{@Data} +statCd,#ignore,survey code metadata,value,{@Data} +regionName,#ignore,region name redundant with regionCd,value,{@Data} +timeCd,#ignore,raw time code redundant with year,value,{@Data} +cycle,#ignore,cycle metadata,value,{@Data} +regionRank,#ignore,regional rank metadata,value,{@Data} +isSeasonal,#ignore,seasonal adjustment flag,value,{@Data} +isProvisional,#ignore,provisional flag,value,{@Data} +-,#IgnoreRow,True,#ignore,suppressed or missing value +0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employmentStatus,Employed,#Multiply,10 +0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employmentStatus,Unemployed,#Multiply,10 +0301010000030010010,dcid,Count_Person_NotInLaborForce,populationType,Person,measuredProperty,count,employmentStatus,BLS_NotInLaborForce,age,Years16Onwards,#Multiply,10 +0301010000020020010,populationType,Person,measuredProperty,unemploymentRate +1201050300000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,PrimarySchool +1201050400000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,MiddleSchool +1201050500000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,HighSchool +1201050800000010000,populationType,Student,measuredProperty,count,enrollmentLevel,Graduate +1201040100000010000,populationType,Teacher,measuredProperty,count,schoolGradeLevel,Kindergarten +1201040300000010000,populationType,Teacher,measuredProperty,count,facultyType,ElementaryTeacher +1201040500000010000,populationType,Teacher,measuredProperty,count,facultyType,UniversityAndCollegeTeacher +1505130000000010030,populationType,MortalityEvent,measuredProperty,count,causeOfDeath,Diabetes +1505130000000010050,populationType,MortalityEvent,measuredProperty,count,causeOfDeath,HeartDiseaseExcludingHypertension +1505130000000010070,populationType,MortalityEvent,measuredProperty,count,causeOfDeath,Suicide +0203020000000010000,dcid,Count_Death,populationType,MortalityEvent,measuredProperty,count +0203020100000010000,dcid,Count_Death_Male,populationType,MortalityEvent,measuredProperty,count,gender,Male +0203020200000010000,dcid,Count_Death_Female,populationType,MortalityEvent,measuredProperty,count,gender,Female +0201010000000010000,populationType,Person,measuredProperty,count +0201010100000010000,populationType,Person,measuredProperty,count,gender,Male +0201010200000010000,populationType,Person,measuredProperty,count,gender,Female +0201010010000010010,populationType,Person,measuredProperty,count,age,YearsUpto14 +0201010010000010020,populationType,Person,measuredProperty,count,age,Years15To64 +0201010010000010030,dcid,Count_Person_65OrMoreYears,populationType,Person,measuredProperty,count,age,Years65Onwards +0201010110000010010,populationType,Person,measuredProperty,count,age,YearsUpto14,gender,Male +0201010110000010020,populationType,Person,measuredProperty,count,age,Years15To64,gender,Male +0201010110000010030,dcid,Count_Person_65OrMoreYears_Male,populationType,Person,measuredProperty,count,age,Years65Onwards,gender,Male +0201010210000010010,populationType,Person,measuredProperty,count,age,YearsUpto14,gender,Female +0201010210000010020,populationType,Person,measuredProperty,count,age,Years15To64,gender,Female +0201010210000010030,dcid,Count_Person_65OrMoreYears_Female,populationType,Person,measuredProperty,count,age,Years65Onwards,gender,Female +0201010040000010070,populationType,Person,measuredProperty,count,age,Years85Onwards +0201010140000010070,populationType,Person,measuredProperty,count,age,Years85Onwards,gender,Male +0201010240000010070,populationType,Person,measuredProperty,count,age,Years85Onwards,gender,Female +0201010020000010010,populationType,Person,measuredProperty,count,age,Years0To4 +0201010120000010010,populationType,Person,measuredProperty,count,age,Years0To4,gender,Male +0201010220000010010,populationType,Person,measuredProperty,count,age,Years0To4,gender,Female +0201010020000010020,populationType,Person,measuredProperty,count,age,Years5To9 +0201010120000010020,populationType,Person,measuredProperty,count,age,Years5To9,gender,Male +0201010220000010020,populationType,Person,measuredProperty,count,age,Years5To9,gender,Female +0201010020000010030,populationType,Person,measuredProperty,count,age,Years10To14 +0201010120000010030,populationType,Person,measuredProperty,count,age,Years10To14,gender,Male +0201010220000010030,populationType,Person,measuredProperty,count,age,Years10To14,gender,Female +0201010020000010040,populationType,Person,measuredProperty,count,age,Years15To19 +0201010120000010040,populationType,Person,measuredProperty,count,age,Years15To19,gender,Male +0201010220000010040,populationType,Person,measuredProperty,count,age,Years15To19,gender,Female +0201010020000010050,populationType,Person,measuredProperty,count,age,Years20To24 +0201010120000010050,dcid,Count_Person_20To24Years_Male,populationType,Person,measuredProperty,count,age,Years20To24,gender,Male +0201010220000010050,dcid,Count_Person_20To24Years_Female,populationType,Person,measuredProperty,count,age,Years20To24,gender,Female +0201010020000010060,populationType,Person,measuredProperty,count,age,Years25To29 +0201010120000010060,populationType,Person,measuredProperty,count,age,Years25To29,gender,Male +0201010220000010060,populationType,Person,measuredProperty,count,age,Years25To29,gender,Female +0201010020000010070,populationType,Person,measuredProperty,count,age,Years30To34 +0201010120000010070,populationType,Person,measuredProperty,count,age,Years30To34,gender,Male +0201010220000010070,populationType,Person,measuredProperty,count,age,Years30To34,gender,Female +0201010020000010080,populationType,Person,measuredProperty,count,age,Years35To39 +0201010120000010080,populationType,Person,measuredProperty,count,age,Years35To39,gender,Male +0201010220000010080,populationType,Person,measuredProperty,count,age,Years35To39,gender,Female +0201010020000010090,populationType,Person,measuredProperty,count,age,Years40To44 +0201010120000010090,populationType,Person,measuredProperty,count,age,Years40To44,gender,Male +0201010220000010090,populationType,Person,measuredProperty,count,age,Years40To44,gender,Female +0201010020000010100,populationType,Person,measuredProperty,count,age,Years45To49 +0201010120000010100,populationType,Person,measuredProperty,count,age,Years45To49,gender,Male +0201010220000010100,populationType,Person,measuredProperty,count,age,Years45To49,gender,Female +0201010020000010110,populationType,Person,measuredProperty,count,age,Years50To54 +0201010120000010110,populationType,Person,measuredProperty,count,age,Years50To54,gender,Male +0201010220000010110,populationType,Person,measuredProperty,count,age,Years50To54,gender,Female +0201010020000010120,populationType,Person,measuredProperty,count,age,Years55To59 +0201010120000010120,populationType,Person,measuredProperty,count,age,Years55To59,gender,Male +0201010220000010120,populationType,Person,measuredProperty,count,age,Years55To59,gender,Female +0201010020000010130,populationType,Person,measuredProperty,count,age,Years60To64 +0201010120000010130,populationType,Person,measuredProperty,count,age,Years60To64,gender,Male +0201010220000010130,populationType,Person,measuredProperty,count,age,Years60To64,gender,Female +0201010020000010140,populationType,Person,measuredProperty,count,age,Years65To69 +0201010120000010140,populationType,Person,measuredProperty,count,age,Years65To69,gender,Male +0201010220000010140,populationType,Person,measuredProperty,count,age,Years65To69,gender,Female +0201010020000010150,populationType,Person,measuredProperty,count,age,Years70To74 +0201010120000010150,populationType,Person,measuredProperty,count,age,Years70To74,gender,Male +0201010220000010150,populationType,Person,measuredProperty,count,age,Years70To74,gender,Female +0201010020000010160,populationType,Person,measuredProperty,count,age,Years75To79 +0201010120000010160,populationType,Person,measuredProperty,count,age,Years75To79,gender,Male +0201010220000010160,populationType,Person,measuredProperty,count,age,Years75To79,gender,Female +0201010020000010170,populationType,Person,measuredProperty,count,age,Years80To84 +0201010120000010170,populationType,Person,measuredProperty,count,age,Years80To84,gender,Male +0201010220000010170,populationType,Person,measuredProperty,count,age,Years80To84,gender,Female +0201010020000010180,populationType,Person,measuredProperty,count,age,Years85To89 +0201010120000010180,populationType,Person,measuredProperty,count,age,Years85To89,gender,Male +0201010220000010180,populationType,Person,measuredProperty,count,age,Years85To89,gender,Female +0201010020000010190,populationType,Person,measuredProperty,count,age,Years90To94 +0201010120000010190,populationType,Person,measuredProperty,count,age,Years90To94,gender,Male +0201010220000010190,populationType,Person,measuredProperty,count,age,Years90To94,gender,Female +0201010020000010200,populationType,Person,measuredProperty,count,age,Years95To99 +0201010120000010200,populationType,Person,measuredProperty,count,age,Years95To99,gender,Male +0201010220000010200,populationType,Person,measuredProperty,count,age,Years95To99,gender,Female +0201010020000010210,populationType,Person,measuredProperty,count,age,Years100To104 +0201010120000010210,populationType,Person,measuredProperty,count,age,Years100To104,gender,Male +0201010220000010210,populationType,Person,measuredProperty,count,age,Years100To104,gender,Female +0201010020000010220,populationType,Person,measuredProperty,count,age,Years105To109 +0201010120000010220,populationType,Person,measuredProperty,count,age,Years105To109,gender,Male +0201010220000010220,populationType,Person,measuredProperty,count,age,Years105To109,gender,Female +0201010020000010230,populationType,Person,measuredProperty,count,age,Years110Onwards +0201010120000010230,populationType,Person,measuredProperty,count,age,Years110Onwards,gender,Male +0201010220000010230,populationType,Person,measuredProperty,count,age,Years110Onwards,gender,Female +0201010030000011000,#IgnoreRow,True,#ignore,100+ aggregate used internally to compute 65+ and 85+ +0201010130000011000,#IgnoreRow,True,#ignore,100+ aggregate used internally to compute 65+ and 85+ +0201010230000011000,#IgnoreRow,True,#ignore,100+ aggregate used internally to compute 65+ and 85+ +00000,observationAbout,country/JPN +01000,observationAbout,wikidataId/Q1037393 +01100,observationAbout,wikidataId/Q37951 +01101,observationAbout,wikidataId/Q1079635 +01102,observationAbout,wikidataId/Q1134013 +01103,observationAbout,wikidataId/Q1134006 +01104,observationAbout,wikidataId/Q647388 +01105,observationAbout,wikidataId/Q1134225 +01106,observationAbout,wikidataId/Q1134027 +01107,observationAbout,wikidataId/Q1135640 +01108,observationAbout,wikidataId/Q1134346 +01109,observationAbout,wikidataId/Q1134339 +01110,observationAbout,wikidataId/Q1134019 +01202,observationAbout,wikidataId/Q26418 +01203,observationAbout,wikidataId/Q242108 +01204,observationAbout,wikidataId/Q200740 +01205,observationAbout,wikidataId/Q496667 +01206,observationAbout,wikidataId/Q271917 +01207,observationAbout,wikidataId/Q177149 +01208,observationAbout,wikidataId/Q265555 +01209,observationAbout,wikidataId/Q637145 +01210,observationAbout,wikidataId/Q693264 +01211,observationAbout,wikidataId/Q305640 +01212,observationAbout,wikidataId/Q714628 +01213,observationAbout,wikidataId/Q390413 +01214,observationAbout,wikidataId/Q267894 +01215,observationAbout,wikidataId/Q817271 +01216,observationAbout,wikidataId/Q725311 +01217,observationAbout,wikidataId/Q496681 +01218,observationAbout,wikidataId/Q413804 +01219,observationAbout,wikidataId/Q692745 +01220,observationAbout,wikidataId/Q766942 +01221,observationAbout,wikidataId/Q820725 +01222,observationAbout,wikidataId/Q842318 +01223,observationAbout,wikidataId/Q502205 +01224,observationAbout,wikidataId/Q693237 +01225,observationAbout,wikidataId/Q842286 +01226,observationAbout,wikidataId/Q507147 +01227,observationAbout,wikidataId/Q842279 +01228,observationAbout,wikidataId/Q817268 +01229,observationAbout,wikidataId/Q654738 +01230,observationAbout,wikidataId/Q428989 +01231,observationAbout,wikidataId/Q733979 +01233,observationAbout,wikidataId/Q693178 +01234,observationAbout,wikidataId/Q817278 +01235,observationAbout,wikidataId/Q693221 +01236,observationAbout,wikidataId/Q842301 +01301,#IgnoreRow,True,#ignore,unmapped historical municipality +01302,#IgnoreRow,True,#ignore,unmapped historical municipality +01303,observationAbout,wikidataId/Q1348675 +01304,observationAbout,wikidataId/Q1355454 +01305,#IgnoreRow,True,#ignore,unmapped historical municipality +01306,#IgnoreRow,True,#ignore,unmapped historical municipality +01331,observationAbout,wikidataId/Q1374364 +01332,observationAbout,wikidataId/Q1354260 +01333,observationAbout,wikidataId/Q1354297 +01334,observationAbout,wikidataId/Q1354308 +01335,#IgnoreRow,True,#ignore,unmapped historical municipality +01336,#IgnoreRow,True,#ignore,unmapped historical municipality +01337,observationAbout,wikidataId/Q1354276 +01339,#IgnoreRow,True,#ignore,unmapped historical municipality +01340,#IgnoreRow,True,#ignore,unmapped historical municipality +01341,#IgnoreRow,True,#ignore,unmapped historical municipality +01342,#IgnoreRow,True,#ignore,unmapped historical municipality +01343,observationAbout,wikidataId/Q773727 +01344,#IgnoreRow,True,#ignore,unmapped historical municipality +01345,observationAbout,wikidataId/Q1354319 +01346,observationAbout,wikidataId/Q1354242 +01347,observationAbout,wikidataId/Q1354333 +01361,observationAbout,wikidataId/Q1348738 +01362,observationAbout,wikidataId/Q1354203 +01363,observationAbout,wikidataId/Q1354252 +01364,observationAbout,wikidataId/Q1354219 +01365,#IgnoreRow,True,#ignore,unmapped historical municipality +01366,#IgnoreRow,True,#ignore,unmapped historical municipality +01367,observationAbout,wikidataId/Q1022916 +01368,#IgnoreRow,True,#ignore,unmapped historical municipality +01369,#IgnoreRow,True,#ignore,unmapped historical municipality +01370,observationAbout,wikidataId/Q928820 +01371,observationAbout,wikidataId/Q1036505 +01391,observationAbout,wikidataId/Q1354462 +01392,observationAbout,wikidataId/Q1355602 +01393,observationAbout,wikidataId/Q1355464 +01394,observationAbout,wikidataId/Q1355416 +01395,observationAbout,wikidataId/Q1204586 +01396,observationAbout,wikidataId/Q1354496 +01397,observationAbout,wikidataId/Q1354413 +01398,observationAbout,wikidataId/Q1355478 +01399,observationAbout,wikidataId/Q1354396 +01400,observationAbout,wikidataId/Q1204596 +01401,observationAbout,wikidataId/Q1354347 +01402,observationAbout,wikidataId/Q1354481 +01403,observationAbout,wikidataId/Q268750 +01404,observationAbout,wikidataId/Q644751 +01405,observationAbout,wikidataId/Q1354431 +01406,observationAbout,wikidataId/Q524318 +01407,observationAbout,wikidataId/Q1354360 +01408,observationAbout,wikidataId/Q1348748 +01409,observationAbout,wikidataId/Q1354513 +01421,#IgnoreRow,True,#ignore,unmapped historical municipality +01422,#IgnoreRow,True,#ignore,unmapped historical municipality +01423,observationAbout,wikidataId/Q1355427 +01424,observationAbout,wikidataId/Q1355445 +01425,observationAbout,wikidataId/Q1357239 +01427,observationAbout,wikidataId/Q1355570 +01428,observationAbout,wikidataId/Q1355522 +01429,observationAbout,wikidataId/Q1357008 +01430,observationAbout,wikidataId/Q1357069 +01431,observationAbout,wikidataId/Q1355559 +01432,observationAbout,wikidataId/Q1357180 +01433,observationAbout,wikidataId/Q942358 +01434,observationAbout,wikidataId/Q1357097 +01436,observationAbout,wikidataId/Q1357037 +01437,observationAbout,wikidataId/Q1357113 +01438,observationAbout,wikidataId/Q1357023 +01439,#IgnoreRow,True,#ignore,unmapped historical municipality +01452,observationAbout,wikidataId/Q1348693 +01453,observationAbout,wikidataId/Q1357054 +01454,observationAbout,wikidataId/Q1357085 +01455,observationAbout,wikidataId/Q1355549 +01456,observationAbout,wikidataId/Q1355434 +01457,observationAbout,wikidataId/Q1348798 +01458,observationAbout,wikidataId/Q1357210 +01459,observationAbout,wikidataId/Q1355491 +01460,observationAbout,wikidataId/Q1357226 +01461,observationAbout,wikidataId/Q680384 +01462,observationAbout,wikidataId/Q1357254 +01463,observationAbout,wikidataId/Q1356989 +01464,observationAbout,wikidataId/Q1357267 +01465,observationAbout,wikidataId/Q1210574 +01466,#IgnoreRow,True,#ignore,unmapped historical municipality +01467,#IgnoreRow,True,#ignore,unmapped historical municipality +01468,observationAbout,wikidataId/Q2364809 +01469,observationAbout,wikidataId/Q1210495 +01470,observationAbout,wikidataId/Q599841 +01471,observationAbout,wikidataId/Q1210568 +01472,observationAbout,wikidataId/Q1355616 +01481,observationAbout,wikidataId/Q746578 +01482,observationAbout,wikidataId/Q1348684 +01483,observationAbout,wikidataId/Q1355396 +01484,observationAbout,wikidataId/Q1357163 +01485,observationAbout,wikidataId/Q1355358 +01486,observationAbout,wikidataId/Q1335041 +01487,observationAbout,wikidataId/Q1355384 +01488,#IgnoreRow,True,#ignore,unmapped historical municipality +01511,observationAbout,wikidataId/Q1210543 +01512,observationAbout,wikidataId/Q1210124 +01513,observationAbout,wikidataId/Q1211211 +01514,observationAbout,wikidataId/Q1210596 +01515,#IgnoreRow,True,#ignore,unmapped historical municipality +01516,observationAbout,wikidataId/Q1210534 +01517,observationAbout,wikidataId/Q1272571 +01518,observationAbout,wikidataId/Q1355370 +01519,observationAbout,wikidataId/Q1355406 +01520,observationAbout,wikidataId/Q955532 +01541,#IgnoreRow,True,#ignore,unmapped historical municipality +01542,#IgnoreRow,True,#ignore,unmapped historical municipality +01543,observationAbout,wikidataId/Q1355074 +01544,observationAbout,wikidataId/Q1357133 +01545,observationAbout,wikidataId/Q1365478 +01546,observationAbout,wikidataId/Q1365488 +01547,observationAbout,wikidataId/Q1357492 +01548,#IgnoreRow,True,#ignore,unmapped historical municipality +01549,observationAbout,wikidataId/Q1365442 +01550,observationAbout,wikidataId/Q1365495 +01551,#IgnoreRow,True,#ignore,unmapped historical municipality +01552,observationAbout,wikidataId/Q1365521 +01553,#IgnoreRow,True,#ignore,unmapped historical municipality +01554,#IgnoreRow,True,#ignore,unmapped historical municipality +01555,observationAbout,wikidataId/Q1348788 +01556,#IgnoreRow,True,#ignore,unmapped historical municipality +01557,#IgnoreRow,True,#ignore,unmapped historical municipality +01558,#IgnoreRow,True,#ignore,unmapped historical municipality +01559,observationAbout,wikidataId/Q1348807 +01560,observationAbout,wikidataId/Q1348703 +01561,observationAbout,wikidataId/Q1348729 +01562,observationAbout,wikidataId/Q1348818 +01563,observationAbout,wikidataId/Q1206664 +01564,observationAbout,wikidataId/Q1208839 +01571,observationAbout,wikidataId/Q1354284 +01572,#IgnoreRow,True,#ignore,unmapped historical municipality +01573,#IgnoreRow,True,#ignore,unmapped historical municipality +01574,#IgnoreRow,True,#ignore,unmapped historical municipality +01575,observationAbout,wikidataId/Q670712 +01578,observationAbout,wikidataId/Q1348760 +01579,#IgnoreRow,True,#ignore,unmapped historical municipality +01580,#IgnoreRow,True,#ignore,unmapped historical municipality +01581,observationAbout,wikidataId/Q1354230 +01582,#IgnoreRow,True,#ignore,unmapped historical municipality +01583,#IgnoreRow,True,#ignore,unmapped historical municipality +01584,observationAbout,wikidataId/Q1022930 +01585,observationAbout,wikidataId/Q1354084 +01586,observationAbout,wikidataId/Q1187152 +01601,observationAbout,wikidataId/Q1354380 +01602,observationAbout,wikidataId/Q1354447 +01603,#IgnoreRow,True,#ignore,unmapped historical municipality +01604,observationAbout,wikidataId/Q1355583 +01605,#IgnoreRow,True,#ignore,unmapped historical municipality +01606,#IgnoreRow,True,#ignore,unmapped historical municipality +01607,observationAbout,wikidataId/Q1355506 +01608,observationAbout,wikidataId/Q1208030 +01609,observationAbout,wikidataId/Q187124 +01610,observationAbout,wikidataId/Q1210679 +01631,observationAbout,wikidataId/Q1210613 +01632,observationAbout,wikidataId/Q1210187 +01633,observationAbout,wikidataId/Q1210152 +01634,observationAbout,wikidataId/Q1357679 +01635,observationAbout,wikidataId/Q1210175 +01636,observationAbout,wikidataId/Q1210528 +01637,observationAbout,wikidataId/Q1210480 +01638,observationAbout,wikidataId/Q1210501 +01639,observationAbout,wikidataId/Q1210582 +01640,#IgnoreRow,True,#ignore,unmapped historical municipality +01641,observationAbout,wikidataId/Q1357479 +01642,observationAbout,wikidataId/Q1210489 +01643,observationAbout,wikidataId/Q1210509 +01644,observationAbout,wikidataId/Q1357667 +01645,observationAbout,wikidataId/Q1210588 +01646,observationAbout,wikidataId/Q1210518 +01647,observationAbout,wikidataId/Q1186253 +01648,observationAbout,wikidataId/Q1185968 +01649,observationAbout,wikidataId/Q1186198 +01661,observationAbout,wikidataId/Q1186155 +01662,observationAbout,wikidataId/Q419720 +01663,observationAbout,wikidataId/Q1185978 +01664,observationAbout,wikidataId/Q1208741 +01665,observationAbout,wikidataId/Q1186104 +01666,#IgnoreRow,True,#ignore,unmapped historical municipality +01667,observationAbout,wikidataId/Q1186097 +01668,observationAbout,wikidataId/Q1186129 +01669,#IgnoreRow,True,#ignore,unmapped historical municipality +01691,observationAbout,wikidataId/Q657530 +01692,observationAbout,wikidataId/Q514324 +01693,observationAbout,wikidataId/Q1186204 +01694,observationAbout,wikidataId/Q1208391 +02000,observationAbout,wikidataId/Q71699 +02201,observationAbout,wikidataId/Q146790 +02202,observationAbout,wikidataId/Q194087 +02203,observationAbout,wikidataId/Q191914 +02204,observationAbout,wikidataId/Q496412 +02205,observationAbout,wikidataId/Q496702 +02206,observationAbout,wikidataId/Q646818 +02207,observationAbout,wikidataId/Q496401 +02208,observationAbout,wikidataId/Q641992 +02209,observationAbout,wikidataId/Q372517 +02210,observationAbout,wikidataId/Q695930 +02301,observationAbout,wikidataId/Q1206422 +02302,#IgnoreRow,True,#ignore,unmapped historical municipality +02303,observationAbout,wikidataId/Q1362366 +02304,observationAbout,wikidataId/Q1347692 +02305,#IgnoreRow,True,#ignore,unmapped historical municipality +02306,#IgnoreRow,True,#ignore,unmapped historical municipality +02307,observationAbout,wikidataId/Q1347655 +02321,observationAbout,wikidataId/Q1347560 +02322,#IgnoreRow,True,#ignore,unmapped historical municipality +02323,observationAbout,wikidataId/Q1347642 +02324,#IgnoreRow,True,#ignore,unmapped historical municipality +02325,#IgnoreRow,True,#ignore,unmapped historical municipality +02326,#IgnoreRow,True,#ignore,unmapped historical municipality +02327,#IgnoreRow,True,#ignore,unmapped historical municipality +02328,#IgnoreRow,True,#ignore,unmapped historical municipality +02341,#IgnoreRow,True,#ignore,unmapped historical municipality +02342,#IgnoreRow,True,#ignore,unmapped historical municipality +02343,observationAbout,wikidataId/Q1347597 +02361,observationAbout,wikidataId/Q1348951 +02362,observationAbout,wikidataId/Q1347678 +02363,#IgnoreRow,True,#ignore,unmapped historical municipality +02364,#IgnoreRow,True,#ignore,unmapped historical municipality +02365,#IgnoreRow,True,#ignore,unmapped historical municipality +02366,#IgnoreRow,True,#ignore,unmapped historical municipality +02367,observationAbout,wikidataId/Q1362354 +02368,#IgnoreRow,True,#ignore,unmapped historical municipality +02381,observationAbout,wikidataId/Q1347574 +02382,#IgnoreRow,True,#ignore,unmapped historical municipality +02383,#IgnoreRow,True,#ignore,unmapped historical municipality +02384,observationAbout,wikidataId/Q2358964 +02385,#IgnoreRow,True,#ignore,unmapped historical municipality +02386,#IgnoreRow,True,#ignore,unmapped historical municipality +02387,observationAbout,wikidataId/Q1347669 +02401,observationAbout,wikidataId/Q1362344 +02402,observationAbout,wikidataId/Q1349448 +02403,#IgnoreRow,True,#ignore,unmapped historical municipality +02404,#IgnoreRow,True,#ignore,unmapped historical municipality +02405,observationAbout,wikidataId/Q1206630 +02406,observationAbout,wikidataId/Q1205910 +02407,#IgnoreRow,True,#ignore,unmapped historical municipality +02408,observationAbout,wikidataId/Q1194394 +02409,#IgnoreRow,True,#ignore,unmapped historical municipality +02410,#IgnoreRow,True,#ignore,unmapped historical municipality +02411,observationAbout,wikidataId/Q1362335 +02412,observationAbout,wikidataId/Q1349124 +02421,#IgnoreRow,True,#ignore,unmapped historical municipality +02422,#IgnoreRow,True,#ignore,unmapped historical municipality +02423,observationAbout,wikidataId/Q1362378 +02424,observationAbout,wikidataId/Q1349156 +02425,observationAbout,wikidataId/Q1349460 +02426,observationAbout,wikidataId/Q1362393 +02427,#IgnoreRow,True,#ignore,unmapped historical municipality +02441,observationAbout,wikidataId/Q2358935 +02442,observationAbout,wikidataId/Q1362324 +02443,observationAbout,wikidataId/Q1349168 +02444,#IgnoreRow,True,#ignore,unmapped historical municipality +02445,observationAbout,wikidataId/Q1206409 +02446,observationAbout,wikidataId/Q1349010 +02447,#IgnoreRow,True,#ignore,unmapped historical municipality +02448,#IgnoreRow,True,#ignore,unmapped historical municipality +02449,#IgnoreRow,True,#ignore,unmapped historical municipality +02450,observationAbout,wikidataId/Q44548 +03000,observationAbout,wikidataId/Q48326 +03201,observationAbout,wikidataId/Q200077 +03202,observationAbout,wikidataId/Q502981 +03203,observationAbout,wikidataId/Q384934 +03204,#IgnoreRow,True,#ignore,unmapped historical municipality +03205,observationAbout,wikidataId/Q317827 +03206,observationAbout,wikidataId/Q695917 +03207,observationAbout,wikidataId/Q695911 +03208,observationAbout,wikidataId/Q646766 +03209,observationAbout,wikidataId/Q390817 +03210,observationAbout,wikidataId/Q270463 +03211,observationAbout,wikidataId/Q329790 +03212,#IgnoreRow,True,#ignore,unmapped historical municipality +03213,observationAbout,wikidataId/Q695907 +03214,observationAbout,wikidataId/Q765884 +03215,observationAbout,wikidataId/Q386052 +03216,observationAbout,wikidataId/Q1205932 +03301,observationAbout,wikidataId/Q1348478 +03302,observationAbout,wikidataId/Q1205166 +03303,observationAbout,wikidataId/Q1206018 +03304,#IgnoreRow,True,#ignore,unmapped historical municipality +03305,#IgnoreRow,True,#ignore,unmapped historical municipality +03306,#IgnoreRow,True,#ignore,unmapped historical municipality +03307,#IgnoreRow,True,#ignore,unmapped historical municipality +03321,observationAbout,wikidataId/Q387336 +03322,observationAbout,wikidataId/Q1205709 +03323,#IgnoreRow,True,#ignore,unmapped historical municipality +03341,#IgnoreRow,True,#ignore,unmapped historical municipality +03342,#IgnoreRow,True,#ignore,unmapped historical municipality +03361,#IgnoreRow,True,#ignore,unmapped historical municipality +03362,#IgnoreRow,True,#ignore,unmapped historical municipality +03363,#IgnoreRow,True,#ignore,unmapped historical municipality +03364,#IgnoreRow,True,#ignore,unmapped historical municipality +03365,#IgnoreRow,True,#ignore,unmapped historical municipality +03366,observationAbout,wikidataId/Q1205938 +03381,observationAbout,wikidataId/Q1205722 +03382,#IgnoreRow,True,#ignore,unmapped historical municipality +03383,#IgnoreRow,True,#ignore,unmapped historical municipality +03384,#IgnoreRow,True,#ignore,unmapped historical municipality +03401,#IgnoreRow,True,#ignore,unmapped historical municipality +03402,observationAbout,wikidataId/Q1203004 +03421,#IgnoreRow,True,#ignore,unmapped historical municipality +03422,#IgnoreRow,True,#ignore,unmapped historical municipality +03423,#IgnoreRow,True,#ignore,unmapped historical municipality +03424,#IgnoreRow,True,#ignore,unmapped historical municipality +03425,#IgnoreRow,True,#ignore,unmapped historical municipality +03426,#IgnoreRow,True,#ignore,unmapped historical municipality +03441,observationAbout,wikidataId/Q1354773 +03442,#IgnoreRow,True,#ignore,unmapped historical municipality +03461,observationAbout,wikidataId/Q386411 +03462,#IgnoreRow,True,#ignore,unmapped historical municipality +03481,#IgnoreRow,True,#ignore,unmapped historical municipality +03482,observationAbout,wikidataId/Q1023790 +03483,observationAbout,wikidataId/Q684442 +03484,observationAbout,wikidataId/Q739087 +03485,observationAbout,wikidataId/Q1349135 +03486,#IgnoreRow,True,#ignore,unmapped historical municipality +03487,#IgnoreRow,True,#ignore,unmapped historical municipality +03501,observationAbout,wikidataId/Q972924 +03502,#IgnoreRow,True,#ignore,unmapped historical municipality +03503,observationAbout,wikidataId/Q733371 +03504,#IgnoreRow,True,#ignore,unmapped historical municipality +03505,#IgnoreRow,True,#ignore,unmapped historical municipality +03506,observationAbout,wikidataId/Q1205734 +03507,observationAbout,wikidataId/Q1347494 +03521,#IgnoreRow,True,#ignore,unmapped historical municipality +03523,#IgnoreRow,True,#ignore,unmapped historical municipality +03524,observationAbout,wikidataId/Q1205998 +04000,observationAbout,wikidataId/Q47896 +04100,observationAbout,wikidataId/Q46747 +04101,observationAbout,wikidataId/Q1042308 +04102,observationAbout,wikidataId/Q1043355 +04103,observationAbout,wikidataId/Q1131743 +04104,observationAbout,wikidataId/Q1043331 +04105,observationAbout,wikidataId/Q1043338 +04201,#IgnoreRow,True,#ignore,unmapped historical municipality +04202,observationAbout,wikidataId/Q270814 +04203,observationAbout,wikidataId/Q496448 +04204,#IgnoreRow,True,#ignore,unmapped historical municipality +04205,observationAbout,wikidataId/Q235746 +04206,observationAbout,wikidataId/Q823489 +04207,observationAbout,wikidataId/Q642018 +04208,observationAbout,wikidataId/Q696425 +04209,observationAbout,wikidataId/Q653355 +04210,#IgnoreRow,True,#ignore,unmapped historical municipality +04211,observationAbout,wikidataId/Q518203 +04212,observationAbout,wikidataId/Q696401 +04213,observationAbout,wikidataId/Q469189 +04214,observationAbout,wikidataId/Q511872 +04215,observationAbout,wikidataId/Q385892 +04216,observationAbout,wikidataId/Q1205173 +04301,observationAbout,wikidataId/Q1347874 +04302,observationAbout,wikidataId/Q1204398 +04321,observationAbout,wikidataId/Q1155682 +04322,observationAbout,wikidataId/Q1347432 +04323,observationAbout,wikidataId/Q1204965 +04324,observationAbout,wikidataId/Q1205337 +04341,observationAbout,wikidataId/Q685775 +04361,observationAbout,wikidataId/Q1205296 +04362,observationAbout,wikidataId/Q1205163 +04382,#IgnoreRow,True,#ignore,unmapped historical municipality +04401,observationAbout,wikidataId/Q1349082 +04404,observationAbout,wikidataId/Q478272 +04405,#IgnoreRow,True,#ignore,unmapped historical municipality +04406,observationAbout,wikidataId/Q1202986 +04421,observationAbout,wikidataId/Q1205377 +04422,observationAbout,wikidataId/Q1204974 +04423,#IgnoreRow,True,#ignore,unmapped historical municipality +04424,observationAbout,wikidataId/Q1204681 +04441,#IgnoreRow,True,#ignore,unmapped historical municipality +04442,#IgnoreRow,True,#ignore,unmapped historical municipality +04443,#IgnoreRow,True,#ignore,unmapped historical municipality +04444,observationAbout,wikidataId/Q1204478 +04445,observationAbout,wikidataId/Q1205414 +04461,#IgnoreRow,True,#ignore,unmapped historical municipality +04462,#IgnoreRow,True,#ignore,unmapped historical municipality +04463,#IgnoreRow,True,#ignore,unmapped historical municipality +04481,#IgnoreRow,True,#ignore,unmapped historical municipality +04482,#IgnoreRow,True,#ignore,unmapped historical municipality +04501,observationAbout,wikidataId/Q1155902 +04502,#IgnoreRow,True,#ignore,unmapped historical municipality +04503,#IgnoreRow,True,#ignore,unmapped historical municipality +04504,#IgnoreRow,True,#ignore,unmapped historical municipality +04505,observationAbout,wikidataId/Q1205284 +04521,#IgnoreRow,True,#ignore,unmapped historical municipality +04522,#IgnoreRow,True,#ignore,unmapped historical municipality +04523,#IgnoreRow,True,#ignore,unmapped historical municipality +04524,#IgnoreRow,True,#ignore,unmapped historical municipality +04525,#IgnoreRow,True,#ignore,unmapped historical municipality +04526,#IgnoreRow,True,#ignore,unmapped historical municipality +04527,#IgnoreRow,True,#ignore,unmapped historical municipality +04528,#IgnoreRow,True,#ignore,unmapped historical municipality +04529,#IgnoreRow,True,#ignore,unmapped historical municipality +04530,#IgnoreRow,True,#ignore,unmapped historical municipality +04541,#IgnoreRow,True,#ignore,unmapped historical municipality +04542,#IgnoreRow,True,#ignore,unmapped historical municipality +04543,#IgnoreRow,True,#ignore,unmapped historical municipality +04544,#IgnoreRow,True,#ignore,unmapped historical municipality +04545,#IgnoreRow,True,#ignore,unmapped historical municipality +04546,#IgnoreRow,True,#ignore,unmapped historical municipality +04547,#IgnoreRow,True,#ignore,unmapped historical municipality +04548,#IgnoreRow,True,#ignore,unmapped historical municipality +04561,#IgnoreRow,True,#ignore,unmapped historical municipality +04562,#IgnoreRow,True,#ignore,unmapped historical municipality +04563,#IgnoreRow,True,#ignore,unmapped historical municipality +04564,#IgnoreRow,True,#ignore,unmapped historical municipality +04565,#IgnoreRow,True,#ignore,unmapped historical municipality +04566,#IgnoreRow,True,#ignore,unmapped historical municipality +04567,#IgnoreRow,True,#ignore,unmapped historical municipality +04581,observationAbout,wikidataId/Q1011185 +04582,#IgnoreRow,True,#ignore,unmapped historical municipality +04601,#IgnoreRow,True,#ignore,unmapped historical municipality +04602,#IgnoreRow,True,#ignore,unmapped historical municipality +04603,#IgnoreRow,True,#ignore,unmapped historical municipality +04604,#IgnoreRow,True,#ignore,unmapped historical municipality +04605,#IgnoreRow,True,#ignore,unmapped historical municipality +04606,observationAbout,wikidataId/Q775888 +05000,observationAbout,wikidataId/Q81863 +05201,observationAbout,wikidataId/Q17139 +05202,observationAbout,wikidataId/Q819587 +05203,observationAbout,wikidataId/Q496479 +05204,observationAbout,wikidataId/Q329996 +05205,#IgnoreRow,True,#ignore,unmapped historical municipality +05206,observationAbout,wikidataId/Q633935 +05207,observationAbout,wikidataId/Q633272 +05208,#IgnoreRow,True,#ignore,unmapped historical municipality +05209,observationAbout,wikidataId/Q496499 +05210,observationAbout,wikidataId/Q602903 +05211,observationAbout,wikidataId/Q646780 +05212,observationAbout,wikidataId/Q695920 +05213,observationAbout,wikidataId/Q653545 +05214,observationAbout,wikidataId/Q847550 +05215,observationAbout,wikidataId/Q846877 +05303,observationAbout,wikidataId/Q1307166 +05321,#IgnoreRow,True,#ignore,unmapped historical municipality +05322,#IgnoreRow,True,#ignore,unmapped historical municipality +05323,#IgnoreRow,True,#ignore,unmapped historical municipality +05324,#IgnoreRow,True,#ignore,unmapped historical municipality +05325,#IgnoreRow,True,#ignore,unmapped historical municipality +05326,#IgnoreRow,True,#ignore,unmapped historical municipality +05327,observationAbout,wikidataId/Q1205482 +05341,#IgnoreRow,True,#ignore,unmapped historical municipality +05342,#IgnoreRow,True,#ignore,unmapped historical municipality +05343,#IgnoreRow,True,#ignore,unmapped historical municipality +05344,#IgnoreRow,True,#ignore,unmapped historical municipality +05345,#IgnoreRow,True,#ignore,unmapped historical municipality +05346,observationAbout,wikidataId/Q1206039 +05347,#IgnoreRow,True,#ignore,unmapped historical municipality +05348,observationAbout,wikidataId/Q1205383 +05349,observationAbout,wikidataId/Q1205948 +05361,observationAbout,wikidataId/Q1205432 +05362,#IgnoreRow,True,#ignore,unmapped historical municipality +05363,observationAbout,wikidataId/Q1205526 +05364,#IgnoreRow,True,#ignore,unmapped historical municipality +05365,#IgnoreRow,True,#ignore,unmapped historical municipality +05366,observationAbout,wikidataId/Q1205369 +05367,#IgnoreRow,True,#ignore,unmapped historical municipality +05368,observationAbout,wikidataId/Q427912 +05381,#IgnoreRow,True,#ignore,unmapped historical municipality +05382,#IgnoreRow,True,#ignore,unmapped historical municipality +05401,#IgnoreRow,True,#ignore,unmapped historical municipality +05402,#IgnoreRow,True,#ignore,unmapped historical municipality +05403,#IgnoreRow,True,#ignore,unmapped historical municipality +05404,#IgnoreRow,True,#ignore,unmapped historical municipality +05405,#IgnoreRow,True,#ignore,unmapped historical municipality +05406,#IgnoreRow,True,#ignore,unmapped historical municipality +05407,#IgnoreRow,True,#ignore,unmapped historical municipality +05408,#IgnoreRow,True,#ignore,unmapped historical municipality +05409,#IgnoreRow,True,#ignore,unmapped historical municipality +05410,#IgnoreRow,True,#ignore,unmapped historical municipality +05421,#IgnoreRow,True,#ignore,unmapped historical municipality +05422,#IgnoreRow,True,#ignore,unmapped historical municipality +05423,#IgnoreRow,True,#ignore,unmapped historical municipality +05424,#IgnoreRow,True,#ignore,unmapped historical municipality +05425,#IgnoreRow,True,#ignore,unmapped historical municipality +05426,#IgnoreRow,True,#ignore,unmapped historical municipality +05427,#IgnoreRow,True,#ignore,unmapped historical municipality +05428,#IgnoreRow,True,#ignore,unmapped historical municipality +05429,#IgnoreRow,True,#ignore,unmapped historical municipality +05430,#IgnoreRow,True,#ignore,unmapped historical municipality +05431,#IgnoreRow,True,#ignore,unmapped historical municipality +05432,#IgnoreRow,True,#ignore,unmapped historical municipality +05433,#IgnoreRow,True,#ignore,unmapped historical municipality +05434,observationAbout,wikidataId/Q1349377 +05441,#IgnoreRow,True,#ignore,unmapped historical municipality +05442,#IgnoreRow,True,#ignore,unmapped historical municipality +05443,#IgnoreRow,True,#ignore,unmapped historical municipality +05444,#IgnoreRow,True,#ignore,unmapped historical municipality +05445,#IgnoreRow,True,#ignore,unmapped historical municipality +05446,#IgnoreRow,True,#ignore,unmapped historical municipality +05447,#IgnoreRow,True,#ignore,unmapped historical municipality +05461,#IgnoreRow,True,#ignore,unmapped historical municipality +05462,#IgnoreRow,True,#ignore,unmapped historical municipality +05463,observationAbout,wikidataId/Q1206051 +05464,observationAbout,wikidataId/Q1205512 +05465,#IgnoreRow,True,#ignore,unmapped historical municipality +06000,observationAbout,wikidataId/Q125863 +06201,observationAbout,wikidataId/Q205526 +06202,observationAbout,wikidataId/Q734378 +06203,observationAbout,wikidataId/Q734416 +06204,observationAbout,wikidataId/Q504690 +06205,observationAbout,wikidataId/Q840603 +06206,observationAbout,wikidataId/Q823955 +06207,observationAbout,wikidataId/Q770187 +06208,observationAbout,wikidataId/Q858008 +06209,observationAbout,wikidataId/Q847557 +06210,observationAbout,wikidataId/Q385779 +06211,observationAbout,wikidataId/Q823927 +06212,observationAbout,wikidataId/Q846342 +06213,observationAbout,wikidataId/Q850496 +06301,observationAbout,wikidataId/Q1347748 +06302,observationAbout,wikidataId/Q1347364 +06321,observationAbout,wikidataId/Q1347734 +06322,observationAbout,wikidataId/Q1348988 +06323,observationAbout,wikidataId/Q1347249 +06324,observationAbout,wikidataId/Q377816 +06341,observationAbout,wikidataId/Q1347263 +06361,observationAbout,wikidataId/Q1347444 +06362,observationAbout,wikidataId/Q1348209 +06363,observationAbout,wikidataId/Q1347342 +06364,observationAbout,wikidataId/Q1349021 +06365,observationAbout,wikidataId/Q1347371 +06366,observationAbout,wikidataId/Q1347420 +06367,observationAbout,wikidataId/Q593586 +06381,observationAbout,wikidataId/Q772103 +06382,observationAbout,wikidataId/Q1349146 +06401,observationAbout,wikidataId/Q1347411 +06402,observationAbout,wikidataId/Q1354787 +06403,observationAbout,wikidataId/Q1347383 +06421,#IgnoreRow,True,#ignore,unmapped historical municipality +06422,#IgnoreRow,True,#ignore,unmapped historical municipality +06423,#IgnoreRow,True,#ignore,unmapped historical municipality +06424,#IgnoreRow,True,#ignore,unmapped historical municipality +06425,#IgnoreRow,True,#ignore,unmapped historical municipality +06426,observationAbout,wikidataId/Q1347453 +06427,#IgnoreRow,True,#ignore,unmapped historical municipality +06428,observationAbout,wikidataId/Q1348827 +06441,#IgnoreRow,True,#ignore,unmapped historical municipality +06461,observationAbout,wikidataId/Q1347481 +06462,#IgnoreRow,True,#ignore,unmapped historical municipality +06463,#IgnoreRow,True,#ignore,unmapped historical municipality +06464,#IgnoreRow,True,#ignore,unmapped historical municipality +07000,observationAbout,wikidataId/Q71707 +07201,observationAbout,wikidataId/Q161176 +07202,observationAbout,wikidataId/Q237699 +07203,observationAbout,wikidataId/Q264206 +07204,observationAbout,wikidataId/Q274384 +07205,observationAbout,wikidataId/Q750587 +07206,#IgnoreRow,True,#ignore,unmapped historical municipality +07207,observationAbout,wikidataId/Q819664 +07208,observationAbout,wikidataId/Q496506 +07209,observationAbout,wikidataId/Q696456 +07210,observationAbout,wikidataId/Q685491 +07211,observationAbout,wikidataId/Q734400 +07212,observationAbout,wikidataId/Q642094 +07213,observationAbout,wikidataId/Q858423 +07214,observationAbout,wikidataId/Q867106 +07301,observationAbout,wikidataId/Q1205184 +07302,#IgnoreRow,True,#ignore,unmapped historical municipality +07303,observationAbout,wikidataId/Q1205612 +07304,#IgnoreRow,True,#ignore,unmapped historical municipality +07305,#IgnoreRow,True,#ignore,unmapped historical municipality +07306,#IgnoreRow,True,#ignore,unmapped historical municipality +07307,#IgnoreRow,True,#ignore,unmapped historical municipality +07308,observationAbout,wikidataId/Q1347240 +07309,#IgnoreRow,True,#ignore,unmapped historical municipality +07321,#IgnoreRow,True,#ignore,unmapped historical municipality +07322,observationAbout,wikidataId/Q1205556 +07323,#IgnoreRow,True,#ignore,unmapped historical municipality +07324,#IgnoreRow,True,#ignore,unmapped historical municipality +07325,#IgnoreRow,True,#ignore,unmapped historical municipality +07326,#IgnoreRow,True,#ignore,unmapped historical municipality +07341,#IgnoreRow,True,#ignore,unmapped historical municipality +07342,observationAbout,wikidataId/Q1205431 +07343,#IgnoreRow,True,#ignore,unmapped historical municipality +07344,observationAbout,wikidataId/Q1205567 +07361,#IgnoreRow,True,#ignore,unmapped historical municipality +07362,observationAbout,wikidataId/Q1205491 +07363,#IgnoreRow,True,#ignore,unmapped historical municipality +07364,observationAbout,wikidataId/Q1205602 +07365,#IgnoreRow,True,#ignore,unmapped historical municipality +07366,#IgnoreRow,True,#ignore,unmapped historical municipality +07367,observationAbout,wikidataId/Q1205514 +07368,observationAbout,wikidataId/Q1205058 +07381,#IgnoreRow,True,#ignore,unmapped historical municipality +07401,#IgnoreRow,True,#ignore,unmapped historical municipality +07402,observationAbout,wikidataId/Q1205581 +07403,#IgnoreRow,True,#ignore,unmapped historical municipality +07404,#IgnoreRow,True,#ignore,unmapped historical municipality +07405,observationAbout,wikidataId/Q784402 +07406,#IgnoreRow,True,#ignore,unmapped historical municipality +07407,observationAbout,wikidataId/Q143208 +07408,observationAbout,wikidataId/Q669633 +07421,observationAbout,wikidataId/Q1204988 +07422,observationAbout,wikidataId/Q1205591 +07423,observationAbout,wikidataId/Q1205525 +07424,#IgnoreRow,True,#ignore,unmapped historical municipality +07441,#IgnoreRow,True,#ignore,unmapped historical municipality +07442,#IgnoreRow,True,#ignore,unmapped historical municipality +07443,#IgnoreRow,True,#ignore,unmapped historical municipality +07444,observationAbout,wikidataId/Q1205004 +07445,observationAbout,wikidataId/Q1205033 +07446,observationAbout,wikidataId/Q1205534 +07447,observationAbout,wikidataId/Q1205357 +07461,observationAbout,wikidataId/Q766027 +07462,#IgnoreRow,True,#ignore,unmapped historical municipality +07463,#IgnoreRow,True,#ignore,unmapped historical municipality +07464,observationAbout,wikidataId/Q1155676 +07465,observationAbout,wikidataId/Q370487 +07466,observationAbout,wikidataId/Q1205046 +07467,#IgnoreRow,True,#ignore,unmapped historical municipality +07481,observationAbout,wikidataId/Q1155858 +07482,observationAbout,wikidataId/Q1205018 +07483,observationAbout,wikidataId/Q1205922 +07484,observationAbout,wikidataId/Q1205497 +07485,#IgnoreRow,True,#ignore,unmapped historical municipality +07501,observationAbout,wikidataId/Q1205335 +07502,observationAbout,wikidataId/Q1205448 +07503,observationAbout,wikidataId/Q1205262 +07504,observationAbout,wikidataId/Q1205700 +07505,observationAbout,wikidataId/Q1205868 +07521,observationAbout,wikidataId/Q1348269 +07522,observationAbout,wikidataId/Q1205991 +07523,#IgnoreRow,True,#ignore,unmapped historical municipality +07524,#IgnoreRow,True,#ignore,unmapped historical municipality +07525,#IgnoreRow,True,#ignore,unmapped historical municipality +07526,#IgnoreRow,True,#ignore,unmapped historical municipality +07527,#IgnoreRow,True,#ignore,unmapped historical municipality +07541,observationAbout,wikidataId/Q1354828 +07542,observationAbout,wikidataId/Q1354867 +07543,observationAbout,wikidataId/Q1023943 +07544,observationAbout,wikidataId/Q1354949 +07545,observationAbout,wikidataId/Q385311 +07546,observationAbout,wikidataId/Q1023949 +07547,observationAbout,wikidataId/Q1023937 +07548,observationAbout,wikidataId/Q1352137 +07561,observationAbout,wikidataId/Q1205416 +07562,#IgnoreRow,True,#ignore,unmapped historical municipality +07563,#IgnoreRow,True,#ignore,unmapped historical municipality +07564,observationAbout,wikidataId/Q674500 +08000,observationAbout,wikidataId/Q83273 +08201,observationAbout,wikidataId/Q204249 +08202,observationAbout,wikidataId/Q633922 +08203,observationAbout,wikidataId/Q653446 +08204,observationAbout,wikidataId/Q836836 +08205,observationAbout,wikidataId/Q819658 +08206,#IgnoreRow,True,#ignore,unmapped historical municipality +08207,observationAbout,wikidataId/Q847569 +08208,observationAbout,wikidataId/Q844138 +08209,#IgnoreRow,True,#ignore,unmapped historical municipality +08210,observationAbout,wikidataId/Q533241 +08211,observationAbout,wikidataId/Q851149 +08212,observationAbout,wikidataId/Q838663 +08213,#IgnoreRow,True,#ignore,unmapped historical municipality +08214,observationAbout,wikidataId/Q520428 +08215,observationAbout,wikidataId/Q838654 +08216,observationAbout,wikidataId/Q845851 +08217,observationAbout,wikidataId/Q638312 +08218,#IgnoreRow,True,#ignore,unmapped historical municipality +08219,observationAbout,wikidataId/Q844133 +08220,observationAbout,wikidataId/Q273863 +08221,observationAbout,wikidataId/Q696772 +08222,observationAbout,wikidataId/Q467471 +08223,observationAbout,wikidataId/Q840598 +08224,observationAbout,wikidataId/Q840608 +08225,observationAbout,wikidataId/Q838681 +08226,observationAbout,wikidataId/Q633647 +08227,observationAbout,wikidataId/Q694322 +08228,observationAbout,wikidataId/Q583931 +08229,observationAbout,wikidataId/Q838675 +08230,observationAbout,wikidataId/Q772817 +08231,observationAbout,wikidataId/Q990350 +08232,observationAbout,wikidataId/Q840595 +08233,observationAbout,wikidataId/Q877184 +08234,observationAbout,wikidataId/Q857297 +08235,observationAbout,wikidataId/Q877254 +08236,observationAbout,wikidataId/Q430584 +08301,#IgnoreRow,True,#ignore,unmapped historical municipality +08302,observationAbout,wikidataId/Q1203390 +08303,#IgnoreRow,True,#ignore,unmapped historical municipality +08304,#IgnoreRow,True,#ignore,unmapped historical municipality +08305,#IgnoreRow,True,#ignore,unmapped historical municipality +08306,#IgnoreRow,True,#ignore,unmapped historical municipality +08307,#IgnoreRow,True,#ignore,unmapped historical municipality +08308,#IgnoreRow,True,#ignore,unmapped historical municipality +08309,observationAbout,wikidataId/Q1349185 +08310,observationAbout,wikidataId/Q1203068 +08321,#IgnoreRow,True,#ignore,unmapped historical municipality +08322,#IgnoreRow,True,#ignore,unmapped historical municipality +08323,#IgnoreRow,True,#ignore,unmapped historical municipality +08324,#IgnoreRow,True,#ignore,unmapped historical municipality +08341,observationAbout,wikidataId/Q1023784 +08342,#IgnoreRow,True,#ignore,unmapped historical municipality +08343,#IgnoreRow,True,#ignore,unmapped historical municipality +08344,#IgnoreRow,True,#ignore,unmapped historical municipality +08345,#IgnoreRow,True,#ignore,unmapped historical municipality +08346,#IgnoreRow,True,#ignore,unmapped historical municipality +08347,#IgnoreRow,True,#ignore,unmapped historical municipality +08361,#IgnoreRow,True,#ignore,unmapped historical municipality +08362,#IgnoreRow,True,#ignore,unmapped historical municipality +08363,#IgnoreRow,True,#ignore,unmapped historical municipality +08364,observationAbout,wikidataId/Q1346817 +08381,#IgnoreRow,True,#ignore,unmapped historical municipality +08401,#IgnoreRow,True,#ignore,unmapped historical municipality +08402,#IgnoreRow,True,#ignore,unmapped historical municipality +08403,#IgnoreRow,True,#ignore,unmapped historical municipality +08404,#IgnoreRow,True,#ignore,unmapped historical municipality +08405,#IgnoreRow,True,#ignore,unmapped historical municipality +08406,#IgnoreRow,True,#ignore,unmapped historical municipality +08407,#IgnoreRow,True,#ignore,unmapped historical municipality +08421,#IgnoreRow,True,#ignore,unmapped historical municipality +08422,#IgnoreRow,True,#ignore,unmapped historical municipality +08423,#IgnoreRow,True,#ignore,unmapped historical municipality +08424,#IgnoreRow,True,#ignore,unmapped historical municipality +08425,#IgnoreRow,True,#ignore,unmapped historical municipality +08441,#IgnoreRow,True,#ignore,unmapped historical municipality +08442,observationAbout,wikidataId/Q1203179 +08443,observationAbout,wikidataId/Q1203814 +08444,#IgnoreRow,True,#ignore,unmapped historical municipality +08445,#IgnoreRow,True,#ignore,unmapped historical municipality +08446,#IgnoreRow,True,#ignore,unmapped historical municipality +08447,observationAbout,wikidataId/Q1203304 +08448,#IgnoreRow,True,#ignore,unmapped historical municipality +08449,#IgnoreRow,True,#ignore,unmapped historical municipality +08461,#IgnoreRow,True,#ignore,unmapped historical municipality +08462,#IgnoreRow,True,#ignore,unmapped historical municipality +08463,#IgnoreRow,True,#ignore,unmapped historical municipality +08464,#IgnoreRow,True,#ignore,unmapped historical municipality +08465,#IgnoreRow,True,#ignore,unmapped historical municipality +08466,#IgnoreRow,True,#ignore,unmapped historical municipality +08481,#IgnoreRow,True,#ignore,unmapped historical municipality +08482,#IgnoreRow,True,#ignore,unmapped historical municipality +08483,#IgnoreRow,True,#ignore,unmapped historical municipality +08484,#IgnoreRow,True,#ignore,unmapped historical municipality +08485,#IgnoreRow,True,#ignore,unmapped historical municipality +08486,#IgnoreRow,True,#ignore,unmapped historical municipality +08501,#IgnoreRow,True,#ignore,unmapped historical municipality +08502,#IgnoreRow,True,#ignore,unmapped historical municipality +08503,#IgnoreRow,True,#ignore,unmapped historical municipality +08504,#IgnoreRow,True,#ignore,unmapped historical municipality +08505,#IgnoreRow,True,#ignore,unmapped historical municipality +08521,observationAbout,wikidataId/Q1203394 +08522,#IgnoreRow,True,#ignore,unmapped historical municipality +08523,#IgnoreRow,True,#ignore,unmapped historical municipality +08541,#IgnoreRow,True,#ignore,unmapped historical municipality +08542,observationAbout,wikidataId/Q680056 +08543,#IgnoreRow,True,#ignore,unmapped historical municipality +08544,#IgnoreRow,True,#ignore,unmapped historical municipality +08546,observationAbout,wikidataId/Q1203337 +08561,#IgnoreRow,True,#ignore,unmapped historical municipality +08563,#IgnoreRow,True,#ignore,unmapped historical municipality +08564,observationAbout,wikidataId/Q1203324 +09000,observationAbout,wikidataId/Q44843 +09201,observationAbout,wikidataId/Q200589 +09202,observationAbout,wikidataId/Q374477 +09203,observationAbout,wikidataId/Q44830 +09204,observationAbout,wikidataId/Q646912 +09205,observationAbout,wikidataId/Q390792 +09206,observationAbout,wikidataId/Q235753 +09207,#IgnoreRow,True,#ignore,unmapped historical municipality +09208,observationAbout,wikidataId/Q374511 +09209,observationAbout,wikidataId/Q498060 +09210,observationAbout,wikidataId/Q386322 +09211,observationAbout,wikidataId/Q695926 +09212,#IgnoreRow,True,#ignore,unmapped historical municipality +09213,observationAbout,wikidataId/Q498027 +09214,observationAbout,wikidataId/Q696394 +09215,observationAbout,wikidataId/Q847562 +09216,observationAbout,wikidataId/Q851804 +09301,observationAbout,wikidataId/Q1345604 +09302,#IgnoreRow,True,#ignore,unmapped historical municipality +09303,#IgnoreRow,True,#ignore,unmapped historical municipality +09304,#IgnoreRow,True,#ignore,unmapped historical municipality +09321,#IgnoreRow,True,#ignore,unmapped historical municipality +09322,#IgnoreRow,True,#ignore,unmapped historical municipality +09323,#IgnoreRow,True,#ignore,unmapped historical municipality +09341,#IgnoreRow,True,#ignore,unmapped historical municipality +09342,observationAbout,wikidataId/Q1346925 +09343,observationAbout,wikidataId/Q644327 +09344,observationAbout,wikidataId/Q577654 +09345,observationAbout,wikidataId/Q1203553 +09361,observationAbout,wikidataId/Q1354910 +09362,#IgnoreRow,True,#ignore,unmapped historical municipality +09363,#IgnoreRow,True,#ignore,unmapped historical municipality +09364,observationAbout,wikidataId/Q1203180 +09365,#IgnoreRow,True,#ignore,unmapped historical municipality +09366,#IgnoreRow,True,#ignore,unmapped historical municipality +09367,observationAbout,wikidataId/Q1203561 +09368,#IgnoreRow,True,#ignore,unmapped historical municipality +09381,#IgnoreRow,True,#ignore,unmapped historical municipality +09382,#IgnoreRow,True,#ignore,unmapped historical municipality +09383,#IgnoreRow,True,#ignore,unmapped historical municipality +09384,observationAbout,wikidataId/Q1203439 +09385,#IgnoreRow,True,#ignore,unmapped historical municipality +09386,observationAbout,wikidataId/Q648859 +09387,#IgnoreRow,True,#ignore,unmapped historical municipality +09401,#IgnoreRow,True,#ignore,unmapped historical municipality +09402,#IgnoreRow,True,#ignore,unmapped historical municipality +09403,#IgnoreRow,True,#ignore,unmapped historical municipality +09404,#IgnoreRow,True,#ignore,unmapped historical municipality +09405,#IgnoreRow,True,#ignore,unmapped historical municipality +09406,#IgnoreRow,True,#ignore,unmapped historical municipality +09407,observationAbout,wikidataId/Q636416 +09409,#IgnoreRow,True,#ignore,unmapped historical municipality +09410,#IgnoreRow,True,#ignore,unmapped historical municipality +09411,observationAbout,wikidataId/Q44603 +09421,#IgnoreRow,True,#ignore,unmapped historical municipality +09422,#IgnoreRow,True,#ignore,unmapped historical municipality +10000,observationAbout,wikidataId/Q129499 +10201,observationAbout,wikidataId/Q201613 +10202,observationAbout,wikidataId/Q336438 +10203,observationAbout,wikidataId/Q634022 +10204,observationAbout,wikidataId/Q328596 +10205,observationAbout,wikidataId/Q386179 +10206,observationAbout,wikidataId/Q666317 +10207,observationAbout,wikidataId/Q858016 +10208,observationAbout,wikidataId/Q851492 +10209,observationAbout,wikidataId/Q596486 +10210,observationAbout,wikidataId/Q873835 +10211,observationAbout,wikidataId/Q564234 +10212,observationAbout,wikidataId/Q576556 +10301,#IgnoreRow,True,#ignore,unmapped historical municipality +10302,#IgnoreRow,True,#ignore,unmapped historical municipality +10303,#IgnoreRow,True,#ignore,unmapped historical municipality +10304,#IgnoreRow,True,#ignore,unmapped historical municipality +10305,observationAbout,wikidataId/Q2068514 +10306,#IgnoreRow,True,#ignore,unmapped historical municipality +10307,#IgnoreRow,True,#ignore,unmapped historical municipality +10308,#IgnoreRow,True,#ignore,unmapped historical municipality +10309,#IgnoreRow,True,#ignore,unmapped historical municipality +10321,#IgnoreRow,True,#ignore,unmapped historical municipality +10322,#IgnoreRow,True,#ignore,unmapped historical municipality +10323,#IgnoreRow,True,#ignore,unmapped historical municipality +10324,#IgnoreRow,True,#ignore,unmapped historical municipality +10341,#IgnoreRow,True,#ignore,unmapped historical municipality +10342,#IgnoreRow,True,#ignore,unmapped historical municipality +10343,#IgnoreRow,True,#ignore,unmapped historical municipality +10344,observationAbout,wikidataId/Q1203698 +10345,observationAbout,wikidataId/Q1154580 +10361,#IgnoreRow,True,#ignore,unmapped historical municipality +10362,#IgnoreRow,True,#ignore,unmapped historical municipality +10363,#IgnoreRow,True,#ignore,unmapped historical municipality +10364,#IgnoreRow,True,#ignore,unmapped historical municipality +10365,#IgnoreRow,True,#ignore,unmapped historical municipality +10366,observationAbout,wikidataId/Q1346914 +10367,observationAbout,wikidataId/Q1203059 +10381,#IgnoreRow,True,#ignore,unmapped historical municipality +10382,observationAbout,wikidataId/Q1346871 +10383,observationAbout,wikidataId/Q374984 +10384,observationAbout,wikidataId/Q1203523 +10401,#IgnoreRow,True,#ignore,unmapped historical municipality +10421,observationAbout,wikidataId/Q967920 +10422,#IgnoreRow,True,#ignore,unmapped historical municipality +10423,#IgnoreRow,True,#ignore,unmapped historical municipality +10424,observationAbout,wikidataId/Q1203366 +10425,observationAbout,wikidataId/Q1203321 +10426,observationAbout,wikidataId/Q1358949 +10427,#IgnoreRow,True,#ignore,unmapped historical municipality +10428,observationAbout,wikidataId/Q1203891 +10429,observationAbout,wikidataId/Q1203419 +10441,#IgnoreRow,True,#ignore,unmapped historical municipality +10442,#IgnoreRow,True,#ignore,unmapped historical municipality +10443,observationAbout,wikidataId/Q1203661 +10444,observationAbout,wikidataId/Q1345636 +10445,#IgnoreRow,True,#ignore,unmapped historical municipality +10446,#IgnoreRow,True,#ignore,unmapped historical municipality +10447,#IgnoreRow,True,#ignore,unmapped historical municipality +10448,observationAbout,wikidataId/Q1203851 +10449,observationAbout,wikidataId/Q1345518 +10461,#IgnoreRow,True,#ignore,unmapped historical municipality +10462,#IgnoreRow,True,#ignore,unmapped historical municipality +10463,#IgnoreRow,True,#ignore,unmapped historical municipality +10464,observationAbout,wikidataId/Q1203600 +10481,#IgnoreRow,True,#ignore,unmapped historical municipality +10482,#IgnoreRow,True,#ignore,unmapped historical municipality +10483,#IgnoreRow,True,#ignore,unmapped historical municipality +10484,#IgnoreRow,True,#ignore,unmapped historical municipality +10501,#IgnoreRow,True,#ignore,unmapped historical municipality +10521,observationAbout,wikidataId/Q1203771 +10522,observationAbout,wikidataId/Q1203274 +10523,observationAbout,wikidataId/Q1023773 +10524,observationAbout,wikidataId/Q385080 +10525,observationAbout,wikidataId/Q656774 +11000,observationAbout,wikidataId/Q128186 +11100,observationAbout,wikidataId/Q134635 +11101,observationAbout,wikidataId/Q1075040 +11102,observationAbout,wikidataId/Q542542 +11103,observationAbout,wikidataId/Q153629 +11104,observationAbout,wikidataId/Q1154570 +11105,observationAbout,wikidataId/Q976945 +11106,observationAbout,wikidataId/Q1062270 +11107,observationAbout,wikidataId/Q1154583 +11108,observationAbout,wikidataId/Q1029251 +11109,observationAbout,wikidataId/Q1130733 +11110,observationAbout,wikidataId/Q1184508 +11201,observationAbout,wikidataId/Q319740 +11202,observationAbout,wikidataId/Q41106 +11203,observationAbout,wikidataId/Q387136 +11204,observationAbout,wikidataId/Q1076743 +11205,#IgnoreRow,True,#ignore,unmapped historical municipality +11206,observationAbout,wikidataId/Q860659 +11207,observationAbout,wikidataId/Q647079 +11208,observationAbout,wikidataId/Q165857 +11209,observationAbout,wikidataId/Q850472 +11210,observationAbout,wikidataId/Q851680 +11211,observationAbout,wikidataId/Q873565 +11212,observationAbout,wikidataId/Q823907 +11213,#IgnoreRow,True,#ignore,unmapped historical municipality +11214,observationAbout,wikidataId/Q376023 +11215,observationAbout,wikidataId/Q824009 +11216,observationAbout,wikidataId/Q847565 +11217,observationAbout,wikidataId/Q734541 +11218,observationAbout,wikidataId/Q734532 +11219,observationAbout,wikidataId/Q163888 +11220,#IgnoreRow,True,#ignore,unmapped historical municipality +11221,observationAbout,wikidataId/Q734442 +11222,observationAbout,wikidataId/Q497952 +11223,observationAbout,wikidataId/Q673010 +11224,observationAbout,wikidataId/Q823874 +11225,observationAbout,wikidataId/Q696925 +11226,#IgnoreRow,True,#ignore,unmapped historical municipality +11227,observationAbout,wikidataId/Q646834 +11228,observationAbout,wikidataId/Q877202 +11229,observationAbout,wikidataId/Q658058 +11230,observationAbout,wikidataId/Q386691 +11231,observationAbout,wikidataId/Q570333 +11232,observationAbout,wikidataId/Q47535 +11233,observationAbout,wikidataId/Q857285 +11234,observationAbout,wikidataId/Q869032 +11235,observationAbout,wikidataId/Q495809 +11236,#IgnoreRow,True,#ignore,unmapped historical municipality +11237,observationAbout,wikidataId/Q853061 +11238,observationAbout,wikidataId/Q857256 +11239,observationAbout,wikidataId/Q850344 +11240,observationAbout,wikidataId/Q868971 +11241,observationAbout,wikidataId/Q869027 +11242,observationAbout,wikidataId/Q905721 +11243,observationAbout,wikidataId/Q873546 +11245,observationAbout,wikidataId/Q873590 +11246,observationAbout,wikidataId/Q1345322 +11301,observationAbout,wikidataId/Q915773 +11304,#IgnoreRow,True,#ignore,unmapped historical municipality +11322,#IgnoreRow,True,#ignore,unmapped historical municipality +11324,observationAbout,wikidataId/Q185238 +11326,observationAbout,wikidataId/Q1354799 +11327,observationAbout,wikidataId/Q783333 +11328,#IgnoreRow,True,#ignore,unmapped historical municipality +11329,#IgnoreRow,True,#ignore,unmapped historical municipality +11330,#IgnoreRow,True,#ignore,unmapped historical municipality +11341,observationAbout,wikidataId/Q1346985 +11342,observationAbout,wikidataId/Q1358642 +11343,observationAbout,wikidataId/Q935037 +11344,#IgnoreRow,True,#ignore,unmapped historical municipality +11345,#IgnoreRow,True,#ignore,unmapped historical municipality +11346,observationAbout,wikidataId/Q1203970 +11347,observationAbout,wikidataId/Q543313 +11348,observationAbout,wikidataId/Q925875 +11349,observationAbout,wikidataId/Q251027 +11361,observationAbout,wikidataId/Q1345277 +11362,observationAbout,wikidataId/Q1345365 +11363,observationAbout,wikidataId/Q1345299 +11364,#IgnoreRow,True,#ignore,unmapped historical municipality +11365,observationAbout,wikidataId/Q1345311 +11366,#IgnoreRow,True,#ignore,unmapped historical municipality +11367,#IgnoreRow,True,#ignore,unmapped historical municipality +11368,#IgnoreRow,True,#ignore,unmapped historical municipality +11369,observationAbout,wikidataId/Q1357435 +11381,observationAbout,wikidataId/Q924129 +11382,#IgnoreRow,True,#ignore,unmapped historical municipality +11383,observationAbout,wikidataId/Q1346757 +11384,#IgnoreRow,True,#ignore,unmapped historical municipality +11385,observationAbout,wikidataId/Q1345287 +11401,#IgnoreRow,True,#ignore,unmapped historical municipality +11402,#IgnoreRow,True,#ignore,unmapped historical municipality +11403,#IgnoreRow,True,#ignore,unmapped historical municipality +11405,#IgnoreRow,True,#ignore,unmapped historical municipality +11406,#IgnoreRow,True,#ignore,unmapped historical municipality +11407,#IgnoreRow,True,#ignore,unmapped historical municipality +11408,observationAbout,wikidataId/Q961604 +11421,#IgnoreRow,True,#ignore,unmapped historical municipality +11422,#IgnoreRow,True,#ignore,unmapped historical municipality +11423,#IgnoreRow,True,#ignore,unmapped historical municipality +11424,#IgnoreRow,True,#ignore,unmapped historical municipality +11425,#IgnoreRow,True,#ignore,unmapped historical municipality +11442,observationAbout,wikidataId/Q155317 +11445,#IgnoreRow,True,#ignore,unmapped historical municipality +11446,#IgnoreRow,True,#ignore,unmapped historical municipality +11461,#IgnoreRow,True,#ignore,unmapped historical municipality +11462,#IgnoreRow,True,#ignore,unmapped historical municipality +11463,#IgnoreRow,True,#ignore,unmapped historical municipality +11464,observationAbout,wikidataId/Q1345351 +11465,observationAbout,wikidataId/Q1346963 +11466,#IgnoreRow,True,#ignore,unmapped historical municipality +11468,#IgnoreRow,True,#ignore,unmapped historical municipality +12000,observationAbout,wikidataId/Q80011 +12100,observationAbout,wikidataId/Q170616 +12101,observationAbout,wikidataId/Q1075869 +12102,observationAbout,wikidataId/Q1075882 +12103,observationAbout,wikidataId/Q1075905 +12104,observationAbout,wikidataId/Q1075889 +12105,observationAbout,wikidataId/Q926492 +12106,observationAbout,wikidataId/Q1075859 +12201,#IgnoreRow,True,#ignore,unmapped historical municipality +12202,observationAbout,wikidataId/Q694317 +12203,observationAbout,wikidataId/Q209785 +12204,observationAbout,wikidataId/Q271417 +12205,observationAbout,wikidataId/Q646755 +12206,observationAbout,wikidataId/Q335427 +12207,observationAbout,wikidataId/Q320943 +12208,observationAbout,wikidataId/Q328619 +12209,#IgnoreRow,True,#ignore,unmapped historical municipality +12210,observationAbout,wikidataId/Q502486 +12211,observationAbout,wikidataId/Q273798 +12212,observationAbout,wikidataId/Q498011 +12213,observationAbout,wikidataId/Q637913 +12214,#IgnoreRow,True,#ignore,unmapped historical municipality +12215,observationAbout,wikidataId/Q637868 +12216,observationAbout,wikidataId/Q318047 +12217,observationAbout,wikidataId/Q319743 +12218,observationAbout,wikidataId/Q696418 +12219,observationAbout,wikidataId/Q207881 +12220,observationAbout,wikidataId/Q328625 +12221,observationAbout,wikidataId/Q328651 +12222,observationAbout,wikidataId/Q243859 +12223,observationAbout,wikidataId/Q387211 +12224,observationAbout,wikidataId/Q387365 +12225,observationAbout,wikidataId/Q637928 +12226,observationAbout,wikidataId/Q498023 +12227,observationAbout,wikidataId/Q319729 +12228,observationAbout,wikidataId/Q429373 +12229,observationAbout,wikidataId/Q646847 +12230,observationAbout,wikidataId/Q638296 +12231,observationAbout,wikidataId/Q637901 +12232,observationAbout,wikidataId/Q519187 +12233,observationAbout,wikidataId/Q692857 +12234,observationAbout,wikidataId/Q873521 +12235,observationAbout,wikidataId/Q858367 +12236,observationAbout,wikidataId/Q858357 +12237,observationAbout,wikidataId/Q873528 +12238,observationAbout,wikidataId/Q840834 +12239,observationAbout,wikidataId/Q1203251 +12301,#IgnoreRow,True,#ignore,unmapped historical municipality +12303,#IgnoreRow,True,#ignore,unmapped historical municipality +12305,#IgnoreRow,True,#ignore,unmapped historical municipality +12321,#IgnoreRow,True,#ignore,unmapped historical municipality +12322,observationAbout,wikidataId/Q782157 +12323,#IgnoreRow,True,#ignore,unmapped historical municipality +12324,#IgnoreRow,True,#ignore,unmapped historical municipality +12325,#IgnoreRow,True,#ignore,unmapped historical municipality +12326,#IgnoreRow,True,#ignore,unmapped historical municipality +12327,#IgnoreRow,True,#ignore,unmapped historical municipality +12328,#IgnoreRow,True,#ignore,unmapped historical municipality +12329,observationAbout,wikidataId/Q1202754 +12341,#IgnoreRow,True,#ignore,unmapped historical municipality +12342,observationAbout,wikidataId/Q1153721 +12343,#IgnoreRow,True,#ignore,unmapped historical municipality +12344,#IgnoreRow,True,#ignore,unmapped historical municipality +12345,#IgnoreRow,True,#ignore,unmapped historical municipality +12346,#IgnoreRow,True,#ignore,unmapped historical municipality +12347,observationAbout,wikidataId/Q1202807 +12348,#IgnoreRow,True,#ignore,unmapped historical municipality +12349,observationAbout,wikidataId/Q1342834 +12361,#IgnoreRow,True,#ignore,unmapped historical municipality +12362,#IgnoreRow,True,#ignore,unmapped historical municipality +12381,#IgnoreRow,True,#ignore,unmapped historical municipality +12382,#IgnoreRow,True,#ignore,unmapped historical municipality +12402,#IgnoreRow,True,#ignore,unmapped historical municipality +12403,observationAbout,wikidataId/Q1202775 +12404,#IgnoreRow,True,#ignore,unmapped historical municipality +12405,#IgnoreRow,True,#ignore,unmapped historical municipality +12406,#IgnoreRow,True,#ignore,unmapped historical municipality +12407,#IgnoreRow,True,#ignore,unmapped historical municipality +12408,#IgnoreRow,True,#ignore,unmapped historical municipality +12409,observationAbout,wikidataId/Q1203689 +12410,observationAbout,wikidataId/Q1203260 +12421,observationAbout,wikidataId/Q1203602 +12422,observationAbout,wikidataId/Q768083 +12423,observationAbout,wikidataId/Q1202647 +12424,observationAbout,wikidataId/Q1203293 +12426,observationAbout,wikidataId/Q1203545 +12427,observationAbout,wikidataId/Q1203194 +12441,observationAbout,wikidataId/Q1346191 +12442,#IgnoreRow,True,#ignore,unmapped historical municipality +12443,observationAbout,wikidataId/Q648466 +12444,#IgnoreRow,True,#ignore,unmapped historical municipality +12445,#IgnoreRow,True,#ignore,unmapped historical municipality +12461,#IgnoreRow,True,#ignore,unmapped historical municipality +12462,#IgnoreRow,True,#ignore,unmapped historical municipality +12463,observationAbout,wikidataId/Q1203580 +12464,#IgnoreRow,True,#ignore,unmapped historical municipality +12465,#IgnoreRow,True,#ignore,unmapped historical municipality +12466,#IgnoreRow,True,#ignore,unmapped historical municipality +12467,#IgnoreRow,True,#ignore,unmapped historical municipality +12468,#IgnoreRow,True,#ignore,unmapped historical municipality +12472,#IgnoreRow,True,#ignore,unmapped historical municipality +12481,#IgnoreRow,True,#ignore,unmapped historical municipality +13000,observationAbout,wikidataId/Q1490 +13100,#IgnoreRow,True,#ignore,unmapped historical municipality +13101,observationAbout,wikidataId/Q214051 +13102,observationAbout,wikidataId/Q212704 +13103,observationAbout,wikidataId/Q190088 +13104,observationAbout,wikidataId/Q179645 +13105,observationAbout,wikidataId/Q212713 +13106,observationAbout,wikidataId/Q232641 +13107,observationAbout,wikidataId/Q235135 +13108,observationAbout,wikidataId/Q215175 +13109,observationAbout,wikidataId/Q233495 +13110,observationAbout,wikidataId/Q233903 +13111,observationAbout,wikidataId/Q217234 +13112,observationAbout,wikidataId/Q231645 +13113,observationAbout,wikidataId/Q193638 +13114,observationAbout,wikidataId/Q234087 +13115,observationAbout,wikidataId/Q232631 +13116,observationAbout,wikidataId/Q236680 +13117,observationAbout,wikidataId/Q235130 +13118,observationAbout,wikidataId/Q232624 +13119,observationAbout,wikidataId/Q232635 +13120,observationAbout,wikidataId/Q232655 +13121,observationAbout,wikidataId/Q213464 +13122,observationAbout,wikidataId/Q232628 +13123,observationAbout,wikidataId/Q214056 +13201,observationAbout,wikidataId/Q208863 +13202,observationAbout,wikidataId/Q269634 +13203,observationAbout,wikidataId/Q210661 +13204,observationAbout,wikidataId/Q242885 +13205,observationAbout,wikidataId/Q237683 +13206,observationAbout,wikidataId/Q208818 +13207,observationAbout,wikidataId/Q212494 +13208,observationAbout,wikidataId/Q210667 +13209,observationAbout,wikidataId/Q210628 +13210,observationAbout,wikidataId/Q242874 +13211,observationAbout,wikidataId/Q213278 +13212,observationAbout,wikidataId/Q236197 +13213,observationAbout,wikidataId/Q240489 +13214,observationAbout,wikidataId/Q317834 +13215,observationAbout,wikidataId/Q330007 +13216,#IgnoreRow,True,#ignore,unmapped historical municipality +13217,#IgnoreRow,True,#ignore,unmapped historical municipality +13218,observationAbout,wikidataId/Q317813 +13219,observationAbout,wikidataId/Q86536 +13220,observationAbout,wikidataId/Q390788 +13221,observationAbout,wikidataId/Q329754 +13222,observationAbout,wikidataId/Q317830 +13223,observationAbout,wikidataId/Q390806 +13224,observationAbout,wikidataId/Q213263 +13225,observationAbout,wikidataId/Q317847 +13226,#IgnoreRow,True,#ignore,unmapped historical municipality +13227,observationAbout,wikidataId/Q270465 +13228,observationAbout,wikidataId/Q240480 +13229,observationAbout,wikidataId/Q242882 +13302,#IgnoreRow,True,#ignore,unmapped historical municipality +13303,observationAbout,wikidataId/Q975797 +13305,observationAbout,wikidataId/Q1154806 +13306,#IgnoreRow,True,#ignore,unmapped historical municipality +13307,observationAbout,wikidataId/Q219576 +13308,observationAbout,wikidataId/Q598626 +13361,observationAbout,wikidataId/Q1204507 +13362,observationAbout,wikidataId/Q1143112 +13363,observationAbout,wikidataId/Q1204534 +13364,observationAbout,wikidataId/Q1346527 +13381,observationAbout,wikidataId/Q1154797 +13382,observationAbout,wikidataId/Q1354993 +13401,observationAbout,wikidataId/Q1354815 +13402,observationAbout,wikidataId/Q1196636 +13421,observationAbout,wikidataId/Q1022925 +14000,observationAbout,wikidataId/Q127513 +14100,observationAbout,wikidataId/Q38283 +14101,observationAbout,wikidataId/Q1202820 +14102,observationAbout,wikidataId/Q1143712 +14103,observationAbout,wikidataId/Q1203393 +14104,observationAbout,wikidataId/Q1141068 +14105,observationAbout,wikidataId/Q1153982 +14106,observationAbout,wikidataId/Q1192964 +14107,observationAbout,wikidataId/Q1141223 +14108,observationAbout,wikidataId/Q1153908 +14109,observationAbout,wikidataId/Q1358965 +14110,observationAbout,wikidataId/Q524038 +14111,observationAbout,wikidataId/Q1153968 +14112,observationAbout,wikidataId/Q1153953 +14113,observationAbout,wikidataId/Q1203373 +14114,observationAbout,wikidataId/Q1153930 +14115,observationAbout,wikidataId/Q1153916 +14116,observationAbout,wikidataId/Q1203386 +14117,observationAbout,wikidataId/Q1203361 +14118,observationAbout,wikidataId/Q1153920 +14130,observationAbout,wikidataId/Q164234 +14131,observationAbout,wikidataId/Q1204949 +14132,observationAbout,wikidataId/Q1153579 +14133,observationAbout,wikidataId/Q1205359 +14134,observationAbout,wikidataId/Q1076736 +14135,observationAbout,wikidataId/Q1076769 +14136,observationAbout,wikidataId/Q1076753 +14137,observationAbout,wikidataId/Q1041779 +14150,observationAbout,wikidataId/Q209779 +14151,observationAbout,wikidataId/Q1132502 +14152,observationAbout,wikidataId/Q1132074 +14153,observationAbout,wikidataId/Q1132081 +14201,observationAbout,wikidataId/Q200578 +14203,observationAbout,wikidataId/Q502199 +14204,observationAbout,wikidataId/Q200267 +14205,observationAbout,wikidataId/Q386661 +14206,observationAbout,wikidataId/Q267258 +14207,observationAbout,wikidataId/Q633911 +14208,observationAbout,wikidataId/Q230843 +14209,#IgnoreRow,True,#ignore,unmapped historical municipality +14210,observationAbout,wikidataId/Q823886 +14211,observationAbout,wikidataId/Q460806 +14212,observationAbout,wikidataId/Q389711 +14213,observationAbout,wikidataId/Q386697 +14214,observationAbout,wikidataId/Q734520 +14215,observationAbout,wikidataId/Q734390 +14216,observationAbout,wikidataId/Q145333 +14217,observationAbout,wikidataId/Q651742 +14218,observationAbout,wikidataId/Q792819 +14301,observationAbout,wikidataId/Q1342807 +14321,observationAbout,wikidataId/Q1342787 +14341,observationAbout,wikidataId/Q385023 +14342,observationAbout,wikidataId/Q1354853 +14361,observationAbout,wikidataId/Q1344396 +14362,observationAbout,wikidataId/Q1202616 +14363,observationAbout,wikidataId/Q1202796 +14364,observationAbout,wikidataId/Q370478 +14366,observationAbout,wikidataId/Q1202613 +14382,observationAbout,wikidataId/Q671040 +14383,observationAbout,wikidataId/Q1202786 +14384,observationAbout,wikidataId/Q532346 +14401,observationAbout,wikidataId/Q1203267 +14402,observationAbout,wikidataId/Q1344386 +14421,#IgnoreRow,True,#ignore,unmapped historical municipality +14422,#IgnoreRow,True,#ignore,unmapped historical municipality +14423,#IgnoreRow,True,#ignore,unmapped historical municipality +14424,#IgnoreRow,True,#ignore,unmapped historical municipality +15000,observationAbout,wikidataId/Q132705 +15100,observationAbout,wikidataId/Q179691 +15101,observationAbout,wikidataId/Q711739 +15102,observationAbout,wikidataId/Q711707 +15103,observationAbout,wikidataId/Q711787 +15104,observationAbout,wikidataId/Q711744 +15105,observationAbout,wikidataId/Q711733 +15106,observationAbout,wikidataId/Q711693 +15107,observationAbout,wikidataId/Q711716 +15108,observationAbout,wikidataId/Q711756 +15201,#IgnoreRow,True,#ignore,unmapped historical municipality +15202,observationAbout,wikidataId/Q540312 +15204,observationAbout,wikidataId/Q817273 +15205,observationAbout,wikidataId/Q633983 +15206,observationAbout,wikidataId/Q601950 +15207,#IgnoreRow,True,#ignore,unmapped historical municipality +15208,observationAbout,wikidataId/Q819174 +15209,observationAbout,wikidataId/Q857293 +15210,observationAbout,wikidataId/Q858369 +15211,observationAbout,wikidataId/Q649650 +15212,observationAbout,wikidataId/Q284503 +15213,observationAbout,wikidataId/Q571051 +15215,#IgnoreRow,True,#ignore,unmapped historical municipality +15216,#IgnoreRow,True,#ignore,unmapped historical municipality +15217,observationAbout,wikidataId/Q279642 +15218,observationAbout,wikidataId/Q858364 +15219,#IgnoreRow,True,#ignore,unmapped historical municipality +15220,#IgnoreRow,True,#ignore,unmapped historical municipality +15221,#IgnoreRow,True,#ignore,unmapped historical municipality +15222,observationAbout,wikidataId/Q582289 +15223,observationAbout,wikidataId/Q390466 +15224,observationAbout,wikidataId/Q499047 +15225,observationAbout,wikidataId/Q850352 +15226,observationAbout,wikidataId/Q850524 +15227,observationAbout,wikidataId/Q877266 +15301,#IgnoreRow,True,#ignore,unmapped historical municipality +15302,#IgnoreRow,True,#ignore,unmapped historical municipality +15303,#IgnoreRow,True,#ignore,unmapped historical municipality +15304,#IgnoreRow,True,#ignore,unmapped historical municipality +15306,#IgnoreRow,True,#ignore,unmapped historical municipality +15307,observationAbout,wikidataId/Q1204293 +15308,#IgnoreRow,True,#ignore,unmapped historical municipality +15309,#IgnoreRow,True,#ignore,unmapped historical municipality +15310,#IgnoreRow,True,#ignore,unmapped historical municipality +15311,#IgnoreRow,True,#ignore,unmapped historical municipality +15321,#IgnoreRow,True,#ignore,unmapped historical municipality +15322,#IgnoreRow,True,#ignore,unmapped historical municipality +15323,#IgnoreRow,True,#ignore,unmapped historical municipality +15324,#IgnoreRow,True,#ignore,unmapped historical municipality +15341,#IgnoreRow,True,#ignore,unmapped historical municipality +15342,observationAbout,wikidataId/Q1349667 +15343,#IgnoreRow,True,#ignore,unmapped historical municipality +15344,#IgnoreRow,True,#ignore,unmapped historical municipality +15345,#IgnoreRow,True,#ignore,unmapped historical municipality +15346,#IgnoreRow,True,#ignore,unmapped historical municipality +15347,#IgnoreRow,True,#ignore,unmapped historical municipality +15348,#IgnoreRow,True,#ignore,unmapped historical municipality +15349,#IgnoreRow,True,#ignore,unmapped historical municipality +15350,#IgnoreRow,True,#ignore,unmapped historical municipality +15351,#IgnoreRow,True,#ignore,unmapped historical municipality +15361,observationAbout,wikidataId/Q1155155 +15362,#IgnoreRow,True,#ignore,unmapped historical municipality +15363,#IgnoreRow,True,#ignore,unmapped historical municipality +15364,#IgnoreRow,True,#ignore,unmapped historical municipality +15381,#IgnoreRow,True,#ignore,unmapped historical municipality +15382,#IgnoreRow,True,#ignore,unmapped historical municipality +15383,#IgnoreRow,True,#ignore,unmapped historical municipality +15384,#IgnoreRow,True,#ignore,unmapped historical municipality +15385,observationAbout,wikidataId/Q1204671 +15401,#IgnoreRow,True,#ignore,unmapped historical municipality +15402,#IgnoreRow,True,#ignore,unmapped historical municipality +15403,#IgnoreRow,True,#ignore,unmapped historical municipality +15404,#IgnoreRow,True,#ignore,unmapped historical municipality +15405,observationAbout,wikidataId/Q1204583 +15406,#IgnoreRow,True,#ignore,unmapped historical municipality +15421,#IgnoreRow,True,#ignore,unmapped historical municipality +15441,#IgnoreRow,True,#ignore,unmapped historical municipality +15442,#IgnoreRow,True,#ignore,unmapped historical municipality +15443,#IgnoreRow,True,#ignore,unmapped historical municipality +15444,#IgnoreRow,True,#ignore,unmapped historical municipality +15445,#IgnoreRow,True,#ignore,unmapped historical municipality +15446,#IgnoreRow,True,#ignore,unmapped historical municipality +15447,#IgnoreRow,True,#ignore,unmapped historical municipality +15461,observationAbout,wikidataId/Q1204269 +15462,#IgnoreRow,True,#ignore,unmapped historical municipality +15463,#IgnoreRow,True,#ignore,unmapped historical municipality +15464,#IgnoreRow,True,#ignore,unmapped historical municipality +15481,#IgnoreRow,True,#ignore,unmapped historical municipality +15482,observationAbout,wikidataId/Q1204292 +15483,#IgnoreRow,True,#ignore,unmapped historical municipality +15501,#IgnoreRow,True,#ignore,unmapped historical municipality +15502,#IgnoreRow,True,#ignore,unmapped historical municipality +15504,observationAbout,wikidataId/Q1347334 +15505,#IgnoreRow,True,#ignore,unmapped historical municipality +15521,#IgnoreRow,True,#ignore,unmapped historical municipality +15522,#IgnoreRow,True,#ignore,unmapped historical municipality +15523,#IgnoreRow,True,#ignore,unmapped historical municipality +15524,#IgnoreRow,True,#ignore,unmapped historical municipality +15525,#IgnoreRow,True,#ignore,unmapped historical municipality +15526,#IgnoreRow,True,#ignore,unmapped historical municipality +15541,#IgnoreRow,True,#ignore,unmapped historical municipality +15542,#IgnoreRow,True,#ignore,unmapped historical municipality +15543,#IgnoreRow,True,#ignore,unmapped historical municipality +15544,#IgnoreRow,True,#ignore,unmapped historical municipality +15545,#IgnoreRow,True,#ignore,unmapped historical municipality +15546,#IgnoreRow,True,#ignore,unmapped historical municipality +15547,#IgnoreRow,True,#ignore,unmapped historical municipality +15548,#IgnoreRow,True,#ignore,unmapped historical municipality +15549,#IgnoreRow,True,#ignore,unmapped historical municipality +15550,#IgnoreRow,True,#ignore,unmapped historical municipality +15561,#IgnoreRow,True,#ignore,unmapped historical municipality +15562,#IgnoreRow,True,#ignore,unmapped historical municipality +15563,#IgnoreRow,True,#ignore,unmapped historical municipality +15581,observationAbout,wikidataId/Q1204648 +15582,#IgnoreRow,True,#ignore,unmapped historical municipality +15583,#IgnoreRow,True,#ignore,unmapped historical municipality +15584,#IgnoreRow,True,#ignore,unmapped historical municipality +15585,#IgnoreRow,True,#ignore,unmapped historical municipality +15586,observationAbout,wikidataId/Q581708 +15601,#IgnoreRow,True,#ignore,unmapped historical municipality +15602,#IgnoreRow,True,#ignore,unmapped historical municipality +15603,#IgnoreRow,True,#ignore,unmapped historical municipality +15604,#IgnoreRow,True,#ignore,unmapped historical municipality +15605,#IgnoreRow,True,#ignore,unmapped historical municipality +15606,#IgnoreRow,True,#ignore,unmapped historical municipality +15607,#IgnoreRow,True,#ignore,unmapped historical municipality +15608,#IgnoreRow,True,#ignore,unmapped historical municipality +15609,#IgnoreRow,True,#ignore,unmapped historical municipality +16000,observationAbout,wikidataId/Q132929 +16201,observationAbout,wikidataId/Q204266 +16202,observationAbout,wikidataId/Q502988 +16203,#IgnoreRow,True,#ignore,unmapped historical municipality +16204,observationAbout,wikidataId/Q819613 +16205,observationAbout,wikidataId/Q646886 +16206,observationAbout,wikidataId/Q823513 +16207,observationAbout,wikidataId/Q753864 +16208,observationAbout,wikidataId/Q526760 +16209,observationAbout,wikidataId/Q734366 +16210,observationAbout,wikidataId/Q646805 +16211,observationAbout,wikidataId/Q851411 +16301,#IgnoreRow,True,#ignore,unmapped historical municipality +16302,#IgnoreRow,True,#ignore,unmapped historical municipality +16321,observationAbout,wikidataId/Q1204430 +16322,observationAbout,wikidataId/Q1204302 +16323,observationAbout,wikidataId/Q670189 +16341,#IgnoreRow,True,#ignore,unmapped historical municipality +16342,observationAbout,wikidataId/Q1203765 +16343,observationAbout,wikidataId/Q1204470 +16361,#IgnoreRow,True,#ignore,unmapped historical municipality +16362,#IgnoreRow,True,#ignore,unmapped historical municipality +16363,#IgnoreRow,True,#ignore,unmapped historical municipality +16364,#IgnoreRow,True,#ignore,unmapped historical municipality +16381,#IgnoreRow,True,#ignore,unmapped historical municipality +16382,#IgnoreRow,True,#ignore,unmapped historical municipality +16383,#IgnoreRow,True,#ignore,unmapped historical municipality +16384,#IgnoreRow,True,#ignore,unmapped historical municipality +16401,#IgnoreRow,True,#ignore,unmapped historical municipality +16402,#IgnoreRow,True,#ignore,unmapped historical municipality +16403,#IgnoreRow,True,#ignore,unmapped historical municipality +16404,#IgnoreRow,True,#ignore,unmapped historical municipality +16405,#IgnoreRow,True,#ignore,unmapped historical municipality +16406,#IgnoreRow,True,#ignore,unmapped historical municipality +16407,#IgnoreRow,True,#ignore,unmapped historical municipality +16408,#IgnoreRow,True,#ignore,unmapped historical municipality +16421,#IgnoreRow,True,#ignore,unmapped historical municipality +16422,#IgnoreRow,True,#ignore,unmapped historical municipality +17000,observationAbout,wikidataId/Q131281 +17201,observationAbout,wikidataId/Q191130 +17202,observationAbout,wikidataId/Q695892 +17203,observationAbout,wikidataId/Q646898 +17204,observationAbout,wikidataId/Q823896 +17205,observationAbout,wikidataId/Q846345 +17206,observationAbout,wikidataId/Q120103 +17207,observationAbout,wikidataId/Q822585 +17208,#IgnoreRow,True,#ignore,unmapped historical municipality +17209,observationAbout,wikidataId/Q846335 +17210,observationAbout,wikidataId/Q696407 +17211,observationAbout,wikidataId/Q658912 +17212,observationAbout,wikidataId/Q1354896 +17301,#IgnoreRow,True,#ignore,unmapped historical municipality +17321,#IgnoreRow,True,#ignore,unmapped historical municipality +17322,#IgnoreRow,True,#ignore,unmapped historical municipality +17323,#IgnoreRow,True,#ignore,unmapped historical municipality +17324,observationAbout,wikidataId/Q1347231 +17342,#IgnoreRow,True,#ignore,unmapped historical municipality +17343,#IgnoreRow,True,#ignore,unmapped historical municipality +17344,#IgnoreRow,True,#ignore,unmapped historical municipality +17345,#IgnoreRow,True,#ignore,unmapped historical municipality +17346,#IgnoreRow,True,#ignore,unmapped historical municipality +17347,#IgnoreRow,True,#ignore,unmapped historical municipality +17348,#IgnoreRow,True,#ignore,unmapped historical municipality +17349,#IgnoreRow,True,#ignore,unmapped historical municipality +17361,observationAbout,wikidataId/Q432014 +17362,#IgnoreRow,True,#ignore,unmapped historical municipality +17363,#IgnoreRow,True,#ignore,unmapped historical municipality +17364,#IgnoreRow,True,#ignore,unmapped historical municipality +17365,observationAbout,wikidataId/Q432027 +17382,#IgnoreRow,True,#ignore,unmapped historical municipality +17383,#IgnoreRow,True,#ignore,unmapped historical municipality +17384,observationAbout,wikidataId/Q1349406 +17385,#IgnoreRow,True,#ignore,unmapped historical municipality +17386,observationAbout,wikidataId/Q1204817 +17401,#IgnoreRow,True,#ignore,unmapped historical municipality +17402,#IgnoreRow,True,#ignore,unmapped historical municipality +17403,#IgnoreRow,True,#ignore,unmapped historical municipality +17404,#IgnoreRow,True,#ignore,unmapped historical municipality +17405,#IgnoreRow,True,#ignore,unmapped historical municipality +17406,#IgnoreRow,True,#ignore,unmapped historical municipality +17407,observationAbout,wikidataId/Q1349436 +17421,#IgnoreRow,True,#ignore,unmapped historical municipality +17422,#IgnoreRow,True,#ignore,unmapped historical municipality +17423,#IgnoreRow,True,#ignore,unmapped historical municipality +17424,#IgnoreRow,True,#ignore,unmapped historical municipality +17441,#IgnoreRow,True,#ignore,unmapped historical municipality +17461,observationAbout,wikidataId/Q1349353 +17462,#IgnoreRow,True,#ignore,unmapped historical municipality +17463,observationAbout,wikidataId/Q1349388 +18000,observationAbout,wikidataId/Q133879 +18201,observationAbout,wikidataId/Q201618 +18202,observationAbout,wikidataId/Q847581 +18203,#IgnoreRow,True,#ignore,unmapped historical municipality +18204,observationAbout,wikidataId/Q41773 +18205,observationAbout,wikidataId/Q385684 +18206,observationAbout,wikidataId/Q847543 +18207,observationAbout,wikidataId/Q847598 +18208,observationAbout,wikidataId/Q538623 +18209,observationAbout,wikidataId/Q877941 +18210,observationAbout,wikidataId/Q990342 +18302,#IgnoreRow,True,#ignore,unmapped historical municipality +18321,#IgnoreRow,True,#ignore,unmapped historical municipality +18322,observationAbout,wikidataId/Q1303631 +18323,#IgnoreRow,True,#ignore,unmapped historical municipality +18342,#IgnoreRow,True,#ignore,unmapped historical municipality +18361,#IgnoreRow,True,#ignore,unmapped historical municipality +18362,#IgnoreRow,True,#ignore,unmapped historical municipality +18363,#IgnoreRow,True,#ignore,unmapped historical municipality +18364,#IgnoreRow,True,#ignore,unmapped historical municipality +18365,#IgnoreRow,True,#ignore,unmapped historical municipality +18366,#IgnoreRow,True,#ignore,unmapped historical municipality +18381,#IgnoreRow,True,#ignore,unmapped historical municipality +18382,observationAbout,wikidataId/Q1347548 +18401,#IgnoreRow,True,#ignore,unmapped historical municipality +18402,#IgnoreRow,True,#ignore,unmapped historical municipality +18403,#IgnoreRow,True,#ignore,unmapped historical municipality +18404,observationAbout,wikidataId/Q1347526 +18421,#IgnoreRow,True,#ignore,unmapped historical municipality +18422,#IgnoreRow,True,#ignore,unmapped historical municipality +18423,observationAbout,wikidataId/Q1280737 +18424,#IgnoreRow,True,#ignore,unmapped historical municipality +18425,#IgnoreRow,True,#ignore,unmapped historical municipality +18426,#IgnoreRow,True,#ignore,unmapped historical municipality +18441,#IgnoreRow,True,#ignore,unmapped historical municipality +18442,observationAbout,wikidataId/Q629384 +18461,#IgnoreRow,True,#ignore,unmapped historical municipality +18462,#IgnoreRow,True,#ignore,unmapped historical municipality +18481,observationAbout,wikidataId/Q1349071 +18482,#IgnoreRow,True,#ignore,unmapped historical municipality +18483,observationAbout,wikidataId/Q384981 +18501,observationAbout,wikidataId/Q1349039 +19000,observationAbout,wikidataId/Q132720 +19201,observationAbout,wikidataId/Q201932 +19202,observationAbout,wikidataId/Q647508 +19203,#IgnoreRow,True,#ignore,unmapped historical municipality +19204,observationAbout,wikidataId/Q747362 +19205,observationAbout,wikidataId/Q840610 +19206,observationAbout,wikidataId/Q386508 +19207,observationAbout,wikidataId/Q290326 +19208,observationAbout,wikidataId/Q824000 +19209,observationAbout,wikidataId/Q825272 +19210,observationAbout,wikidataId/Q577309 +19211,observationAbout,wikidataId/Q849434 +19212,observationAbout,wikidataId/Q851030 +19213,observationAbout,wikidataId/Q248723 +19214,observationAbout,wikidataId/Q666049 +19301,#IgnoreRow,True,#ignore,unmapped historical municipality +19302,#IgnoreRow,True,#ignore,unmapped historical municipality +19303,#IgnoreRow,True,#ignore,unmapped historical municipality +19304,#IgnoreRow,True,#ignore,unmapped historical municipality +19305,#IgnoreRow,True,#ignore,unmapped historical municipality +19321,#IgnoreRow,True,#ignore,unmapped historical municipality +19322,#IgnoreRow,True,#ignore,unmapped historical municipality +19323,#IgnoreRow,True,#ignore,unmapped historical municipality +19324,#IgnoreRow,True,#ignore,unmapped historical municipality +19325,#IgnoreRow,True,#ignore,unmapped historical municipality +19326,#IgnoreRow,True,#ignore,unmapped historical municipality +19327,#IgnoreRow,True,#ignore,unmapped historical municipality +19328,#IgnoreRow,True,#ignore,unmapped historical municipality +19341,#IgnoreRow,True,#ignore,unmapped historical municipality +19342,#IgnoreRow,True,#ignore,unmapped historical municipality +19343,#IgnoreRow,True,#ignore,unmapped historical municipality +19344,#IgnoreRow,True,#ignore,unmapped historical municipality +19345,#IgnoreRow,True,#ignore,unmapped historical municipality +19346,observationAbout,wikidataId/Q1204453 +19361,#IgnoreRow,True,#ignore,unmapped historical municipality +19362,#IgnoreRow,True,#ignore,unmapped historical municipality +19363,#IgnoreRow,True,#ignore,unmapped historical municipality +19364,observationAbout,wikidataId/Q1204436 +19365,observationAbout,wikidataId/Q1204435 +19366,observationAbout,wikidataId/Q1204802 +19367,#IgnoreRow,True,#ignore,unmapped historical municipality +19368,observationAbout,wikidataId/Q1351723 +19381,#IgnoreRow,True,#ignore,unmapped historical municipality +19382,#IgnoreRow,True,#ignore,unmapped historical municipality +19383,#IgnoreRow,True,#ignore,unmapped historical municipality +19384,observationAbout,wikidataId/Q1204276 +19385,#IgnoreRow,True,#ignore,unmapped historical municipality +19386,#IgnoreRow,True,#ignore,unmapped historical municipality +19387,#IgnoreRow,True,#ignore,unmapped historical municipality +19388,#IgnoreRow,True,#ignore,unmapped historical municipality +19389,#IgnoreRow,True,#ignore,unmapped historical municipality +19390,#IgnoreRow,True,#ignore,unmapped historical municipality +19391,#IgnoreRow,True,#ignore,unmapped historical municipality +19401,#IgnoreRow,True,#ignore,unmapped historical municipality +19402,#IgnoreRow,True,#ignore,unmapped historical municipality +19403,#IgnoreRow,True,#ignore,unmapped historical municipality +19404,#IgnoreRow,True,#ignore,unmapped historical municipality +19405,#IgnoreRow,True,#ignore,unmapped historical municipality +19406,#IgnoreRow,True,#ignore,unmapped historical municipality +19407,#IgnoreRow,True,#ignore,unmapped historical municipality +19408,#IgnoreRow,True,#ignore,unmapped historical municipality +19409,#IgnoreRow,True,#ignore,unmapped historical municipality +19421,#IgnoreRow,True,#ignore,unmapped historical municipality +19422,observationAbout,wikidataId/Q1204426 +19423,observationAbout,wikidataId/Q1203881 +19424,observationAbout,wikidataId/Q1348944 +19425,observationAbout,wikidataId/Q1204394 +19426,#IgnoreRow,True,#ignore,unmapped historical municipality +19427,#IgnoreRow,True,#ignore,unmapped historical municipality +19428,#IgnoreRow,True,#ignore,unmapped historical municipality +19429,observationAbout,wikidataId/Q1204384 +19430,observationAbout,wikidataId/Q1004231 +19441,#IgnoreRow,True,#ignore,unmapped historical municipality +19442,observationAbout,wikidataId/Q29463 +19443,observationAbout,wikidataId/Q1204259 +20000,observationAbout,wikidataId/Q127877 +20201,observationAbout,wikidataId/Q128849 +20202,observationAbout,wikidataId/Q213324 +20203,observationAbout,wikidataId/Q844852 +20204,observationAbout,wikidataId/Q838672 +20205,observationAbout,wikidataId/Q841129 +20206,observationAbout,wikidataId/Q846338 +20207,observationAbout,wikidataId/Q850461 +20208,observationAbout,wikidataId/Q838657 +20209,observationAbout,wikidataId/Q840888 +20210,observationAbout,wikidataId/Q838684 +20211,observationAbout,wikidataId/Q838652 +20212,observationAbout,wikidataId/Q385375 +20213,observationAbout,wikidataId/Q851097 +20214,observationAbout,wikidataId/Q838660 +20215,observationAbout,wikidataId/Q857272 +20216,#IgnoreRow,True,#ignore,unmapped historical municipality +20217,observationAbout,wikidataId/Q495821 +20218,observationAbout,wikidataId/Q179757 +20219,observationAbout,wikidataId/Q840859 +20220,observationAbout,wikidataId/Q534667 +20301,#IgnoreRow,True,#ignore,unmapped historical municipality +20302,#IgnoreRow,True,#ignore,unmapped historical municipality +20303,observationAbout,wikidataId/Q1204222 +20304,observationAbout,wikidataId/Q588091 +20305,observationAbout,wikidataId/Q1203240 +20306,observationAbout,wikidataId/Q763885 +20307,observationAbout,wikidataId/Q1202620 +20308,#IgnoreRow,True,#ignore,unmapped historical municipality +20309,observationAbout,wikidataId/Q1203700 +20321,observationAbout,wikidataId/Q1012064 +20322,#IgnoreRow,True,#ignore,unmapped historical municipality +20323,observationAbout,wikidataId/Q1204123 +20324,observationAbout,wikidataId/Q1346895 +20325,#IgnoreRow,True,#ignore,unmapped historical municipality +20326,#IgnoreRow,True,#ignore,unmapped historical municipality +20341,#IgnoreRow,True,#ignore,unmapped historical municipality +20342,#IgnoreRow,True,#ignore,unmapped historical municipality +20343,#IgnoreRow,True,#ignore,unmapped historical municipality +20345,#IgnoreRow,True,#ignore,unmapped historical municipality +20346,#IgnoreRow,True,#ignore,unmapped historical municipality +20347,#IgnoreRow,True,#ignore,unmapped historical municipality +20349,observationAbout,wikidataId/Q615719 +20350,observationAbout,wikidataId/Q1346846 +20361,observationAbout,wikidataId/Q1204211 +20362,observationAbout,wikidataId/Q1204145 +20363,observationAbout,wikidataId/Q1203041 +20381,#IgnoreRow,True,#ignore,unmapped historical municipality +20382,observationAbout,wikidataId/Q1347504 +20383,observationAbout,wikidataId/Q1346975 +20384,observationAbout,wikidataId/Q522462 +20385,observationAbout,wikidataId/Q1347404 +20386,observationAbout,wikidataId/Q1194229 +20387,#IgnoreRow,True,#ignore,unmapped historical municipality +20388,observationAbout,wikidataId/Q959501 +20401,#IgnoreRow,True,#ignore,unmapped historical municipality +20402,observationAbout,wikidataId/Q222825 +20403,observationAbout,wikidataId/Q1203136 +20404,observationAbout,wikidataId/Q1203314 +20405,#IgnoreRow,True,#ignore,unmapped historical municipality +20406,#IgnoreRow,True,#ignore,unmapped historical municipality +20407,observationAbout,wikidataId/Q1203279 +20408,#IgnoreRow,True,#ignore,unmapped historical municipality +20409,observationAbout,wikidataId/Q1346768 +20410,observationAbout,wikidataId/Q1204111 +20411,observationAbout,wikidataId/Q1203050 +20412,observationAbout,wikidataId/Q1203126 +20413,observationAbout,wikidataId/Q1203975 +20414,observationAbout,wikidataId/Q902559 +20415,observationAbout,wikidataId/Q1203939 +20416,observationAbout,wikidataId/Q44545 +20417,observationAbout,wikidataId/Q1203751 +20418,#IgnoreRow,True,#ignore,unmapped historical municipality +20419,#IgnoreRow,True,#ignore,unmapped historical municipality +20421,#IgnoreRow,True,#ignore,unmapped historical municipality +20422,observationAbout,wikidataId/Q374859 +20423,observationAbout,wikidataId/Q1310803 +20424,#IgnoreRow,True,#ignore,unmapped historical municipality +20425,observationAbout,wikidataId/Q1743742 +20426,#IgnoreRow,True,#ignore,unmapped historical municipality +20427,#IgnoreRow,True,#ignore,unmapped historical municipality +20428,#IgnoreRow,True,#ignore,unmapped historical municipality +20429,observationAbout,wikidataId/Q1204275 +20430,observationAbout,wikidataId/Q1204461 +20431,#IgnoreRow,True,#ignore,unmapped historical municipality +20432,observationAbout,wikidataId/Q1347536 +20441,#IgnoreRow,True,#ignore,unmapped historical municipality +20443,#IgnoreRow,True,#ignore,unmapped historical municipality +20444,#IgnoreRow,True,#ignore,unmapped historical municipality +20445,#IgnoreRow,True,#ignore,unmapped historical municipality +20446,observationAbout,wikidataId/Q1349060 +20447,#IgnoreRow,True,#ignore,unmapped historical municipality +20448,observationAbout,wikidataId/Q1347517 +20449,#IgnoreRow,True,#ignore,unmapped historical municipality +20450,observationAbout,wikidataId/Q1348998 +20451,observationAbout,wikidataId/Q720461 +20452,observationAbout,wikidataId/Q1072656 +20461,#IgnoreRow,True,#ignore,unmapped historical municipality +20462,#IgnoreRow,True,#ignore,unmapped historical municipality +20463,#IgnoreRow,True,#ignore,unmapped historical municipality +20464,#IgnoreRow,True,#ignore,unmapped historical municipality +20465,#IgnoreRow,True,#ignore,unmapped historical municipality +20466,#IgnoreRow,True,#ignore,unmapped historical municipality +20467,#IgnoreRow,True,#ignore,unmapped historical municipality +20481,observationAbout,wikidataId/Q1204048 +20482,observationAbout,wikidataId/Q1204524 +20483,#IgnoreRow,True,#ignore,unmapped historical municipality +20484,#IgnoreRow,True,#ignore,unmapped historical municipality +20485,observationAbout,wikidataId/Q1011157 +20486,observationAbout,wikidataId/Q1204497 +20501,#IgnoreRow,True,#ignore,unmapped historical municipality +20502,#IgnoreRow,True,#ignore,unmapped historical municipality +20521,observationAbout,wikidataId/Q1348962 +20522,#IgnoreRow,True,#ignore,unmapped historical municipality +20541,observationAbout,wikidataId/Q1348977 +20543,observationAbout,wikidataId/Q1349115 +20561,observationAbout,wikidataId/Q1204468 +20562,observationAbout,wikidataId/Q1204471 +20563,observationAbout,wikidataId/Q1354760 +20581,#IgnoreRow,True,#ignore,unmapped historical municipality +20582,#IgnoreRow,True,#ignore,unmapped historical municipality +20583,observationAbout,wikidataId/Q1349050 +20584,#IgnoreRow,True,#ignore,unmapped historical municipality +20585,#IgnoreRow,True,#ignore,unmapped historical municipality +20586,#IgnoreRow,True,#ignore,unmapped historical municipality +20587,#IgnoreRow,True,#ignore,unmapped historical municipality +20588,observationAbout,wikidataId/Q1348971 +20589,#IgnoreRow,True,#ignore,unmapped historical municipality +20590,observationAbout,wikidataId/Q772034 +20601,#IgnoreRow,True,#ignore,unmapped historical municipality +20602,observationAbout,wikidataId/Q1204286 +21000,observationAbout,wikidataId/Q131277 +21201,observationAbout,wikidataId/Q45798 +21202,observationAbout,wikidataId/Q384926 +21203,observationAbout,wikidataId/Q622594 +21204,observationAbout,wikidataId/Q819689 +21205,observationAbout,wikidataId/Q860647 +21206,observationAbout,wikidataId/Q844052 +21207,observationAbout,wikidataId/Q853835 +21208,observationAbout,wikidataId/Q853821 +21209,observationAbout,wikidataId/Q819595 +21210,observationAbout,wikidataId/Q819653 +21211,observationAbout,wikidataId/Q857305 +21212,observationAbout,wikidataId/Q847332 +21213,observationAbout,wikidataId/Q641491 +21214,observationAbout,wikidataId/Q853805 +21215,observationAbout,wikidataId/Q773311 +21216,observationAbout,wikidataId/Q782103 +21217,observationAbout,wikidataId/Q847319 +21218,observationAbout,wikidataId/Q844047 +21219,observationAbout,wikidataId/Q857243 +21220,observationAbout,wikidataId/Q819639 +21221,observationAbout,wikidataId/Q844056 +21301,#IgnoreRow,True,#ignore,unmapped historical municipality +21302,observationAbout,wikidataId/Q1204285 +21303,observationAbout,wikidataId/Q1203551 +21304,#IgnoreRow,True,#ignore,unmapped historical municipality +21321,#IgnoreRow,True,#ignore,unmapped historical municipality +21322,#IgnoreRow,True,#ignore,unmapped historical municipality +21323,#IgnoreRow,True,#ignore,unmapped historical municipality +21341,observationAbout,wikidataId/Q153814 +21342,#IgnoreRow,True,#ignore,unmapped historical municipality +21361,observationAbout,wikidataId/Q1345531 +21362,observationAbout,wikidataId/Q427600 +21381,observationAbout,wikidataId/Q1203575 +21382,observationAbout,wikidataId/Q1204291 +21383,observationAbout,wikidataId/Q474384 +21384,#IgnoreRow,True,#ignore,unmapped historical municipality +21401,observationAbout,wikidataId/Q1204046 +21402,#IgnoreRow,True,#ignore,unmapped historical municipality +21403,observationAbout,wikidataId/Q1203308 +21404,observationAbout,wikidataId/Q1203599 +21405,#IgnoreRow,True,#ignore,unmapped historical municipality +21406,#IgnoreRow,True,#ignore,unmapped historical municipality +21407,#IgnoreRow,True,#ignore,unmapped historical municipality +21408,#IgnoreRow,True,#ignore,unmapped historical municipality +21409,#IgnoreRow,True,#ignore,unmapped historical municipality +21421,observationAbout,wikidataId/Q1203696 +21422,#IgnoreRow,True,#ignore,unmapped historical municipality +21423,#IgnoreRow,True,#ignore,unmapped historical municipality +21424,#IgnoreRow,True,#ignore,unmapped historical municipality +21425,#IgnoreRow,True,#ignore,unmapped historical municipality +21426,#IgnoreRow,True,#ignore,unmapped historical municipality +21427,#IgnoreRow,True,#ignore,unmapped historical municipality +21441,#IgnoreRow,True,#ignore,unmapped historical municipality +21442,#IgnoreRow,True,#ignore,unmapped historical municipality +21443,#IgnoreRow,True,#ignore,unmapped historical municipality +21461,#IgnoreRow,True,#ignore,unmapped historical municipality +21462,#IgnoreRow,True,#ignore,unmapped historical municipality +21463,#IgnoreRow,True,#ignore,unmapped historical municipality +21464,#IgnoreRow,True,#ignore,unmapped historical municipality +21465,#IgnoreRow,True,#ignore,unmapped historical municipality +21481,#IgnoreRow,True,#ignore,unmapped historical municipality +21482,#IgnoreRow,True,#ignore,unmapped historical municipality +21483,#IgnoreRow,True,#ignore,unmapped historical municipality +21484,#IgnoreRow,True,#ignore,unmapped historical municipality +21485,#IgnoreRow,True,#ignore,unmapped historical municipality +21486,#IgnoreRow,True,#ignore,unmapped historical municipality +21487,#IgnoreRow,True,#ignore,unmapped historical municipality +21501,observationAbout,wikidataId/Q598446 +21502,observationAbout,wikidataId/Q1203855 +21503,observationAbout,wikidataId/Q1203748 +21504,observationAbout,wikidataId/Q1203839 +21505,observationAbout,wikidataId/Q1346952 +21506,observationAbout,wikidataId/Q1203350 +21507,observationAbout,wikidataId/Q1202765 +21521,observationAbout,wikidataId/Q640925 +21522,#IgnoreRow,True,#ignore,unmapped historical municipality +21523,#IgnoreRow,True,#ignore,unmapped historical municipality +21541,#IgnoreRow,True,#ignore,unmapped historical municipality +21561,#IgnoreRow,True,#ignore,unmapped historical municipality +21562,#IgnoreRow,True,#ignore,unmapped historical municipality +21563,#IgnoreRow,True,#ignore,unmapped historical municipality +21564,#IgnoreRow,True,#ignore,unmapped historical municipality +21565,#IgnoreRow,True,#ignore,unmapped historical municipality +21566,#IgnoreRow,True,#ignore,unmapped historical municipality +21567,#IgnoreRow,True,#ignore,unmapped historical municipality +21568,#IgnoreRow,True,#ignore,unmapped historical municipality +21569,#IgnoreRow,True,#ignore,unmapped historical municipality +21570,#IgnoreRow,True,#ignore,unmapped historical municipality +21571,#IgnoreRow,True,#ignore,unmapped historical municipality +21581,#IgnoreRow,True,#ignore,unmapped historical municipality +21582,#IgnoreRow,True,#ignore,unmapped historical municipality +21583,#IgnoreRow,True,#ignore,unmapped historical municipality +21584,#IgnoreRow,True,#ignore,unmapped historical municipality +21585,#IgnoreRow,True,#ignore,unmapped historical municipality +21601,#IgnoreRow,True,#ignore,unmapped historical municipality +21602,#IgnoreRow,True,#ignore,unmapped historical municipality +21603,#IgnoreRow,True,#ignore,unmapped historical municipality +21604,observationAbout,wikidataId/Q1203744 +21605,#IgnoreRow,True,#ignore,unmapped historical municipality +21606,#IgnoreRow,True,#ignore,unmapped historical municipality +21607,#IgnoreRow,True,#ignore,unmapped historical municipality +21608,#IgnoreRow,True,#ignore,unmapped historical municipality +21621,#IgnoreRow,True,#ignore,unmapped historical municipality +21622,#IgnoreRow,True,#ignore,unmapped historical municipality +21623,#IgnoreRow,True,#ignore,unmapped historical municipality +21624,#IgnoreRow,True,#ignore,unmapped historical municipality +21625,#IgnoreRow,True,#ignore,unmapped historical municipality +21626,#IgnoreRow,True,#ignore,unmapped historical municipality +22000,observationAbout,wikidataId/Q131320 +22100,observationAbout,wikidataId/Q174691 +22101,observationAbout,wikidataId/Q1131483 +22102,observationAbout,wikidataId/Q1042630 +22103,observationAbout,wikidataId/Q79385 +22130,observationAbout,wikidataId/Q185125 +22131,observationAbout,wikidataId/Q1075863 +22132,observationAbout,wikidataId/Q1131084 +22133,observationAbout,wikidataId/Q1131075 +22134,observationAbout,wikidataId/Q1131512 +22135,observationAbout,wikidataId/Q1131477 +22136,observationAbout,wikidataId/Q1203662 +22137,observationAbout,wikidataId/Q1131460 +22138,#IgnoreRow,True,#ignore,unmapped historical municipality +22139,#IgnoreRow,True,#ignore,unmapped historical municipality +22140,#IgnoreRow,True,#ignore,unmapped historical municipality +22201,#IgnoreRow,True,#ignore,unmapped historical municipality +22202,#IgnoreRow,True,#ignore,unmapped historical municipality +22203,observationAbout,wikidataId/Q241037 +22204,#IgnoreRow,True,#ignore,unmapped historical municipality +22205,observationAbout,wikidataId/Q653364 +22206,observationAbout,wikidataId/Q653478 +22207,observationAbout,wikidataId/Q496072 +22208,observationAbout,wikidataId/Q721163 +22209,observationAbout,wikidataId/Q605256 +22210,observationAbout,wikidataId/Q328613 +22211,observationAbout,wikidataId/Q386651 +22212,observationAbout,wikidataId/Q696783 +22213,observationAbout,wikidataId/Q823988 +22214,observationAbout,wikidataId/Q734432 +22215,observationAbout,wikidataId/Q735175 +22216,observationAbout,wikidataId/Q821337 +22217,#IgnoreRow,True,#ignore,unmapped historical municipality +22218,#IgnoreRow,True,#ignore,unmapped historical municipality +22219,observationAbout,wikidataId/Q653402 +22220,observationAbout,wikidataId/Q847577 +22221,observationAbout,wikidataId/Q850515 +22222,observationAbout,wikidataId/Q696840 +22223,observationAbout,wikidataId/Q850380 +22224,observationAbout,wikidataId/Q726565 +22225,observationAbout,wikidataId/Q863758 +22226,observationAbout,wikidataId/Q850527 +22301,observationAbout,wikidataId/Q304491 +22302,observationAbout,wikidataId/Q1203861 +22304,observationAbout,wikidataId/Q1203741 +22305,observationAbout,wikidataId/Q1204017 +22306,observationAbout,wikidataId/Q1203949 +22307,#IgnoreRow,True,#ignore,unmapped historical municipality +22321,#IgnoreRow,True,#ignore,unmapped historical municipality +22322,#IgnoreRow,True,#ignore,unmapped historical municipality +22323,#IgnoreRow,True,#ignore,unmapped historical municipality +22324,#IgnoreRow,True,#ignore,unmapped historical municipality +22325,observationAbout,wikidataId/Q1203533 +22326,#IgnoreRow,True,#ignore,unmapped historical municipality +22327,#IgnoreRow,True,#ignore,unmapped historical municipality +22328,#IgnoreRow,True,#ignore,unmapped historical municipality +22329,#IgnoreRow,True,#ignore,unmapped historical municipality +22341,observationAbout,wikidataId/Q1203168 +22342,observationAbout,wikidataId/Q1204459 +22344,observationAbout,wikidataId/Q726418 +22361,#IgnoreRow,True,#ignore,unmapped historical municipality +22381,#IgnoreRow,True,#ignore,unmapped historical municipality +22382,#IgnoreRow,True,#ignore,unmapped historical municipality +22383,#IgnoreRow,True,#ignore,unmapped historical municipality +22401,#IgnoreRow,True,#ignore,unmapped historical municipality +22402,#IgnoreRow,True,#ignore,unmapped historical municipality +22421,#IgnoreRow,True,#ignore,unmapped historical municipality +22422,#IgnoreRow,True,#ignore,unmapped historical municipality +22423,#IgnoreRow,True,#ignore,unmapped historical municipality +22424,observationAbout,wikidataId/Q1202789 +22425,#IgnoreRow,True,#ignore,unmapped historical municipality +22426,#IgnoreRow,True,#ignore,unmapped historical municipality +22427,#IgnoreRow,True,#ignore,unmapped historical municipality +22428,#IgnoreRow,True,#ignore,unmapped historical municipality +22429,observationAbout,wikidataId/Q1204025 +22442,#IgnoreRow,True,#ignore,unmapped historical municipality +22444,#IgnoreRow,True,#ignore,unmapped historical municipality +22445,#IgnoreRow,True,#ignore,unmapped historical municipality +22446,#IgnoreRow,True,#ignore,unmapped historical municipality +22447,#IgnoreRow,True,#ignore,unmapped historical municipality +22461,observationAbout,wikidataId/Q1345571 +22462,#IgnoreRow,True,#ignore,unmapped historical municipality +22481,#IgnoreRow,True,#ignore,unmapped historical municipality +22482,#IgnoreRow,True,#ignore,unmapped historical municipality +22483,#IgnoreRow,True,#ignore,unmapped historical municipality +22484,#IgnoreRow,True,#ignore,unmapped historical municipality +22485,#IgnoreRow,True,#ignore,unmapped historical municipality +22486,#IgnoreRow,True,#ignore,unmapped historical municipality +22487,#IgnoreRow,True,#ignore,unmapped historical municipality +22488,#IgnoreRow,True,#ignore,unmapped historical municipality +22501,#IgnoreRow,True,#ignore,unmapped historical municipality +22502,#IgnoreRow,True,#ignore,unmapped historical municipality +22503,#IgnoreRow,True,#ignore,unmapped historical municipality +22505,#IgnoreRow,True,#ignore,unmapped historical municipality +22521,#IgnoreRow,True,#ignore,unmapped historical municipality +22522,#IgnoreRow,True,#ignore,unmapped historical municipality +22523,#IgnoreRow,True,#ignore,unmapped historical municipality +23000,observationAbout,wikidataId/Q80434 +23100,observationAbout,wikidataId/Q11751 +23101,observationAbout,wikidataId/Q1207951 +23102,observationAbout,wikidataId/Q578824 +23103,observationAbout,wikidataId/Q1207964 +23104,observationAbout,wikidataId/Q1207973 +23105,observationAbout,wikidataId/Q780521 +23106,observationAbout,wikidataId/Q1207921 +23107,observationAbout,wikidataId/Q1155273 +23108,observationAbout,wikidataId/Q1207957 +23109,observationAbout,wikidataId/Q1355625 +23110,observationAbout,wikidataId/Q1155226 +23111,observationAbout,wikidataId/Q1207982 +23112,observationAbout,wikidataId/Q1155286 +23113,observationAbout,wikidataId/Q1155291 +23114,observationAbout,wikidataId/Q1155209 +23115,observationAbout,wikidataId/Q1155280 +23116,observationAbout,wikidataId/Q1155245 +23201,observationAbout,wikidataId/Q336431 +23202,observationAbout,wikidataId/Q242783 +23203,observationAbout,wikidataId/Q270810 +23204,observationAbout,wikidataId/Q499493 +23205,observationAbout,wikidataId/Q463171 +23206,observationAbout,wikidataId/Q26101 +23207,observationAbout,wikidataId/Q389495 +23208,observationAbout,wikidataId/Q816736 +23209,observationAbout,wikidataId/Q495802 +23210,observationAbout,wikidataId/Q389515 +23211,observationAbout,wikidataId/Q201117 +23212,observationAbout,wikidataId/Q270806 +23213,observationAbout,wikidataId/Q317844 +23214,observationAbout,wikidataId/Q694930 +23215,observationAbout,wikidataId/Q503411 +23216,observationAbout,wikidataId/Q209683 +23217,observationAbout,wikidataId/Q820496 +23218,#IgnoreRow,True,#ignore,unmapped historical municipality +23219,observationAbout,wikidataId/Q389537 +23220,observationAbout,wikidataId/Q389562 +23221,observationAbout,wikidataId/Q694923 +23222,observationAbout,wikidataId/Q622620 +23223,observationAbout,wikidataId/Q384809 +23224,observationAbout,wikidataId/Q495826 +23225,observationAbout,wikidataId/Q223496 +23226,observationAbout,wikidataId/Q694893 +23227,observationAbout,wikidataId/Q816734 +23228,observationAbout,wikidataId/Q648275 +23229,observationAbout,wikidataId/Q817280 +23230,observationAbout,wikidataId/Q696777 +23231,observationAbout,wikidataId/Q501891 +23232,observationAbout,wikidataId/Q269644 +23233,observationAbout,wikidataId/Q502407 +23234,observationAbout,wikidataId/Q597594 +23235,observationAbout,wikidataId/Q332825 +23236,observationAbout,wikidataId/Q1023781 +23237,observationAbout,wikidataId/Q451661 +23238,observationAbout,wikidataId/Q930396 +23302,observationAbout,wikidataId/Q265050 +23303,#IgnoreRow,True,#ignore,unmapped historical municipality +23304,#IgnoreRow,True,#ignore,unmapped historical municipality +23341,#IgnoreRow,True,#ignore,unmapped historical municipality +23342,observationAbout,wikidataId/Q1345562 +23343,#IgnoreRow,True,#ignore,unmapped historical municipality +23344,#IgnoreRow,True,#ignore,unmapped historical municipality +23345,#IgnoreRow,True,#ignore,unmapped historical municipality +23346,#IgnoreRow,True,#ignore,unmapped historical municipality +23347,#IgnoreRow,True,#ignore,unmapped historical municipality +23361,observationAbout,wikidataId/Q1203960 +23362,observationAbout,wikidataId/Q1203107 +23381,#IgnoreRow,True,#ignore,unmapped historical municipality +23401,#IgnoreRow,True,#ignore,unmapped historical municipality +23402,#IgnoreRow,True,#ignore,unmapped historical municipality +23421,#IgnoreRow,True,#ignore,unmapped historical municipality +23422,#IgnoreRow,True,#ignore,unmapped historical municipality +23423,#IgnoreRow,True,#ignore,unmapped historical municipality +23424,observationAbout,wikidataId/Q1203226 +23425,observationAbout,wikidataId/Q1203992 +23426,#IgnoreRow,True,#ignore,unmapped historical municipality +23427,observationAbout,wikidataId/Q1203952 +23428,#IgnoreRow,True,#ignore,unmapped historical municipality +23429,#IgnoreRow,True,#ignore,unmapped historical municipality +23430,#IgnoreRow,True,#ignore,unmapped historical municipality +23431,#IgnoreRow,True,#ignore,unmapped historical municipality +23432,#IgnoreRow,True,#ignore,unmapped historical municipality +23441,observationAbout,wikidataId/Q1203868 +23442,observationAbout,wikidataId/Q677745 +23445,observationAbout,wikidataId/Q1203589 +23446,observationAbout,wikidataId/Q1203828 +23447,observationAbout,wikidataId/Q1203232 +23481,#IgnoreRow,True,#ignore,unmapped historical municipality +23482,#IgnoreRow,True,#ignore,unmapped historical municipality +23483,#IgnoreRow,True,#ignore,unmapped historical municipality +23501,observationAbout,wikidataId/Q1345954 +23502,#IgnoreRow,True,#ignore,unmapped historical municipality +23521,#IgnoreRow,True,#ignore,unmapped historical municipality +23522,#IgnoreRow,True,#ignore,unmapped historical municipality +23523,#IgnoreRow,True,#ignore,unmapped historical municipality +23541,#IgnoreRow,True,#ignore,unmapped historical municipality +23543,#IgnoreRow,True,#ignore,unmapped historical municipality +23544,#IgnoreRow,True,#ignore,unmapped historical municipality +23561,observationAbout,wikidataId/Q1346777 +23562,observationAbout,wikidataId/Q1354877 +23563,observationAbout,wikidataId/Q153618 +23564,#IgnoreRow,True,#ignore,unmapped historical municipality +23565,#IgnoreRow,True,#ignore,unmapped historical municipality +23566,#IgnoreRow,True,#ignore,unmapped historical municipality +23581,#IgnoreRow,True,#ignore,unmapped historical municipality +23582,#IgnoreRow,True,#ignore,unmapped historical municipality +23601,#IgnoreRow,True,#ignore,unmapped historical municipality +23602,#IgnoreRow,True,#ignore,unmapped historical municipality +23603,#IgnoreRow,True,#ignore,unmapped historical municipality +23604,#IgnoreRow,True,#ignore,unmapped historical municipality +23621,#IgnoreRow,True,#ignore,unmapped historical municipality +23622,#IgnoreRow,True,#ignore,unmapped historical municipality +23623,#IgnoreRow,True,#ignore,unmapped historical municipality +24000,observationAbout,wikidataId/Q128196 +24201,observationAbout,wikidataId/Q203027 +24202,observationAbout,wikidataId/Q26410 +24203,observationAbout,wikidataId/Q328067 +24204,observationAbout,wikidataId/Q460829 +24205,observationAbout,wikidataId/Q496472 +24206,#IgnoreRow,True,#ignore,unmapped historical municipality +24207,observationAbout,wikidataId/Q207311 +24208,observationAbout,wikidataId/Q166183 +24209,observationAbout,wikidataId/Q847554 +24210,observationAbout,wikidataId/Q646858 +24211,observationAbout,wikidataId/Q819629 +24212,observationAbout,wikidataId/Q819672 +24213,#IgnoreRow,True,#ignore,unmapped historical municipality +24214,observationAbout,wikidataId/Q281817 +24215,observationAbout,wikidataId/Q338009 +24216,observationAbout,wikidataId/Q460820 +24301,#IgnoreRow,True,#ignore,unmapped historical municipality +24302,#IgnoreRow,True,#ignore,unmapped historical municipality +24303,observationAbout,wikidataId/Q1203943 +24321,#IgnoreRow,True,#ignore,unmapped historical municipality +24322,#IgnoreRow,True,#ignore,unmapped historical municipality +24323,#IgnoreRow,True,#ignore,unmapped historical municipality +24324,observationAbout,wikidataId/Q1204086 +24325,#IgnoreRow,True,#ignore,unmapped historical municipality +24341,observationAbout,wikidataId/Q1204099 +24342,#IgnoreRow,True,#ignore,unmapped historical municipality +24343,observationAbout,wikidataId/Q598905 +24344,observationAbout,wikidataId/Q1345551 +24361,#IgnoreRow,True,#ignore,unmapped historical municipality +24381,#IgnoreRow,True,#ignore,unmapped historical municipality +24382,#IgnoreRow,True,#ignore,unmapped historical municipality +24383,#IgnoreRow,True,#ignore,unmapped historical municipality +24384,#IgnoreRow,True,#ignore,unmapped historical municipality +24402,#IgnoreRow,True,#ignore,unmapped historical municipality +24403,#IgnoreRow,True,#ignore,unmapped historical municipality +24404,#IgnoreRow,True,#ignore,unmapped historical municipality +24405,#IgnoreRow,True,#ignore,unmapped historical municipality +24406,#IgnoreRow,True,#ignore,unmapped historical municipality +24407,#IgnoreRow,True,#ignore,unmapped historical municipality +24421,#IgnoreRow,True,#ignore,unmapped historical municipality +24422,#IgnoreRow,True,#ignore,unmapped historical municipality +24441,observationAbout,wikidataId/Q1203963 +24442,observationAbout,wikidataId/Q1203258 +24443,observationAbout,wikidataId/Q1204505 +24444,#IgnoreRow,True,#ignore,unmapped historical municipality +24445,#IgnoreRow,True,#ignore,unmapped historical municipality +24461,observationAbout,wikidataId/Q587348 +24462,#IgnoreRow,True,#ignore,unmapped historical municipality +24463,#IgnoreRow,True,#ignore,unmapped historical municipality +24464,#IgnoreRow,True,#ignore,unmapped historical municipality +24465,#IgnoreRow,True,#ignore,unmapped historical municipality +24466,#IgnoreRow,True,#ignore,unmapped historical municipality +24467,#IgnoreRow,True,#ignore,unmapped historical municipality +24468,#IgnoreRow,True,#ignore,unmapped historical municipality +24469,#IgnoreRow,True,#ignore,unmapped historical municipality +24470,observationAbout,wikidataId/Q1203708 +24471,observationAbout,wikidataId/Q1203189 +24472,observationAbout,wikidataId/Q1111031 +24481,#IgnoreRow,True,#ignore,unmapped historical municipality +24482,#IgnoreRow,True,#ignore,unmapped historical municipality +24483,#IgnoreRow,True,#ignore,unmapped historical municipality +24484,#IgnoreRow,True,#ignore,unmapped historical municipality +24501,#IgnoreRow,True,#ignore,unmapped historical municipality +24521,#IgnoreRow,True,#ignore,unmapped historical municipality +24522,#IgnoreRow,True,#ignore,unmapped historical municipality +24523,#IgnoreRow,True,#ignore,unmapped historical municipality +24524,#IgnoreRow,True,#ignore,unmapped historical municipality +24525,#IgnoreRow,True,#ignore,unmapped historical municipality +24541,#IgnoreRow,True,#ignore,unmapped historical municipality +24542,#IgnoreRow,True,#ignore,unmapped historical municipality +24543,observationAbout,wikidataId/Q1203791 +24561,observationAbout,wikidataId/Q1203799 +24562,observationAbout,wikidataId/Q1203117 +24563,#IgnoreRow,True,#ignore,unmapped historical municipality +24564,#IgnoreRow,True,#ignore,unmapped historical municipality +25000,observationAbout,wikidataId/Q131358 +25201,observationAbout,wikidataId/Q202907 +25202,observationAbout,wikidataId/Q386635 +25203,observationAbout,wikidataId/Q844130 +25204,observationAbout,wikidataId/Q385461 +25205,#IgnoreRow,True,#ignore,unmapped historical municipality +25206,observationAbout,wikidataId/Q386677 +25207,observationAbout,wikidataId/Q857225 +25208,observationAbout,wikidataId/Q857310 +25209,observationAbout,wikidataId/Q847584 +25210,observationAbout,wikidataId/Q611642 +25211,observationAbout,wikidataId/Q857194 +25212,observationAbout,wikidataId/Q845794 +25213,observationAbout,wikidataId/Q240891 +25214,observationAbout,wikidataId/Q824020 +25301,#IgnoreRow,True,#ignore,unmapped historical municipality +25321,#IgnoreRow,True,#ignore,unmapped historical municipality +25342,#IgnoreRow,True,#ignore,unmapped historical municipality +25343,#IgnoreRow,True,#ignore,unmapped historical municipality +25361,#IgnoreRow,True,#ignore,unmapped historical municipality +25362,#IgnoreRow,True,#ignore,unmapped historical municipality +25363,#IgnoreRow,True,#ignore,unmapped historical municipality +25364,#IgnoreRow,True,#ignore,unmapped historical municipality +25365,#IgnoreRow,True,#ignore,unmapped historical municipality +25366,#IgnoreRow,True,#ignore,unmapped historical municipality +25367,#IgnoreRow,True,#ignore,unmapped historical municipality +25381,#IgnoreRow,True,#ignore,unmapped historical municipality +25382,#IgnoreRow,True,#ignore,unmapped historical municipality +25383,observationAbout,wikidataId/Q1208020 +25384,observationAbout,wikidataId/Q1087560 +25401,#IgnoreRow,True,#ignore,unmapped historical municipality +25402,#IgnoreRow,True,#ignore,unmapped historical municipality +25403,#IgnoreRow,True,#ignore,unmapped historical municipality +25421,#IgnoreRow,True,#ignore,unmapped historical municipality +25422,#IgnoreRow,True,#ignore,unmapped historical municipality +25423,#IgnoreRow,True,#ignore,unmapped historical municipality +25424,#IgnoreRow,True,#ignore,unmapped historical municipality +25425,observationAbout,wikidataId/Q1346806 +25441,observationAbout,wikidataId/Q937037 +25442,observationAbout,wikidataId/Q1354747 +25443,observationAbout,wikidataId/Q1346747 +25461,#IgnoreRow,True,#ignore,unmapped historical municipality +25462,#IgnoreRow,True,#ignore,unmapped historical municipality +25463,#IgnoreRow,True,#ignore,unmapped historical municipality +25464,#IgnoreRow,True,#ignore,unmapped historical municipality +25481,#IgnoreRow,True,#ignore,unmapped historical municipality +25482,#IgnoreRow,True,#ignore,unmapped historical municipality +25483,#IgnoreRow,True,#ignore,unmapped historical municipality +25484,#IgnoreRow,True,#ignore,unmapped historical municipality +25501,#IgnoreRow,True,#ignore,unmapped historical municipality +25502,#IgnoreRow,True,#ignore,unmapped historical municipality +25503,#IgnoreRow,True,#ignore,unmapped historical municipality +25504,#IgnoreRow,True,#ignore,unmapped historical municipality +25521,#IgnoreRow,True,#ignore,unmapped historical municipality +25522,#IgnoreRow,True,#ignore,unmapped historical municipality +25523,#IgnoreRow,True,#ignore,unmapped historical municipality +25524,#IgnoreRow,True,#ignore,unmapped historical municipality +25525,#IgnoreRow,True,#ignore,unmapped historical municipality +25526,#IgnoreRow,True,#ignore,unmapped historical municipality +26000,observationAbout,wikidataId/Q120730 +26100,observationAbout,wikidataId/Q34600 +26101,observationAbout,wikidataId/Q1154970 +26102,observationAbout,wikidataId/Q1321824 +26103,observationAbout,wikidataId/Q1321832 +26104,observationAbout,wikidataId/Q1075455 +26105,observationAbout,wikidataId/Q1203862 +26106,observationAbout,wikidataId/Q1077022 +26107,observationAbout,wikidataId/Q1154649 +26108,observationAbout,wikidataId/Q580186 +26109,observationAbout,wikidataId/Q1197771 +26110,observationAbout,wikidataId/Q1076993 +26111,observationAbout,wikidataId/Q1154588 +26201,observationAbout,wikidataId/Q3355306 +26202,observationAbout,wikidataId/Q750573 +26203,observationAbout,wikidataId/Q694370 +26204,observationAbout,wikidataId/Q326956 +26205,observationAbout,wikidataId/Q846351 +26206,observationAbout,wikidataId/Q633996 +26207,observationAbout,wikidataId/Q817276 +26208,observationAbout,wikidataId/Q850640 +26209,observationAbout,wikidataId/Q861246 +26210,observationAbout,wikidataId/Q260923 +26211,observationAbout,wikidataId/Q816728 +26212,observationAbout,wikidataId/Q513897 +26213,observationAbout,wikidataId/Q170421 +26214,observationAbout,wikidataId/Q990352 +26303,observationAbout,wikidataId/Q450287 +26322,observationAbout,wikidataId/Q1203490 +26342,#IgnoreRow,True,#ignore,unmapped historical municipality +26343,observationAbout,wikidataId/Q1203097 +26344,observationAbout,wikidataId/Q1203224 +26361,#IgnoreRow,True,#ignore,unmapped historical municipality +26362,#IgnoreRow,True,#ignore,unmapped historical municipality +26363,#IgnoreRow,True,#ignore,unmapped historical municipality +26364,observationAbout,wikidataId/Q1203404 +26365,observationAbout,wikidataId/Q1203583 +26366,observationAbout,wikidataId/Q1203237 +26367,observationAbout,wikidataId/Q1203714 +26381,#IgnoreRow,True,#ignore,unmapped historical municipality +26382,#IgnoreRow,True,#ignore,unmapped historical municipality +26401,#IgnoreRow,True,#ignore,unmapped historical municipality +26402,#IgnoreRow,True,#ignore,unmapped historical municipality +26403,#IgnoreRow,True,#ignore,unmapped historical municipality +26404,#IgnoreRow,True,#ignore,unmapped historical municipality +26405,#IgnoreRow,True,#ignore,unmapped historical municipality +26406,#IgnoreRow,True,#ignore,unmapped historical municipality +26407,observationAbout,wikidataId/Q1203236 +26421,#IgnoreRow,True,#ignore,unmapped historical municipality +26422,#IgnoreRow,True,#ignore,unmapped historical municipality +26441,#IgnoreRow,True,#ignore,unmapped historical municipality +26461,#IgnoreRow,True,#ignore,unmapped historical municipality +26462,#IgnoreRow,True,#ignore,unmapped historical municipality +26463,observationAbout,wikidataId/Q1345614 +26464,#IgnoreRow,True,#ignore,unmapped historical municipality +26465,observationAbout,wikidataId/Q1203259 +26481,#IgnoreRow,True,#ignore,unmapped historical municipality +26482,#IgnoreRow,True,#ignore,unmapped historical municipality +26501,#IgnoreRow,True,#ignore,unmapped historical municipality +26502,#IgnoreRow,True,#ignore,unmapped historical municipality +26503,#IgnoreRow,True,#ignore,unmapped historical municipality +26521,#IgnoreRow,True,#ignore,unmapped historical municipality +27000,observationAbout,wikidataId/Q122723 +27100,observationAbout,wikidataId/Q35765 +27101,#IgnoreRow,True,#ignore,unmapped historical municipality +27102,observationAbout,wikidataId/Q490581 +27103,observationAbout,wikidataId/Q985728 +27104,observationAbout,wikidataId/Q960258 +27105,#IgnoreRow,True,#ignore,unmapped historical municipality +27106,observationAbout,wikidataId/Q275123 +27107,observationAbout,wikidataId/Q1365114 +27108,observationAbout,wikidataId/Q570027 +27109,observationAbout,wikidataId/Q490592 +27110,#IgnoreRow,True,#ignore,unmapped historical municipality +27111,observationAbout,wikidataId/Q861618 +27112,#IgnoreRow,True,#ignore,unmapped historical municipality +27113,observationAbout,wikidataId/Q985765 +27114,observationAbout,wikidataId/Q490514 +27115,observationAbout,wikidataId/Q985701 +27116,observationAbout,wikidataId/Q490520 +27117,observationAbout,wikidataId/Q861271 +27118,observationAbout,wikidataId/Q490508 +27119,observationAbout,wikidataId/Q489009 +27120,observationAbout,wikidataId/Q985702 +27121,observationAbout,wikidataId/Q861264 +27122,observationAbout,wikidataId/Q985695 +27123,observationAbout,wikidataId/Q490594 +27124,observationAbout,wikidataId/Q490526 +27125,observationAbout,wikidataId/Q985711 +27126,observationAbout,wikidataId/Q861256 +27127,observationAbout,wikidataId/Q1207746 +27128,observationAbout,wikidataId/Q876485 +27140,observationAbout,wikidataId/Q193428 +27141,observationAbout,wikidataId/Q1042008 +27142,observationAbout,wikidataId/Q1042487 +27143,observationAbout,wikidataId/Q1042014 +27144,observationAbout,wikidataId/Q1042499 +27145,observationAbout,wikidataId/Q1041996 +27146,observationAbout,wikidataId/Q1042504 +27147,observationAbout,wikidataId/Q1042511 +27201,#IgnoreRow,True,#ignore,unmapped historical municipality +27202,observationAbout,wikidataId/Q740456 +27203,observationAbout,wikidataId/Q467479 +27204,observationAbout,wikidataId/Q817965 +27205,observationAbout,wikidataId/Q653510 +27206,observationAbout,wikidataId/Q873553 +27207,observationAbout,wikidataId/Q335423 +27208,observationAbout,wikidataId/Q908972 +27209,observationAbout,wikidataId/Q694154 +27210,observationAbout,wikidataId/Q270912 +27211,observationAbout,wikidataId/Q502403 +27212,observationAbout,wikidataId/Q490872 +27213,observationAbout,wikidataId/Q852018 +27214,observationAbout,wikidataId/Q490928 +27215,observationAbout,wikidataId/Q389633 +27216,observationAbout,wikidataId/Q819572 +27217,observationAbout,wikidataId/Q774952 +27218,observationAbout,wikidataId/Q386668 +27219,observationAbout,wikidataId/Q696412 +27220,observationAbout,wikidataId/Q665097 +27221,observationAbout,wikidataId/Q873583 +27222,observationAbout,wikidataId/Q694308 +27223,observationAbout,wikidataId/Q496080 +27224,observationAbout,wikidataId/Q643878 +27225,observationAbout,wikidataId/Q857218 +27226,observationAbout,wikidataId/Q634011 +27227,observationAbout,wikidataId/Q243863 +27228,observationAbout,wikidataId/Q838809 +27229,observationAbout,wikidataId/Q742160 +27230,observationAbout,wikidataId/Q819646 +27231,observationAbout,wikidataId/Q385864 +27232,observationAbout,wikidataId/Q819670 +27301,observationAbout,wikidataId/Q1347149 +27321,observationAbout,wikidataId/Q1352491 +27322,observationAbout,wikidataId/Q1203349 +27341,observationAbout,wikidataId/Q1347139 +27361,observationAbout,wikidataId/Q1347195 +27362,observationAbout,wikidataId/Q1347129 +27366,observationAbout,wikidataId/Q1358900 +27367,#IgnoreRow,True,#ignore,unmapped historical municipality +27381,observationAbout,wikidataId/Q1347173 +27382,observationAbout,wikidataId/Q2305508 +27383,observationAbout,wikidataId/Q1072574 +27384,#IgnoreRow,True,#ignore,unmapped historical municipality +27385,#IgnoreRow,True,#ignore,unmapped historical municipality +28000,observationAbout,wikidataId/Q130290 +28100,observationAbout,wikidataId/Q48320 +28101,observationAbout,wikidataId/Q277434 +28102,observationAbout,wikidataId/Q1154555 +28103,#IgnoreRow,True,#ignore,unmapped historical municipality +28104,#IgnoreRow,True,#ignore,unmapped historical municipality +28105,observationAbout,wikidataId/Q1078790 +28106,observationAbout,wikidataId/Q1078801 +28107,observationAbout,wikidataId/Q1155010 +28108,observationAbout,wikidataId/Q1078806 +28109,observationAbout,wikidataId/Q1078814 +28110,observationAbout,wikidataId/Q686802 +28111,observationAbout,wikidataId/Q1154552 +28201,observationAbout,wikidataId/Q209801 +28202,observationAbout,wikidataId/Q213318 +28203,observationAbout,wikidataId/Q26406 +28204,observationAbout,wikidataId/Q243853 +28205,observationAbout,wikidataId/Q369628 +28206,observationAbout,wikidataId/Q725514 +28207,observationAbout,wikidataId/Q335444 +28208,observationAbout,wikidataId/Q406318 +28209,observationAbout,wikidataId/Q696388 +28210,observationAbout,wikidataId/Q750569 +28211,#IgnoreRow,True,#ignore,unmapped historical municipality +28212,observationAbout,wikidataId/Q424813 +28213,observationAbout,wikidataId/Q869020 +28214,observationAbout,wikidataId/Q823967 +28215,observationAbout,wikidataId/Q31238 +28216,observationAbout,wikidataId/Q869037 +28217,observationAbout,wikidataId/Q609350 +28218,observationAbout,wikidataId/Q911717 +28219,observationAbout,wikidataId/Q847607 +28220,observationAbout,wikidataId/Q851155 +28221,observationAbout,wikidataId/Q877169 +28222,observationAbout,wikidataId/Q869014 +28223,observationAbout,wikidataId/Q877196 +28224,observationAbout,wikidataId/Q869004 +28225,observationAbout,wikidataId/Q720402 +28226,observationAbout,wikidataId/Q373107 +28227,observationAbout,wikidataId/Q877216 +28228,observationAbout,wikidataId/Q877164 +28229,observationAbout,wikidataId/Q877190 +28301,observationAbout,wikidataId/Q1155267 +28321,#IgnoreRow,True,#ignore,unmapped historical municipality +28341,#IgnoreRow,True,#ignore,unmapped historical municipality +28342,#IgnoreRow,True,#ignore,unmapped historical municipality +28343,#IgnoreRow,True,#ignore,unmapped historical municipality +28361,#IgnoreRow,True,#ignore,unmapped historical municipality +28362,#IgnoreRow,True,#ignore,unmapped historical municipality +28363,#IgnoreRow,True,#ignore,unmapped historical municipality +28364,#IgnoreRow,True,#ignore,unmapped historical municipality +28365,observationAbout,wikidataId/Q1155253 +28381,observationAbout,wikidataId/Q1155194 +28382,observationAbout,wikidataId/Q1155297 +28421,#IgnoreRow,True,#ignore,unmapped historical municipality +28422,#IgnoreRow,True,#ignore,unmapped historical municipality +28441,#IgnoreRow,True,#ignore,unmapped historical municipality +28442,observationAbout,wikidataId/Q1203688 +28443,observationAbout,wikidataId/Q1207159 +28444,#IgnoreRow,True,#ignore,unmapped historical municipality +28445,#IgnoreRow,True,#ignore,unmapped historical municipality +28446,observationAbout,wikidataId/Q1155215 +28461,#IgnoreRow,True,#ignore,unmapped historical municipality +28462,#IgnoreRow,True,#ignore,unmapped historical municipality +28463,#IgnoreRow,True,#ignore,unmapped historical municipality +28464,observationAbout,wikidataId/Q1155186 +28481,observationAbout,wikidataId/Q1155179 +28501,observationAbout,wikidataId/Q1184480 +28502,#IgnoreRow,True,#ignore,unmapped historical municipality +28503,#IgnoreRow,True,#ignore,unmapped historical municipality +28504,#IgnoreRow,True,#ignore,unmapped historical municipality +28521,#IgnoreRow,True,#ignore,unmapped historical municipality +28522,#IgnoreRow,True,#ignore,unmapped historical municipality +28523,#IgnoreRow,True,#ignore,unmapped historical municipality +28524,#IgnoreRow,True,#ignore,unmapped historical municipality +28525,#IgnoreRow,True,#ignore,unmapped historical municipality +28541,#IgnoreRow,True,#ignore,unmapped historical municipality +28542,#IgnoreRow,True,#ignore,unmapped historical municipality +28543,#IgnoreRow,True,#ignore,unmapped historical municipality +28544,#IgnoreRow,True,#ignore,unmapped historical municipality +28561,#IgnoreRow,True,#ignore,unmapped historical municipality +28562,#IgnoreRow,True,#ignore,unmapped historical municipality +28581,#IgnoreRow,True,#ignore,unmapped historical municipality +28582,#IgnoreRow,True,#ignore,unmapped historical municipality +28583,#IgnoreRow,True,#ignore,unmapped historical municipality +28584,#IgnoreRow,True,#ignore,unmapped historical municipality +28585,observationAbout,wikidataId/Q1155233 +28586,observationAbout,wikidataId/Q1154955 +28601,#IgnoreRow,True,#ignore,unmapped historical municipality +28602,#IgnoreRow,True,#ignore,unmapped historical municipality +28603,#IgnoreRow,True,#ignore,unmapped historical municipality +28604,#IgnoreRow,True,#ignore,unmapped historical municipality +28621,#IgnoreRow,True,#ignore,unmapped historical municipality +28622,#IgnoreRow,True,#ignore,unmapped historical municipality +28623,#IgnoreRow,True,#ignore,unmapped historical municipality +28624,#IgnoreRow,True,#ignore,unmapped historical municipality +28641,#IgnoreRow,True,#ignore,unmapped historical municipality +28642,#IgnoreRow,True,#ignore,unmapped historical municipality +28643,#IgnoreRow,True,#ignore,unmapped historical municipality +28644,#IgnoreRow,True,#ignore,unmapped historical municipality +28645,#IgnoreRow,True,#ignore,unmapped historical municipality +28646,#IgnoreRow,True,#ignore,unmapped historical municipality +28661,#IgnoreRow,True,#ignore,unmapped historical municipality +28664,#IgnoreRow,True,#ignore,unmapped historical municipality +28665,#IgnoreRow,True,#ignore,unmapped historical municipality +28666,#IgnoreRow,True,#ignore,unmapped historical municipality +28681,#IgnoreRow,True,#ignore,unmapped historical municipality +28682,#IgnoreRow,True,#ignore,unmapped historical municipality +28683,#IgnoreRow,True,#ignore,unmapped historical municipality +28684,#IgnoreRow,True,#ignore,unmapped historical municipality +28685,#IgnoreRow,True,#ignore,unmapped historical municipality +28686,#IgnoreRow,True,#ignore,unmapped historical municipality +28701,#IgnoreRow,True,#ignore,unmapped historical municipality +28702,#IgnoreRow,True,#ignore,unmapped historical municipality +28703,#IgnoreRow,True,#ignore,unmapped historical municipality +28704,#IgnoreRow,True,#ignore,unmapped historical municipality +29000,observationAbout,wikidataId/Q131287 +29201,observationAbout,wikidataId/Q169134 +29202,observationAbout,wikidataId/Q838669 +29203,observationAbout,wikidataId/Q118213 +29204,observationAbout,wikidataId/Q502196 +29205,observationAbout,wikidataId/Q496319 +29206,observationAbout,wikidataId/Q127026 +29207,observationAbout,wikidataId/Q602571 +29208,observationAbout,wikidataId/Q847334 +29209,observationAbout,wikidataId/Q328610 +29210,observationAbout,wikidataId/Q696643 +29211,observationAbout,wikidataId/Q659672 +29212,observationAbout,wikidataId/Q863747 +29301,#IgnoreRow,True,#ignore,unmapped historical municipality +29321,#IgnoreRow,True,#ignore,unmapped historical municipality +29322,observationAbout,wikidataId/Q1203208 +29342,observationAbout,wikidataId/Q1346790 +29343,observationAbout,wikidataId/Q1203139 +29344,observationAbout,wikidataId/Q379233 +29345,observationAbout,wikidataId/Q1203680 +29361,observationAbout,wikidataId/Q1203732 +29362,observationAbout,wikidataId/Q1203513 +29363,observationAbout,wikidataId/Q1203159 +29381,#IgnoreRow,True,#ignore,unmapped historical municipality +29382,#IgnoreRow,True,#ignore,unmapped historical municipality +29383,#IgnoreRow,True,#ignore,unmapped historical municipality +29384,#IgnoreRow,True,#ignore,unmapped historical municipality +29385,observationAbout,wikidataId/Q305098 +29386,observationAbout,wikidataId/Q930076 +29401,observationAbout,wikidataId/Q1202681 +29402,observationAbout,wikidataId/Q752397 +29421,#IgnoreRow,True,#ignore,unmapped historical municipality +29422,#IgnoreRow,True,#ignore,unmapped historical municipality +29423,#IgnoreRow,True,#ignore,unmapped historical municipality +29424,observationAbout,wikidataId/Q1203559 +29425,observationAbout,wikidataId/Q1346392 +29426,observationAbout,wikidataId/Q1346830 +29427,observationAbout,wikidataId/Q1203880 +29441,observationAbout,wikidataId/Q1203731 +29442,observationAbout,wikidataId/Q629510 +29443,observationAbout,wikidataId/Q1203064 +29444,observationAbout,wikidataId/Q1203154 +29445,#IgnoreRow,True,#ignore,unmapped historical municipality +29446,observationAbout,wikidataId/Q1203501 +29447,observationAbout,wikidataId/Q1203246 +29448,#IgnoreRow,True,#ignore,unmapped historical municipality +29449,observationAbout,wikidataId/Q1202740 +29450,observationAbout,wikidataId/Q1203760 +29451,observationAbout,wikidataId/Q1203167 +29452,observationAbout,wikidataId/Q1204424 +29453,observationAbout,wikidataId/Q1203148 +30000,observationAbout,wikidataId/Q131314 +30201,observationAbout,wikidataId/Q200747 +30202,observationAbout,wikidataId/Q841309 +30203,observationAbout,wikidataId/Q842098 +30204,observationAbout,wikidataId/Q657843 +30205,observationAbout,wikidataId/Q847572 +30206,observationAbout,wikidataId/Q304266 +30207,observationAbout,wikidataId/Q823526 +30208,observationAbout,wikidataId/Q697002 +30209,observationAbout,wikidataId/Q579623 +30301,#IgnoreRow,True,#ignore,unmapped historical municipality +30302,#IgnoreRow,True,#ignore,unmapped historical municipality +30303,#IgnoreRow,True,#ignore,unmapped historical municipality +30304,observationAbout,wikidataId/Q1354884 +30321,#IgnoreRow,True,#ignore,unmapped historical municipality +30322,#IgnoreRow,True,#ignore,unmapped historical municipality +30323,#IgnoreRow,True,#ignore,unmapped historical municipality +30324,#IgnoreRow,True,#ignore,unmapped historical municipality +30325,#IgnoreRow,True,#ignore,unmapped historical municipality +30326,#IgnoreRow,True,#ignore,unmapped historical municipality +30341,observationAbout,wikidataId/Q1345594 +30342,#IgnoreRow,True,#ignore,unmapped historical municipality +30343,observationAbout,wikidataId/Q1354976 +30344,observationAbout,wikidataId/Q1346907 +30345,#IgnoreRow,True,#ignore,unmapped historical municipality +30361,observationAbout,wikidataId/Q1345625 +30362,observationAbout,wikidataId/Q1345650 +30363,#IgnoreRow,True,#ignore,unmapped historical municipality +30364,#IgnoreRow,True,#ignore,unmapped historical municipality +30365,#IgnoreRow,True,#ignore,unmapped historical municipality +30366,observationAbout,wikidataId/Q1345889 +30381,observationAbout,wikidataId/Q1203565 +30382,observationAbout,wikidataId/Q1345541 +30383,observationAbout,wikidataId/Q1203250 +30384,#IgnoreRow,True,#ignore,unmapped historical municipality +30385,#IgnoreRow,True,#ignore,unmapped historical municipality +30386,#IgnoreRow,True,#ignore,unmapped historical municipality +30387,#IgnoreRow,True,#ignore,unmapped historical municipality +30388,#IgnoreRow,True,#ignore,unmapped historical municipality +30389,#IgnoreRow,True,#ignore,unmapped historical municipality +30390,observationAbout,wikidataId/Q1203734 +30391,observationAbout,wikidataId/Q1345966 +30392,observationAbout,wikidataId/Q1345505 +30401,observationAbout,wikidataId/Q1202728 +30402,#IgnoreRow,True,#ignore,unmapped historical municipality +30403,#IgnoreRow,True,#ignore,unmapped historical municipality +30404,observationAbout,wikidataId/Q1346882 +30405,#IgnoreRow,True,#ignore,unmapped historical municipality +30406,observationAbout,wikidataId/Q1346937 +30407,#IgnoreRow,True,#ignore,unmapped historical municipality +30421,observationAbout,wikidataId/Q1346730 +30422,observationAbout,wikidataId/Q1023770 +30423,#IgnoreRow,True,#ignore,unmapped historical municipality +30424,observationAbout,wikidataId/Q1346858 +30425,#IgnoreRow,True,#ignore,unmapped historical municipality +30426,#IgnoreRow,True,#ignore,unmapped historical municipality +30427,observationAbout,wikidataId/Q1354841 +30428,observationAbout,wikidataId/Q1343733 +31000,observationAbout,wikidataId/Q133935 +31201,observationAbout,wikidataId/Q200731 +31202,observationAbout,wikidataId/Q320964 +31203,observationAbout,wikidataId/Q642006 +31204,observationAbout,wikidataId/Q496431 +31301,#IgnoreRow,True,#ignore,unmapped historical municipality +31302,observationAbout,wikidataId/Q281925 +31303,#IgnoreRow,True,#ignore,unmapped historical municipality +31321,#IgnoreRow,True,#ignore,unmapped historical municipality +31322,#IgnoreRow,True,#ignore,unmapped historical municipality +31323,#IgnoreRow,True,#ignore,unmapped historical municipality +31324,#IgnoreRow,True,#ignore,unmapped historical municipality +31325,observationAbout,wikidataId/Q1156116 +31326,#IgnoreRow,True,#ignore,unmapped historical municipality +31327,#IgnoreRow,True,#ignore,unmapped historical municipality +31328,observationAbout,wikidataId/Q1156212 +31329,observationAbout,wikidataId/Q1156136 +31341,#IgnoreRow,True,#ignore,unmapped historical municipality +31342,#IgnoreRow,True,#ignore,unmapped historical municipality +31343,#IgnoreRow,True,#ignore,unmapped historical municipality +31361,#IgnoreRow,True,#ignore,unmapped historical municipality +31362,#IgnoreRow,True,#ignore,unmapped historical municipality +31363,#IgnoreRow,True,#ignore,unmapped historical municipality +31364,observationAbout,wikidataId/Q1156186 +31365,#IgnoreRow,True,#ignore,unmapped historical municipality +31366,#IgnoreRow,True,#ignore,unmapped historical municipality +31367,#IgnoreRow,True,#ignore,unmapped historical municipality +31368,#IgnoreRow,True,#ignore,unmapped historical municipality +31369,#IgnoreRow,True,#ignore,unmapped historical municipality +31370,observationAbout,wikidataId/Q1156158 +31371,observationAbout,wikidataId/Q1204853 +31372,observationAbout,wikidataId/Q1205642 +31381,#IgnoreRow,True,#ignore,unmapped historical municipality +31382,#IgnoreRow,True,#ignore,unmapped historical municipality +31383,#IgnoreRow,True,#ignore,unmapped historical municipality +31384,observationAbout,wikidataId/Q1204441 +31385,#IgnoreRow,True,#ignore,unmapped historical municipality +31386,observationAbout,wikidataId/Q1204975 +31387,#IgnoreRow,True,#ignore,unmapped historical municipality +31388,#IgnoreRow,True,#ignore,unmapped historical municipality +31389,observationAbout,wikidataId/Q1156169 +31390,observationAbout,wikidataId/Q1204493 +31401,observationAbout,wikidataId/Q1204860 +31402,observationAbout,wikidataId/Q1156127 +31403,observationAbout,wikidataId/Q1156182 +31404,#IgnoreRow,True,#ignore,unmapped historical municipality +32000,observationAbout,wikidataId/Q132751 +32201,observationAbout,wikidataId/Q207321 +32202,observationAbout,wikidataId/Q432055 +32203,observationAbout,wikidataId/Q825166 +32204,observationAbout,wikidataId/Q753731 +32205,observationAbout,wikidataId/Q384796 +32206,observationAbout,wikidataId/Q851114 +32207,observationAbout,wikidataId/Q850395 +32208,#IgnoreRow,True,#ignore,unmapped historical municipality +32209,observationAbout,wikidataId/Q877935 +32301,#IgnoreRow,True,#ignore,unmapped historical municipality +32302,#IgnoreRow,True,#ignore,unmapped historical municipality +32303,#IgnoreRow,True,#ignore,unmapped historical municipality +32304,#IgnoreRow,True,#ignore,unmapped historical municipality +32305,#IgnoreRow,True,#ignore,unmapped historical municipality +32306,#IgnoreRow,True,#ignore,unmapped historical municipality +32307,#IgnoreRow,True,#ignore,unmapped historical municipality +32308,#IgnoreRow,True,#ignore,unmapped historical municipality +32321,#IgnoreRow,True,#ignore,unmapped historical municipality +32322,#IgnoreRow,True,#ignore,unmapped historical municipality +32341,#IgnoreRow,True,#ignore,unmapped historical municipality +32342,#IgnoreRow,True,#ignore,unmapped historical municipality +32343,observationAbout,wikidataId/Q1156216 +32361,#IgnoreRow,True,#ignore,unmapped historical municipality +32362,#IgnoreRow,True,#ignore,unmapped historical municipality +32363,#IgnoreRow,True,#ignore,unmapped historical municipality +32381,#IgnoreRow,True,#ignore,unmapped historical municipality +32382,#IgnoreRow,True,#ignore,unmapped historical municipality +32383,#IgnoreRow,True,#ignore,unmapped historical municipality +32384,#IgnoreRow,True,#ignore,unmapped historical municipality +32385,#IgnoreRow,True,#ignore,unmapped historical municipality +32386,observationAbout,wikidataId/Q660516 +32401,#IgnoreRow,True,#ignore,unmapped historical municipality +32402,#IgnoreRow,True,#ignore,unmapped historical municipality +32403,#IgnoreRow,True,#ignore,unmapped historical municipality +32404,#IgnoreRow,True,#ignore,unmapped historical municipality +32405,#IgnoreRow,True,#ignore,unmapped historical municipality +32421,#IgnoreRow,True,#ignore,unmapped historical municipality +32422,#IgnoreRow,True,#ignore,unmapped historical municipality +32441,observationAbout,wikidataId/Q1156174 +32442,#IgnoreRow,True,#ignore,unmapped historical municipality +32443,#IgnoreRow,True,#ignore,unmapped historical municipality +32444,#IgnoreRow,True,#ignore,unmapped historical municipality +32445,#IgnoreRow,True,#ignore,unmapped historical municipality +32446,#IgnoreRow,True,#ignore,unmapped historical municipality +32447,#IgnoreRow,True,#ignore,unmapped historical municipality +32448,observationAbout,wikidataId/Q1156165 +32449,observationAbout,wikidataId/Q385598 +32462,#IgnoreRow,True,#ignore,unmapped historical municipality +32463,#IgnoreRow,True,#ignore,unmapped historical municipality +32464,#IgnoreRow,True,#ignore,unmapped historical municipality +32465,#IgnoreRow,True,#ignore,unmapped historical municipality +32481,#IgnoreRow,True,#ignore,unmapped historical municipality +32482,#IgnoreRow,True,#ignore,unmapped historical municipality +32501,observationAbout,wikidataId/Q640586 +32502,#IgnoreRow,True,#ignore,unmapped historical municipality +32503,#IgnoreRow,True,#ignore,unmapped historical municipality +32504,#IgnoreRow,True,#ignore,unmapped historical municipality +32505,observationAbout,wikidataId/Q279149 +32521,#IgnoreRow,True,#ignore,unmapped historical municipality +32522,#IgnoreRow,True,#ignore,unmapped historical municipality +32523,#IgnoreRow,True,#ignore,unmapped historical municipality +32524,#IgnoreRow,True,#ignore,unmapped historical municipality +32525,observationAbout,wikidataId/Q685865 +32526,observationAbout,wikidataId/Q543193 +32527,observationAbout,wikidataId/Q1204968 +32528,observationAbout,wikidataId/Q1205039 +33000,observationAbout,wikidataId/Q132936 +33100,observationAbout,wikidataId/Q200078 +33101,observationAbout,wikidataId/Q855478 +33102,observationAbout,wikidataId/Q852116 +33103,observationAbout,wikidataId/Q855491 +33104,observationAbout,wikidataId/Q1042680 +33201,#IgnoreRow,True,#ignore,unmapped historical municipality +33202,observationAbout,wikidataId/Q467487 +33203,observationAbout,wikidataId/Q841125 +33204,observationAbout,wikidataId/Q850335 +33205,observationAbout,wikidataId/Q863739 +33207,observationAbout,wikidataId/Q863754 +33208,observationAbout,wikidataId/Q873536 +33209,observationAbout,wikidataId/Q851675 +33210,observationAbout,wikidataId/Q873596 +33211,observationAbout,wikidataId/Q863741 +33212,observationAbout,wikidataId/Q691691 +33213,observationAbout,wikidataId/Q415372 +33214,observationAbout,wikidataId/Q861230 +33215,observationAbout,wikidataId/Q861279 +33216,observationAbout,wikidataId/Q990353 +33301,#IgnoreRow,True,#ignore,unmapped historical municipality +33303,#IgnoreRow,True,#ignore,unmapped historical municipality +33305,#IgnoreRow,True,#ignore,unmapped historical municipality +33321,#IgnoreRow,True,#ignore,unmapped historical municipality +33322,#IgnoreRow,True,#ignore,unmapped historical municipality +33323,#IgnoreRow,True,#ignore,unmapped historical municipality +33324,#IgnoreRow,True,#ignore,unmapped historical municipality +33325,#IgnoreRow,True,#ignore,unmapped historical municipality +33342,#IgnoreRow,True,#ignore,unmapped historical municipality +33344,#IgnoreRow,True,#ignore,unmapped historical municipality +33345,#IgnoreRow,True,#ignore,unmapped historical municipality +33346,observationAbout,wikidataId/Q1156120 +33361,#IgnoreRow,True,#ignore,unmapped historical municipality +33362,#IgnoreRow,True,#ignore,unmapped historical municipality +33363,#IgnoreRow,True,#ignore,unmapped historical municipality +33401,#IgnoreRow,True,#ignore,unmapped historical municipality +33423,observationAbout,wikidataId/Q1155361 +33427,#IgnoreRow,True,#ignore,unmapped historical municipality +33428,#IgnoreRow,True,#ignore,unmapped historical municipality +33441,#IgnoreRow,True,#ignore,unmapped historical municipality +33442,#IgnoreRow,True,#ignore,unmapped historical municipality +33443,#IgnoreRow,True,#ignore,unmapped historical municipality +33444,#IgnoreRow,True,#ignore,unmapped historical municipality +33445,observationAbout,wikidataId/Q1155044 +33461,observationAbout,wikidataId/Q1155340 +33462,#IgnoreRow,True,#ignore,unmapped historical municipality +33481,#IgnoreRow,True,#ignore,unmapped historical municipality +33503,#IgnoreRow,True,#ignore,unmapped historical municipality +33521,#IgnoreRow,True,#ignore,unmapped historical municipality +33522,#IgnoreRow,True,#ignore,unmapped historical municipality +33523,#IgnoreRow,True,#ignore,unmapped historical municipality +33541,#IgnoreRow,True,#ignore,unmapped historical municipality +33542,#IgnoreRow,True,#ignore,unmapped historical municipality +33543,#IgnoreRow,True,#ignore,unmapped historical municipality +33561,#IgnoreRow,True,#ignore,unmapped historical municipality +33562,#IgnoreRow,True,#ignore,unmapped historical municipality +33563,#IgnoreRow,True,#ignore,unmapped historical municipality +33564,#IgnoreRow,True,#ignore,unmapped historical municipality +33581,#IgnoreRow,True,#ignore,unmapped historical municipality +33582,#IgnoreRow,True,#ignore,unmapped historical municipality +33583,#IgnoreRow,True,#ignore,unmapped historical municipality +33584,#IgnoreRow,True,#ignore,unmapped historical municipality +33585,#IgnoreRow,True,#ignore,unmapped historical municipality +33586,observationAbout,wikidataId/Q1156191 +33587,#IgnoreRow,True,#ignore,unmapped historical municipality +33588,#IgnoreRow,True,#ignore,unmapped historical municipality +33589,#IgnoreRow,True,#ignore,unmapped historical municipality +33601,#IgnoreRow,True,#ignore,unmapped historical municipality +33602,#IgnoreRow,True,#ignore,unmapped historical municipality +33603,#IgnoreRow,True,#ignore,unmapped historical municipality +33604,#IgnoreRow,True,#ignore,unmapped historical municipality +33605,#IgnoreRow,True,#ignore,unmapped historical municipality +33606,observationAbout,wikidataId/Q1155346 +33621,#IgnoreRow,True,#ignore,unmapped historical municipality +33622,observationAbout,wikidataId/Q1156223 +33623,observationAbout,wikidataId/Q1156096 +33624,#IgnoreRow,True,#ignore,unmapped historical municipality +33641,#IgnoreRow,True,#ignore,unmapped historical municipality +33642,#IgnoreRow,True,#ignore,unmapped historical municipality +33643,observationAbout,wikidataId/Q1156206 +33644,#IgnoreRow,True,#ignore,unmapped historical municipality +33645,#IgnoreRow,True,#ignore,unmapped historical municipality +33646,#IgnoreRow,True,#ignore,unmapped historical municipality +33661,#IgnoreRow,True,#ignore,unmapped historical municipality +33662,#IgnoreRow,True,#ignore,unmapped historical municipality +33663,observationAbout,wikidataId/Q1156111 +33664,#IgnoreRow,True,#ignore,unmapped historical municipality +33665,#IgnoreRow,True,#ignore,unmapped historical municipality +33666,observationAbout,wikidataId/Q1156100 +33681,observationAbout,wikidataId/Q1156219 +34000,observationAbout,wikidataId/Q617375 +34100,observationAbout,wikidataId/Q34664 +34101,observationAbout,wikidataId/Q63436 +34102,observationAbout,wikidataId/Q73724 +34103,observationAbout,wikidataId/Q63442 +34104,observationAbout,wikidataId/Q73729 +34105,observationAbout,wikidataId/Q63440 +34106,observationAbout,wikidataId/Q63444 +34107,observationAbout,wikidataId/Q63446 +34108,observationAbout,wikidataId/Q73718 +34202,observationAbout,wikidataId/Q223439 +34203,observationAbout,wikidataId/Q200500 +34204,observationAbout,wikidataId/Q820760 +34205,observationAbout,wikidataId/Q696694 +34206,#IgnoreRow,True,#ignore,unmapped historical municipality +34207,observationAbout,wikidataId/Q661038 +34208,observationAbout,wikidataId/Q686775 +34209,observationAbout,wikidataId/Q840591 +34210,observationAbout,wikidataId/Q857234 +34211,observationAbout,wikidataId/Q386250 +34212,observationAbout,wikidataId/Q653372 +34213,observationAbout,wikidataId/Q736897 +34214,observationAbout,wikidataId/Q419392 +34215,observationAbout,wikidataId/Q865506 +34302,observationAbout,wikidataId/Q1204737 +34304,observationAbout,wikidataId/Q1347353 +34307,observationAbout,wikidataId/Q1204741 +34309,observationAbout,wikidataId/Q1351062 +34310,#IgnoreRow,True,#ignore,unmapped historical municipality +34311,#IgnoreRow,True,#ignore,unmapped historical municipality +34312,#IgnoreRow,True,#ignore,unmapped historical municipality +34313,#IgnoreRow,True,#ignore,unmapped historical municipality +34314,#IgnoreRow,True,#ignore,unmapped historical municipality +34321,#IgnoreRow,True,#ignore,unmapped historical municipality +34322,#IgnoreRow,True,#ignore,unmapped historical municipality +34323,#IgnoreRow,True,#ignore,unmapped historical municipality +34324,#IgnoreRow,True,#ignore,unmapped historical municipality +34325,#IgnoreRow,True,#ignore,unmapped historical municipality +34326,#IgnoreRow,True,#ignore,unmapped historical municipality +34327,#IgnoreRow,True,#ignore,unmapped historical municipality +34328,#IgnoreRow,True,#ignore,unmapped historical municipality +34329,#IgnoreRow,True,#ignore,unmapped historical municipality +34330,#IgnoreRow,True,#ignore,unmapped historical municipality +34361,#IgnoreRow,True,#ignore,unmapped historical municipality +34362,#IgnoreRow,True,#ignore,unmapped historical municipality +34363,#IgnoreRow,True,#ignore,unmapped historical municipality +34364,#IgnoreRow,True,#ignore,unmapped historical municipality +34365,#IgnoreRow,True,#ignore,unmapped historical municipality +34366,#IgnoreRow,True,#ignore,unmapped historical municipality +34367,#IgnoreRow,True,#ignore,unmapped historical municipality +34368,observationAbout,wikidataId/Q1204574 +34369,observationAbout,wikidataId/Q1204518 +34381,#IgnoreRow,True,#ignore,unmapped historical municipality +34382,#IgnoreRow,True,#ignore,unmapped historical municipality +34383,#IgnoreRow,True,#ignore,unmapped historical municipality +34384,#IgnoreRow,True,#ignore,unmapped historical municipality +34385,#IgnoreRow,True,#ignore,unmapped historical municipality +34386,#IgnoreRow,True,#ignore,unmapped historical municipality +34402,#IgnoreRow,True,#ignore,unmapped historical municipality +34405,#IgnoreRow,True,#ignore,unmapped historical municipality +34406,#IgnoreRow,True,#ignore,unmapped historical municipality +34407,#IgnoreRow,True,#ignore,unmapped historical municipality +34408,#IgnoreRow,True,#ignore,unmapped historical municipality +34421,#IgnoreRow,True,#ignore,unmapped historical municipality +34422,#IgnoreRow,True,#ignore,unmapped historical municipality +34423,#IgnoreRow,True,#ignore,unmapped historical municipality +34424,#IgnoreRow,True,#ignore,unmapped historical municipality +34425,#IgnoreRow,True,#ignore,unmapped historical municipality +34426,#IgnoreRow,True,#ignore,unmapped historical municipality +34427,#IgnoreRow,True,#ignore,unmapped historical municipality +34428,#IgnoreRow,True,#ignore,unmapped historical municipality +34429,#IgnoreRow,True,#ignore,unmapped historical municipality +34430,#IgnoreRow,True,#ignore,unmapped historical municipality +34431,observationAbout,wikidataId/Q1205663 +34441,#IgnoreRow,True,#ignore,unmapped historical municipality +34442,#IgnoreRow,True,#ignore,unmapped historical municipality +34444,#IgnoreRow,True,#ignore,unmapped historical municipality +34461,#IgnoreRow,True,#ignore,unmapped historical municipality +34462,observationAbout,wikidataId/Q1349563 +34463,#IgnoreRow,True,#ignore,unmapped historical municipality +34481,#IgnoreRow,True,#ignore,unmapped historical municipality +34482,#IgnoreRow,True,#ignore,unmapped historical municipality +34501,#IgnoreRow,True,#ignore,unmapped historical municipality +34524,#IgnoreRow,True,#ignore,unmapped historical municipality +34541,#IgnoreRow,True,#ignore,unmapped historical municipality +34542,#IgnoreRow,True,#ignore,unmapped historical municipality +34543,#IgnoreRow,True,#ignore,unmapped historical municipality +34544,#IgnoreRow,True,#ignore,unmapped historical municipality +34545,observationAbout,wikidataId/Q1352479 +34561,#IgnoreRow,True,#ignore,unmapped historical municipality +34562,#IgnoreRow,True,#ignore,unmapped historical municipality +34563,#IgnoreRow,True,#ignore,unmapped historical municipality +34581,#IgnoreRow,True,#ignore,unmapped historical municipality +34582,#IgnoreRow,True,#ignore,unmapped historical municipality +34583,#IgnoreRow,True,#ignore,unmapped historical municipality +34584,#IgnoreRow,True,#ignore,unmapped historical municipality +34585,#IgnoreRow,True,#ignore,unmapped historical municipality +34586,#IgnoreRow,True,#ignore,unmapped historical municipality +34601,#IgnoreRow,True,#ignore,unmapped historical municipality +34602,#IgnoreRow,True,#ignore,unmapped historical municipality +34603,#IgnoreRow,True,#ignore,unmapped historical municipality +34604,#IgnoreRow,True,#ignore,unmapped historical municipality +34605,#IgnoreRow,True,#ignore,unmapped historical municipality +35000,observationAbout,wikidataId/Q127264 +35201,observationAbout,wikidataId/Q203308 +35202,observationAbout,wikidataId/Q468765 +35203,observationAbout,wikidataId/Q207876 +35204,observationAbout,wikidataId/Q378225 +35205,#IgnoreRow,True,#ignore,unmapped historical municipality +35206,observationAbout,wikidataId/Q502209 +35207,observationAbout,wikidataId/Q841316 +35208,observationAbout,wikidataId/Q329797 +35209,#IgnoreRow,True,#ignore,unmapped historical municipality +35210,observationAbout,wikidataId/Q823502 +35211,observationAbout,wikidataId/Q390812 +35212,observationAbout,wikidataId/Q819678 +35213,observationAbout,wikidataId/Q749885 +35214,#IgnoreRow,True,#ignore,unmapped historical municipality +35215,observationAbout,wikidataId/Q643929 +35216,observationAbout,wikidataId/Q861239 +35301,#IgnoreRow,True,#ignore,unmapped historical municipality +35302,#IgnoreRow,True,#ignore,unmapped historical municipality +35303,#IgnoreRow,True,#ignore,unmapped historical municipality +35304,#IgnoreRow,True,#ignore,unmapped historical municipality +35305,observationAbout,wikidataId/Q1205034 +35321,observationAbout,wikidataId/Q1354919 +35322,#IgnoreRow,True,#ignore,unmapped historical municipality +35323,#IgnoreRow,True,#ignore,unmapped historical municipality +35324,#IgnoreRow,True,#ignore,unmapped historical municipality +35325,#IgnoreRow,True,#ignore,unmapped historical municipality +35326,#IgnoreRow,True,#ignore,unmapped historical municipality +35327,#IgnoreRow,True,#ignore,unmapped historical municipality +35328,#IgnoreRow,True,#ignore,unmapped historical municipality +35329,#IgnoreRow,True,#ignore,unmapped historical municipality +35341,observationAbout,wikidataId/Q1203220 +35342,#IgnoreRow,True,#ignore,unmapped historical municipality +35343,observationAbout,wikidataId/Q1205014 +35344,observationAbout,wikidataId/Q1205194 +35345,#IgnoreRow,True,#ignore,unmapped historical municipality +35361,#IgnoreRow,True,#ignore,unmapped historical municipality +35381,#IgnoreRow,True,#ignore,unmapped historical municipality +35401,#IgnoreRow,True,#ignore,unmapped historical municipality +35402,#IgnoreRow,True,#ignore,unmapped historical municipality +35403,#IgnoreRow,True,#ignore,unmapped historical municipality +35421,#IgnoreRow,True,#ignore,unmapped historical municipality +35422,#IgnoreRow,True,#ignore,unmapped historical municipality +35441,#IgnoreRow,True,#ignore,unmapped historical municipality +35442,#IgnoreRow,True,#ignore,unmapped historical municipality +35443,#IgnoreRow,True,#ignore,unmapped historical municipality +35444,#IgnoreRow,True,#ignore,unmapped historical municipality +35461,#IgnoreRow,True,#ignore,unmapped historical municipality +35462,#IgnoreRow,True,#ignore,unmapped historical municipality +35481,#IgnoreRow,True,#ignore,unmapped historical municipality +35482,#IgnoreRow,True,#ignore,unmapped historical municipality +35483,#IgnoreRow,True,#ignore,unmapped historical municipality +35501,#IgnoreRow,True,#ignore,unmapped historical municipality +35502,observationAbout,wikidataId/Q1349028 +35503,#IgnoreRow,True,#ignore,unmapped historical municipality +35504,#IgnoreRow,True,#ignore,unmapped historical municipality +35505,#IgnoreRow,True,#ignore,unmapped historical municipality +35506,#IgnoreRow,True,#ignore,unmapped historical municipality +35507,#IgnoreRow,True,#ignore,unmapped historical municipality +35508,#IgnoreRow,True,#ignore,unmapped historical municipality +36000,observationAbout,wikidataId/Q160734 +36201,observationAbout,wikidataId/Q207309 +36202,observationAbout,wikidataId/Q460812 +36203,observationAbout,wikidataId/Q823493 +36204,observationAbout,wikidataId/Q485583 +36205,observationAbout,wikidataId/Q592500 +36206,observationAbout,wikidataId/Q791587 +36207,observationAbout,wikidataId/Q857828 +36208,observationAbout,wikidataId/Q990371 +36301,observationAbout,wikidataId/Q974969 +36302,observationAbout,wikidataId/Q1205957 +36321,observationAbout,wikidataId/Q1206355 +36341,observationAbout,wikidataId/Q1205943 +36342,observationAbout,wikidataId/Q1350922 +36361,#IgnoreRow,True,#ignore,unmapped historical municipality +36362,#IgnoreRow,True,#ignore,unmapped historical municipality +36363,#IgnoreRow,True,#ignore,unmapped historical municipality +36364,#IgnoreRow,True,#ignore,unmapped historical municipality +36365,#IgnoreRow,True,#ignore,unmapped historical municipality +36366,#IgnoreRow,True,#ignore,unmapped historical municipality +36367,#IgnoreRow,True,#ignore,unmapped historical municipality +36368,observationAbout,wikidataId/Q1205685 +36381,#IgnoreRow,True,#ignore,unmapped historical municipality +36382,#IgnoreRow,True,#ignore,unmapped historical municipality +36383,observationAbout,wikidataId/Q1352217 +36384,#IgnoreRow,True,#ignore,unmapped historical municipality +36385,#IgnoreRow,True,#ignore,unmapped historical municipality +36386,#IgnoreRow,True,#ignore,unmapped historical municipality +36387,observationAbout,wikidataId/Q1351153 +36388,observationAbout,wikidataId/Q1205954 +36401,observationAbout,wikidataId/Q1206010 +36402,observationAbout,wikidataId/Q1351649 +36403,observationAbout,wikidataId/Q1206365 +36404,observationAbout,wikidataId/Q1350904 +36405,observationAbout,wikidataId/Q1350072 +36406,#IgnoreRow,True,#ignore,unmapped historical municipality +36407,#IgnoreRow,True,#ignore,unmapped historical municipality +36421,#IgnoreRow,True,#ignore,unmapped historical municipality +36422,#IgnoreRow,True,#ignore,unmapped historical municipality +36441,#IgnoreRow,True,#ignore,unmapped historical municipality +36442,#IgnoreRow,True,#ignore,unmapped historical municipality +36443,#IgnoreRow,True,#ignore,unmapped historical municipality +36444,#IgnoreRow,True,#ignore,unmapped historical municipality +36461,#IgnoreRow,True,#ignore,unmapped historical municipality +36462,#IgnoreRow,True,#ignore,unmapped historical municipality +36463,#IgnoreRow,True,#ignore,unmapped historical municipality +36464,#IgnoreRow,True,#ignore,unmapped historical municipality +36465,#IgnoreRow,True,#ignore,unmapped historical municipality +36466,#IgnoreRow,True,#ignore,unmapped historical municipality +36467,#IgnoreRow,True,#ignore,unmapped historical municipality +36468,observationAbout,wikidataId/Q1206005 +36481,#IgnoreRow,True,#ignore,unmapped historical municipality +36482,#IgnoreRow,True,#ignore,unmapped historical municipality +36483,#IgnoreRow,True,#ignore,unmapped historical municipality +36484,#IgnoreRow,True,#ignore,unmapped historical municipality +36485,#IgnoreRow,True,#ignore,unmapped historical municipality +36486,#IgnoreRow,True,#ignore,unmapped historical municipality +36487,#IgnoreRow,True,#ignore,unmapped historical municipality +36488,#IgnoreRow,True,#ignore,unmapped historical municipality +36489,observationAbout,wikidataId/Q1206006 +37000,observationAbout,wikidataId/Q161454 +37201,observationAbout,wikidataId/Q200707 +37202,observationAbout,wikidataId/Q460835 +37203,observationAbout,wikidataId/Q855059 +37204,observationAbout,wikidataId/Q191300 +37205,observationAbout,wikidataId/Q861270 +37206,observationAbout,wikidataId/Q857202 +37207,observationAbout,wikidataId/Q861255 +37208,observationAbout,wikidataId/Q873578 +37301,#IgnoreRow,True,#ignore,unmapped historical municipality +37302,#IgnoreRow,True,#ignore,unmapped historical municipality +37303,#IgnoreRow,True,#ignore,unmapped historical municipality +37304,#IgnoreRow,True,#ignore,unmapped historical municipality +37305,#IgnoreRow,True,#ignore,unmapped historical municipality +37306,#IgnoreRow,True,#ignore,unmapped historical municipality +37307,#IgnoreRow,True,#ignore,unmapped historical municipality +37308,#IgnoreRow,True,#ignore,unmapped historical municipality +37321,#IgnoreRow,True,#ignore,unmapped historical municipality +37322,observationAbout,wikidataId/Q1349486 +37323,#IgnoreRow,True,#ignore,unmapped historical municipality +37324,observationAbout,wikidataId/Q1349102 +37341,observationAbout,wikidataId/Q1205980 +37342,#IgnoreRow,True,#ignore,unmapped historical municipality +37343,#IgnoreRow,True,#ignore,unmapped historical municipality +37361,#IgnoreRow,True,#ignore,unmapped historical municipality +37362,#IgnoreRow,True,#ignore,unmapped historical municipality +37363,#IgnoreRow,True,#ignore,unmapped historical municipality +37364,observationAbout,wikidataId/Q1350060 +37381,#IgnoreRow,True,#ignore,unmapped historical municipality +37382,#IgnoreRow,True,#ignore,unmapped historical municipality +37383,#IgnoreRow,True,#ignore,unmapped historical municipality +37384,#IgnoreRow,True,#ignore,unmapped historical municipality +37385,#IgnoreRow,True,#ignore,unmapped historical municipality +37386,observationAbout,wikidataId/Q1156290 +37387,observationAbout,wikidataId/Q1156043 +37401,#IgnoreRow,True,#ignore,unmapped historical municipality +37402,#IgnoreRow,True,#ignore,unmapped historical municipality +37403,observationAbout,wikidataId/Q1271483 +37404,observationAbout,wikidataId/Q1357346 +37405,#IgnoreRow,True,#ignore,unmapped historical municipality +37406,observationAbout,wikidataId/Q1350204 +37421,#IgnoreRow,True,#ignore,unmapped historical municipality +37422,#IgnoreRow,True,#ignore,unmapped historical municipality +37423,#IgnoreRow,True,#ignore,unmapped historical municipality +37424,#IgnoreRow,True,#ignore,unmapped historical municipality +37425,#IgnoreRow,True,#ignore,unmapped historical municipality +37426,#IgnoreRow,True,#ignore,unmapped historical municipality +37427,#IgnoreRow,True,#ignore,unmapped historical municipality +37428,#IgnoreRow,True,#ignore,unmapped historical municipality +37429,#IgnoreRow,True,#ignore,unmapped historical municipality +38000,observationAbout,wikidataId/Q123376 +38201,observationAbout,wikidataId/Q200022 +38202,observationAbout,wikidataId/Q461020 +38203,observationAbout,wikidataId/Q845857 +38204,observationAbout,wikidataId/Q200603 +38205,observationAbout,wikidataId/Q386628 +38206,observationAbout,wikidataId/Q633945 +38207,observationAbout,wikidataId/Q386655 +38208,#IgnoreRow,True,#ignore,unmapped historical municipality +38209,#IgnoreRow,True,#ignore,unmapped historical municipality +38210,observationAbout,wikidataId/Q838678 +38211,#IgnoreRow,True,#ignore,unmapped historical municipality +38212,#IgnoreRow,True,#ignore,unmapped historical municipality +38213,observationAbout,wikidataId/Q838666 +38214,observationAbout,wikidataId/Q819620 +38215,observationAbout,wikidataId/Q132412 +38301,#IgnoreRow,True,#ignore,unmapped historical municipality +38302,#IgnoreRow,True,#ignore,unmapped historical municipality +38303,#IgnoreRow,True,#ignore,unmapped historical municipality +38321,#IgnoreRow,True,#ignore,unmapped historical municipality +38323,#IgnoreRow,True,#ignore,unmapped historical municipality +38341,#IgnoreRow,True,#ignore,unmapped historical municipality +38342,#IgnoreRow,True,#ignore,unmapped historical municipality +38343,#IgnoreRow,True,#ignore,unmapped historical municipality +38344,#IgnoreRow,True,#ignore,unmapped historical municipality +38345,#IgnoreRow,True,#ignore,unmapped historical municipality +38346,#IgnoreRow,True,#ignore,unmapped historical municipality +38347,#IgnoreRow,True,#ignore,unmapped historical municipality +38348,#IgnoreRow,True,#ignore,unmapped historical municipality +38349,#IgnoreRow,True,#ignore,unmapped historical municipality +38350,#IgnoreRow,True,#ignore,unmapped historical municipality +38351,#IgnoreRow,True,#ignore,unmapped historical municipality +38352,#IgnoreRow,True,#ignore,unmapped historical municipality +38353,#IgnoreRow,True,#ignore,unmapped historical municipality +38354,#IgnoreRow,True,#ignore,unmapped historical municipality +38355,#IgnoreRow,True,#ignore,unmapped historical municipality +38356,observationAbout,wikidataId/Q1350804 +38361,#IgnoreRow,True,#ignore,unmapped historical municipality +38362,#IgnoreRow,True,#ignore,unmapped historical municipality +38363,#IgnoreRow,True,#ignore,unmapped historical municipality +38381,#IgnoreRow,True,#ignore,unmapped historical municipality +38382,#IgnoreRow,True,#ignore,unmapped historical municipality +38383,#IgnoreRow,True,#ignore,unmapped historical municipality +38384,#IgnoreRow,True,#ignore,unmapped historical municipality +38385,#IgnoreRow,True,#ignore,unmapped historical municipality +38386,observationAbout,wikidataId/Q1351134 +38401,observationAbout,wikidataId/Q1350953 +38402,observationAbout,wikidataId/Q1350249 +38403,#IgnoreRow,True,#ignore,unmapped historical municipality +38404,#IgnoreRow,True,#ignore,unmapped historical municipality +38405,#IgnoreRow,True,#ignore,unmapped historical municipality +38421,#IgnoreRow,True,#ignore,unmapped historical municipality +38422,observationAbout,wikidataId/Q1018717 +38423,#IgnoreRow,True,#ignore,unmapped historical municipality +38424,#IgnoreRow,True,#ignore,unmapped historical municipality +38425,#IgnoreRow,True,#ignore,unmapped historical municipality +38441,#IgnoreRow,True,#ignore,unmapped historical municipality +38442,observationAbout,wikidataId/Q1206399 +38443,#IgnoreRow,True,#ignore,unmapped historical municipality +38444,#IgnoreRow,True,#ignore,unmapped historical municipality +38445,#IgnoreRow,True,#ignore,unmapped historical municipality +38461,#IgnoreRow,True,#ignore,unmapped historical municipality +38462,#IgnoreRow,True,#ignore,unmapped historical municipality +38463,#IgnoreRow,True,#ignore,unmapped historical municipality +38464,#IgnoreRow,True,#ignore,unmapped historical municipality +38481,#IgnoreRow,True,#ignore,unmapped historical municipality +38482,#IgnoreRow,True,#ignore,unmapped historical municipality +38483,#IgnoreRow,True,#ignore,unmapped historical municipality +38484,observationAbout,wikidataId/Q1350180 +38485,#IgnoreRow,True,#ignore,unmapped historical municipality +38486,#IgnoreRow,True,#ignore,unmapped historical municipality +38488,observationAbout,wikidataId/Q1351015 +38501,#IgnoreRow,True,#ignore,unmapped historical municipality +38502,#IgnoreRow,True,#ignore,unmapped historical municipality +38503,#IgnoreRow,True,#ignore,unmapped historical municipality +38504,#IgnoreRow,True,#ignore,unmapped historical municipality +38505,#IgnoreRow,True,#ignore,unmapped historical municipality +38506,observationAbout,wikidataId/Q980346 +39000,observationAbout,wikidataId/Q134093 +39201,observationAbout,wikidataId/Q200251 +39202,#IgnoreRow,True,#ignore,unmapped historical municipality +39203,observationAbout,wikidataId/Q417825 +39204,observationAbout,wikidataId/Q861621 +39205,observationAbout,wikidataId/Q851163 +39206,observationAbout,wikidataId/Q751154 +39207,#IgnoreRow,True,#ignore,unmapped historical municipality +39208,observationAbout,wikidataId/Q655258 +39209,observationAbout,wikidataId/Q573115 +39210,observationAbout,wikidataId/Q166946 +39211,observationAbout,wikidataId/Q877209 +39212,observationAbout,wikidataId/Q877689 +39301,observationAbout,wikidataId/Q1205386 +39302,observationAbout,wikidataId/Q1205759 +39303,observationAbout,wikidataId/Q1205916 +39304,observationAbout,wikidataId/Q1156177 +39305,observationAbout,wikidataId/Q1350216 +39306,observationAbout,wikidataId/Q1206380 +39307,observationAbout,wikidataId/Q1206376 +39321,#IgnoreRow,True,#ignore,unmapped historical municipality +39322,#IgnoreRow,True,#ignore,unmapped historical municipality +39323,#IgnoreRow,True,#ignore,unmapped historical municipality +39324,#IgnoreRow,True,#ignore,unmapped historical municipality +39325,#IgnoreRow,True,#ignore,unmapped historical municipality +39326,#IgnoreRow,True,#ignore,unmapped historical municipality +39327,#IgnoreRow,True,#ignore,unmapped historical municipality +39328,#IgnoreRow,True,#ignore,unmapped historical municipality +39341,observationAbout,wikidataId/Q735494 +39344,observationAbout,wikidataId/Q1206441 +39361,#IgnoreRow,True,#ignore,unmapped historical municipality +39362,#IgnoreRow,True,#ignore,unmapped historical municipality +39363,observationAbout,wikidataId/Q1156499 +39364,observationAbout,wikidataId/Q1206551 +39365,#IgnoreRow,True,#ignore,unmapped historical municipality +39381,#IgnoreRow,True,#ignore,unmapped historical municipality +39382,#IgnoreRow,True,#ignore,unmapped historical municipality +39383,#IgnoreRow,True,#ignore,unmapped historical municipality +39384,#IgnoreRow,True,#ignore,unmapped historical municipality +39385,#IgnoreRow,True,#ignore,unmapped historical municipality +39386,observationAbout,wikidataId/Q1206590 +39387,observationAbout,wikidataId/Q1206371 +39401,observationAbout,wikidataId/Q1350193 +39402,observationAbout,wikidataId/Q1205751 +39403,observationAbout,wikidataId/Q1205739 +39404,#IgnoreRow,True,#ignore,unmapped historical municipality +39405,observationAbout,wikidataId/Q1206286 +39406,#IgnoreRow,True,#ignore,unmapped historical municipality +39407,#IgnoreRow,True,#ignore,unmapped historical municipality +39408,#IgnoreRow,True,#ignore,unmapped historical municipality +39409,#IgnoreRow,True,#ignore,unmapped historical municipality +39410,observationAbout,wikidataId/Q1156151 +39411,observationAbout,wikidataId/Q1156308 +39412,observationAbout,wikidataId/Q1156035 +39421,#IgnoreRow,True,#ignore,unmapped historical municipality +39422,#IgnoreRow,True,#ignore,unmapped historical municipality +39423,#IgnoreRow,True,#ignore,unmapped historical municipality +39424,observationAbout,wikidataId/Q1206036 +39425,#IgnoreRow,True,#ignore,unmapped historical municipality +39426,#IgnoreRow,True,#ignore,unmapped historical municipality +39427,observationAbout,wikidataId/Q1206074 +39428,observationAbout,wikidataId/Q1351626 +40000,observationAbout,wikidataId/Q123258 +40100,observationAbout,wikidataId/Q188806 +40101,observationAbout,wikidataId/Q1045113 +40103,observationAbout,wikidataId/Q858184 +40105,observationAbout,wikidataId/Q858143 +40106,observationAbout,wikidataId/Q1042291 +40107,observationAbout,wikidataId/Q858201 +40108,observationAbout,wikidataId/Q857895 +40109,observationAbout,wikidataId/Q1044377 +40130,observationAbout,wikidataId/Q26600 +40131,observationAbout,wikidataId/Q1206659 +40132,observationAbout,wikidataId/Q1330839 +40133,observationAbout,wikidataId/Q1156273 +40134,observationAbout,wikidataId/Q1132419 +40135,observationAbout,wikidataId/Q1132402 +40136,observationAbout,wikidataId/Q948351 +40137,observationAbout,wikidataId/Q1071779 +40202,observationAbout,wikidataId/Q385623 +40203,observationAbout,wikidataId/Q320934 +40204,observationAbout,wikidataId/Q861296 +40205,observationAbout,wikidataId/Q328605 +40206,observationAbout,wikidataId/Q850332 +40207,observationAbout,wikidataId/Q585630 +40208,#IgnoreRow,True,#ignore,unmapped historical municipality +40209,#IgnoreRow,True,#ignore,unmapped historical municipality +40210,observationAbout,wikidataId/Q850492 +40211,observationAbout,wikidataId/Q601137 +40212,observationAbout,wikidataId/Q385248 +40213,observationAbout,wikidataId/Q851668 +40214,observationAbout,wikidataId/Q633601 +40215,observationAbout,wikidataId/Q847601 +40216,observationAbout,wikidataId/Q683248 +40217,observationAbout,wikidataId/Q844140 +40218,observationAbout,wikidataId/Q47084 +40219,observationAbout,wikidataId/Q385762 +40220,observationAbout,wikidataId/Q819603 +40221,observationAbout,wikidataId/Q728218 +40222,#IgnoreRow,True,#ignore,unmapped historical municipality +40223,observationAbout,wikidataId/Q768356 +40224,observationAbout,wikidataId/Q825700 +40225,observationAbout,wikidataId/Q850373 +40226,observationAbout,wikidataId/Q990431 +40227,observationAbout,wikidataId/Q376901 +40228,observationAbout,wikidataId/Q720625 +40229,observationAbout,wikidataId/Q990465 +40230,observationAbout,wikidataId/Q1011145 +40231,observationAbout,wikidataId/Q1206001 +40302,#IgnoreRow,True,#ignore,unmapped historical municipality +40305,#IgnoreRow,True,#ignore,unmapped historical municipality +40341,observationAbout,wikidataId/Q945518 +40342,observationAbout,wikidataId/Q1206130 +40343,observationAbout,wikidataId/Q1206887 +40344,observationAbout,wikidataId/Q1205588 +40345,observationAbout,wikidataId/Q179185 +40347,#IgnoreRow,True,#ignore,unmapped historical municipality +40348,observationAbout,wikidataId/Q1206089 +40349,observationAbout,wikidataId/Q1206141 +40361,#IgnoreRow,True,#ignore,unmapped historical municipality +40362,#IgnoreRow,True,#ignore,unmapped historical municipality +40363,#IgnoreRow,True,#ignore,unmapped historical municipality +40364,#IgnoreRow,True,#ignore,unmapped historical municipality +40365,#IgnoreRow,True,#ignore,unmapped historical municipality +40381,observationAbout,wikidataId/Q725518 +40382,observationAbout,wikidataId/Q1350098 +40383,observationAbout,wikidataId/Q1206102 +40384,observationAbout,wikidataId/Q1205554 +40401,observationAbout,wikidataId/Q1206635 +40402,observationAbout,wikidataId/Q629799 +40403,#IgnoreRow,True,#ignore,unmapped historical municipality +40404,#IgnoreRow,True,#ignore,unmapped historical municipality +40421,observationAbout,wikidataId/Q1206263 +40422,#IgnoreRow,True,#ignore,unmapped historical municipality +40423,#IgnoreRow,True,#ignore,unmapped historical municipality +40424,#IgnoreRow,True,#ignore,unmapped historical municipality +40425,#IgnoreRow,True,#ignore,unmapped historical municipality +40426,#IgnoreRow,True,#ignore,unmapped historical municipality +40427,#IgnoreRow,True,#ignore,unmapped historical municipality +40428,#IgnoreRow,True,#ignore,unmapped historical municipality +40441,#IgnoreRow,True,#ignore,unmapped historical municipality +40442,#IgnoreRow,True,#ignore,unmapped historical municipality +40443,#IgnoreRow,True,#ignore,unmapped historical municipality +40444,#IgnoreRow,True,#ignore,unmapped historical municipality +40445,#IgnoreRow,True,#ignore,unmapped historical municipality +40446,#IgnoreRow,True,#ignore,unmapped historical municipality +40447,observationAbout,wikidataId/Q1072679 +40448,observationAbout,wikidataId/Q1206282 +40461,#IgnoreRow,True,#ignore,unmapped historical municipality +40462,#IgnoreRow,True,#ignore,unmapped historical municipality +40463,#IgnoreRow,True,#ignore,unmapped historical municipality +40481,#IgnoreRow,True,#ignore,unmapped historical municipality +40482,#IgnoreRow,True,#ignore,unmapped historical municipality +40483,#IgnoreRow,True,#ignore,unmapped historical municipality +40501,#IgnoreRow,True,#ignore,unmapped historical municipality +40503,observationAbout,wikidataId/Q1205864 +40521,#IgnoreRow,True,#ignore,unmapped historical municipality +40522,observationAbout,wikidataId/Q385252 +40523,#IgnoreRow,True,#ignore,unmapped historical municipality +40541,#IgnoreRow,True,#ignore,unmapped historical municipality +40542,#IgnoreRow,True,#ignore,unmapped historical municipality +40543,#IgnoreRow,True,#ignore,unmapped historical municipality +40544,observationAbout,wikidataId/Q1206740 +40545,#IgnoreRow,True,#ignore,unmapped historical municipality +40546,#IgnoreRow,True,#ignore,unmapped historical municipality +40561,#IgnoreRow,True,#ignore,unmapped historical municipality +40562,#IgnoreRow,True,#ignore,unmapped historical municipality +40563,#IgnoreRow,True,#ignore,unmapped historical municipality +40564,#IgnoreRow,True,#ignore,unmapped historical municipality +40581,#IgnoreRow,True,#ignore,unmapped historical municipality +40601,observationAbout,wikidataId/Q573632 +40602,observationAbout,wikidataId/Q1205805 +40603,#IgnoreRow,True,#ignore,unmapped historical municipality +40604,observationAbout,wikidataId/Q1208544 +40605,observationAbout,wikidataId/Q1206042 +40606,#IgnoreRow,True,#ignore,unmapped historical municipality +40607,#IgnoreRow,True,#ignore,unmapped historical municipality +40608,observationAbout,wikidataId/Q386472 +40609,observationAbout,wikidataId/Q119837 +40610,observationAbout,wikidataId/Q1205781 +40621,observationAbout,wikidataId/Q1206876 +40622,#IgnoreRow,True,#ignore,unmapped historical municipality +40623,#IgnoreRow,True,#ignore,unmapped historical municipality +40624,#IgnoreRow,True,#ignore,unmapped historical municipality +40625,observationAbout,wikidataId/Q1206868 +40641,#IgnoreRow,True,#ignore,unmapped historical municipality +40642,observationAbout,wikidataId/Q1205793 +40643,#IgnoreRow,True,#ignore,unmapped historical municipality +40644,#IgnoreRow,True,#ignore,unmapped historical municipality +40645,#IgnoreRow,True,#ignore,unmapped historical municipality +40646,observationAbout,wikidataId/Q1206598 +40647,observationAbout,wikidataId/Q1072651 +41000,observationAbout,wikidataId/Q160420 +41201,observationAbout,wikidataId/Q209766 +41202,observationAbout,wikidataId/Q738275 +41203,observationAbout,wikidataId/Q860674 +41204,observationAbout,wikidataId/Q877177 +41205,observationAbout,wikidataId/Q857266 +41206,observationAbout,wikidataId/Q877220 +41207,observationAbout,wikidataId/Q684494 +41208,observationAbout,wikidataId/Q877227 +41209,observationAbout,wikidataId/Q634130 +41210,observationAbout,wikidataId/Q990392 +41301,#IgnoreRow,True,#ignore,unmapped historical municipality +41302,#IgnoreRow,True,#ignore,unmapped historical municipality +41303,#IgnoreRow,True,#ignore,unmapped historical municipality +41304,#IgnoreRow,True,#ignore,unmapped historical municipality +41305,#IgnoreRow,True,#ignore,unmapped historical municipality +41306,#IgnoreRow,True,#ignore,unmapped historical municipality +41321,#IgnoreRow,True,#ignore,unmapped historical municipality +41322,#IgnoreRow,True,#ignore,unmapped historical municipality +41323,#IgnoreRow,True,#ignore,unmapped historical municipality +41324,#IgnoreRow,True,#ignore,unmapped historical municipality +41325,#IgnoreRow,True,#ignore,unmapped historical municipality +41326,#IgnoreRow,True,#ignore,unmapped historical municipality +41327,observationAbout,wikidataId/Q1206915 +41341,observationAbout,wikidataId/Q1206602 +41342,#IgnoreRow,True,#ignore,unmapped historical municipality +41343,#IgnoreRow,True,#ignore,unmapped historical municipality +41344,#IgnoreRow,True,#ignore,unmapped historical municipality +41345,observationAbout,wikidataId/Q1367170 +41346,observationAbout,wikidataId/Q1351707 +41361,#IgnoreRow,True,#ignore,unmapped historical municipality +41362,#IgnoreRow,True,#ignore,unmapped historical municipality +41363,#IgnoreRow,True,#ignore,unmapped historical municipality +41364,#IgnoreRow,True,#ignore,unmapped historical municipality +41381,#IgnoreRow,True,#ignore,unmapped historical municipality +41382,#IgnoreRow,True,#ignore,unmapped historical municipality +41383,#IgnoreRow,True,#ignore,unmapped historical municipality +41384,#IgnoreRow,True,#ignore,unmapped historical municipality +41385,#IgnoreRow,True,#ignore,unmapped historical municipality +41386,#IgnoreRow,True,#ignore,unmapped historical municipality +41387,observationAbout,wikidataId/Q1206926 +41388,#IgnoreRow,True,#ignore,unmapped historical municipality +41389,#IgnoreRow,True,#ignore,unmapped historical municipality +41401,observationAbout,wikidataId/Q668427 +41402,#IgnoreRow,True,#ignore,unmapped historical municipality +41421,#IgnoreRow,True,#ignore,unmapped historical municipality +41422,#IgnoreRow,True,#ignore,unmapped historical municipality +41423,observationAbout,wikidataId/Q1206025 +41424,observationAbout,wikidataId/Q1206934 +41425,observationAbout,wikidataId/Q1206958 +41426,#IgnoreRow,True,#ignore,unmapped historical municipality +41427,#IgnoreRow,True,#ignore,unmapped historical municipality +41441,observationAbout,wikidataId/Q1206942 +41442,#IgnoreRow,True,#ignore,unmapped historical municipality +41443,#IgnoreRow,True,#ignore,unmapped historical municipality +42000,observationAbout,wikidataId/Q169376 +42201,observationAbout,wikidataId/Q38234 +42202,observationAbout,wikidataId/Q328000 +42203,observationAbout,wikidataId/Q847594 +42204,observationAbout,wikidataId/Q596469 +42205,observationAbout,wikidataId/Q385591 +42206,#IgnoreRow,True,#ignore,unmapped historical municipality +42207,observationAbout,wikidataId/Q328601 +42208,observationAbout,wikidataId/Q850465 +42209,observationAbout,wikidataId/Q991032 +42210,observationAbout,wikidataId/Q861519 +42211,observationAbout,wikidataId/Q201659 +42212,observationAbout,wikidataId/Q520267 +42213,observationAbout,wikidataId/Q999453 +42214,observationAbout,wikidataId/Q990376 +42301,#IgnoreRow,True,#ignore,unmapped historical municipality +42302,#IgnoreRow,True,#ignore,unmapped historical municipality +42303,#IgnoreRow,True,#ignore,unmapped historical municipality +42304,#IgnoreRow,True,#ignore,unmapped historical municipality +42305,#IgnoreRow,True,#ignore,unmapped historical municipality +42306,#IgnoreRow,True,#ignore,unmapped historical municipality +42307,observationAbout,wikidataId/Q683858 +42308,observationAbout,wikidataId/Q1205851 +42309,#IgnoreRow,True,#ignore,unmapped historical municipality +42310,#IgnoreRow,True,#ignore,unmapped historical municipality +42311,#IgnoreRow,True,#ignore,unmapped historical municipality +42312,#IgnoreRow,True,#ignore,unmapped historical municipality +42313,#IgnoreRow,True,#ignore,unmapped historical municipality +42314,#IgnoreRow,True,#ignore,unmapped historical municipality +42315,#IgnoreRow,True,#ignore,unmapped historical municipality +42321,observationAbout,wikidataId/Q942035 +42322,observationAbout,wikidataId/Q784698 +42323,observationAbout,wikidataId/Q1351244 +42341,#IgnoreRow,True,#ignore,unmapped historical municipality +42342,#IgnoreRow,True,#ignore,unmapped historical municipality +42343,#IgnoreRow,True,#ignore,unmapped historical municipality +42344,#IgnoreRow,True,#ignore,unmapped historical municipality +42361,#IgnoreRow,True,#ignore,unmapped historical municipality +42362,#IgnoreRow,True,#ignore,unmapped historical municipality +42363,#IgnoreRow,True,#ignore,unmapped historical municipality +42364,#IgnoreRow,True,#ignore,unmapped historical municipality +42365,#IgnoreRow,True,#ignore,unmapped historical municipality +42366,#IgnoreRow,True,#ignore,unmapped historical municipality +42367,#IgnoreRow,True,#ignore,unmapped historical municipality +42368,#IgnoreRow,True,#ignore,unmapped historical municipality +42369,#IgnoreRow,True,#ignore,unmapped historical municipality +42370,#IgnoreRow,True,#ignore,unmapped historical municipality +42371,#IgnoreRow,True,#ignore,unmapped historical municipality +42372,#IgnoreRow,True,#ignore,unmapped historical municipality +42373,#IgnoreRow,True,#ignore,unmapped historical municipality +42374,#IgnoreRow,True,#ignore,unmapped historical municipality +42375,#IgnoreRow,True,#ignore,unmapped historical municipality +42376,#IgnoreRow,True,#ignore,unmapped historical municipality +42381,#IgnoreRow,True,#ignore,unmapped historical municipality +42382,#IgnoreRow,True,#ignore,unmapped historical municipality +42383,observationAbout,wikidataId/Q633422 +42384,#IgnoreRow,True,#ignore,unmapped historical municipality +42385,#IgnoreRow,True,#ignore,unmapped historical municipality +42386,#IgnoreRow,True,#ignore,unmapped historical municipality +42387,#IgnoreRow,True,#ignore,unmapped historical municipality +42388,#IgnoreRow,True,#ignore,unmapped historical municipality +42389,#IgnoreRow,True,#ignore,unmapped historical municipality +42390,#IgnoreRow,True,#ignore,unmapped historical municipality +42391,observationAbout,wikidataId/Q1350858 +42392,#IgnoreRow,True,#ignore,unmapped historical municipality +42393,#IgnoreRow,True,#ignore,unmapped historical municipality +42401,#IgnoreRow,True,#ignore,unmapped historical municipality +42402,#IgnoreRow,True,#ignore,unmapped historical municipality +42403,#IgnoreRow,True,#ignore,unmapped historical municipality +42404,#IgnoreRow,True,#ignore,unmapped historical municipality +42405,#IgnoreRow,True,#ignore,unmapped historical municipality +42406,#IgnoreRow,True,#ignore,unmapped historical municipality +42407,#IgnoreRow,True,#ignore,unmapped historical municipality +42408,#IgnoreRow,True,#ignore,unmapped historical municipality +42409,#IgnoreRow,True,#ignore,unmapped historical municipality +42410,#IgnoreRow,True,#ignore,unmapped historical municipality +42411,observationAbout,wikidataId/Q1206790 +42421,#IgnoreRow,True,#ignore,unmapped historical municipality +42422,#IgnoreRow,True,#ignore,unmapped historical municipality +42423,#IgnoreRow,True,#ignore,unmapped historical municipality +42424,#IgnoreRow,True,#ignore,unmapped historical municipality +42441,#IgnoreRow,True,#ignore,unmapped historical municipality +42442,#IgnoreRow,True,#ignore,unmapped historical municipality +42443,#IgnoreRow,True,#ignore,unmapped historical municipality +42444,#IgnoreRow,True,#ignore,unmapped historical municipality +42445,#IgnoreRow,True,#ignore,unmapped historical municipality +42446,#IgnoreRow,True,#ignore,unmapped historical municipality +43000,observationAbout,wikidataId/Q130308 +43100,observationAbout,wikidataId/Q199889 +43101,observationAbout,wikidataId/Q137716 +43102,observationAbout,wikidataId/Q875464 +43103,observationAbout,wikidataId/Q877003 +43104,observationAbout,wikidataId/Q876960 +43105,observationAbout,wikidataId/Q876926 +43201,#IgnoreRow,True,#ignore,unmapped historical municipality +43202,observationAbout,wikidataId/Q734479 +43203,observationAbout,wikidataId/Q861292 +43204,observationAbout,wikidataId/Q626118 +43205,observationAbout,wikidataId/Q386110 +43206,observationAbout,wikidataId/Q861610 +43207,#IgnoreRow,True,#ignore,unmapped historical municipality +43208,observationAbout,wikidataId/Q861214 +43209,#IgnoreRow,True,#ignore,unmapped historical municipality +43210,observationAbout,wikidataId/Q911709 +43211,observationAbout,wikidataId/Q861262 +43212,observationAbout,wikidataId/Q877232 +43213,observationAbout,wikidataId/Q782825 +43214,observationAbout,wikidataId/Q733681 +43215,observationAbout,wikidataId/Q452634 +43216,observationAbout,wikidataId/Q873558 +43301,#IgnoreRow,True,#ignore,unmapped historical municipality +43302,#IgnoreRow,True,#ignore,unmapped historical municipality +43303,#IgnoreRow,True,#ignore,unmapped historical municipality +43304,#IgnoreRow,True,#ignore,unmapped historical municipality +43321,#IgnoreRow,True,#ignore,unmapped historical municipality +43322,#IgnoreRow,True,#ignore,unmapped historical municipality +43341,#IgnoreRow,True,#ignore,unmapped historical municipality +43342,#IgnoreRow,True,#ignore,unmapped historical municipality +43343,#IgnoreRow,True,#ignore,unmapped historical municipality +43344,#IgnoreRow,True,#ignore,unmapped historical municipality +43345,#IgnoreRow,True,#ignore,unmapped historical municipality +43346,#IgnoreRow,True,#ignore,unmapped historical municipality +43347,#IgnoreRow,True,#ignore,unmapped historical municipality +43348,observationAbout,wikidataId/Q674749 +43361,#IgnoreRow,True,#ignore,unmapped historical municipality +43362,#IgnoreRow,True,#ignore,unmapped historical municipality +43363,#IgnoreRow,True,#ignore,unmapped historical municipality +43364,observationAbout,wikidataId/Q1156401 +43365,#IgnoreRow,True,#ignore,unmapped historical municipality +43366,#IgnoreRow,True,#ignore,unmapped historical municipality +43367,observationAbout,wikidataId/Q1156398 +43368,observationAbout,wikidataId/Q1156411 +43369,observationAbout,wikidataId/Q1156415 +43381,#IgnoreRow,True,#ignore,unmapped historical municipality +43382,#IgnoreRow,True,#ignore,unmapped historical municipality +43383,#IgnoreRow,True,#ignore,unmapped historical municipality +43384,#IgnoreRow,True,#ignore,unmapped historical municipality +43385,#IgnoreRow,True,#ignore,unmapped historical municipality +43401,#IgnoreRow,True,#ignore,unmapped historical municipality +43402,#IgnoreRow,True,#ignore,unmapped historical municipality +43403,observationAbout,wikidataId/Q1156406 +43404,observationAbout,wikidataId/Q1156423 +43405,#IgnoreRow,True,#ignore,unmapped historical municipality +43406,#IgnoreRow,True,#ignore,unmapped historical municipality +43407,#IgnoreRow,True,#ignore,unmapped historical municipality +43421,#IgnoreRow,True,#ignore,unmapped historical municipality +43422,#IgnoreRow,True,#ignore,unmapped historical municipality +43423,observationAbout,wikidataId/Q1184473 +43424,observationAbout,wikidataId/Q649948 +43425,observationAbout,wikidataId/Q1184667 +43426,#IgnoreRow,True,#ignore,unmapped historical municipality +43427,#IgnoreRow,True,#ignore,unmapped historical municipality +43428,observationAbout,wikidataId/Q1184676 +43429,#IgnoreRow,True,#ignore,unmapped historical municipality +43430,#IgnoreRow,True,#ignore,unmapped historical municipality +43431,#IgnoreRow,True,#ignore,unmapped historical municipality +43432,observationAbout,wikidataId/Q961555 +43433,observationAbout,wikidataId/Q1184681 +43441,observationAbout,wikidataId/Q1208526 +43442,observationAbout,wikidataId/Q1184801 +43443,observationAbout,wikidataId/Q1206076 +43444,observationAbout,wikidataId/Q1184749 +43445,#IgnoreRow,True,#ignore,unmapped historical municipality +43446,#IgnoreRow,True,#ignore,unmapped historical municipality +43447,observationAbout,wikidataId/Q1184757 +43461,#IgnoreRow,True,#ignore,unmapped historical municipality +43462,#IgnoreRow,True,#ignore,unmapped historical municipality +43463,#IgnoreRow,True,#ignore,unmapped historical municipality +43464,#IgnoreRow,True,#ignore,unmapped historical municipality +43465,#IgnoreRow,True,#ignore,unmapped historical municipality +43466,#IgnoreRow,True,#ignore,unmapped historical municipality +43467,#IgnoreRow,True,#ignore,unmapped historical municipality +43468,observationAbout,wikidataId/Q1184800 +43481,#IgnoreRow,True,#ignore,unmapped historical municipality +43482,observationAbout,wikidataId/Q1184620 +43484,observationAbout,wikidataId/Q1184734 +43501,observationAbout,wikidataId/Q1184741 +43502,#IgnoreRow,True,#ignore,unmapped historical municipality +43503,#IgnoreRow,True,#ignore,unmapped historical municipality +43504,#IgnoreRow,True,#ignore,unmapped historical municipality +43505,observationAbout,wikidataId/Q1184655 +43506,observationAbout,wikidataId/Q1184768 +43507,observationAbout,wikidataId/Q1184789 +43508,#IgnoreRow,True,#ignore,unmapped historical municipality +43509,#IgnoreRow,True,#ignore,unmapped historical municipality +43510,observationAbout,wikidataId/Q1184723 +43511,observationAbout,wikidataId/Q1184716 +43512,observationAbout,wikidataId/Q1184728 +43513,observationAbout,wikidataId/Q1184708 +43514,observationAbout,wikidataId/Q1184763 +43521,#IgnoreRow,True,#ignore,unmapped historical municipality +43522,#IgnoreRow,True,#ignore,unmapped historical municipality +43523,#IgnoreRow,True,#ignore,unmapped historical municipality +43524,#IgnoreRow,True,#ignore,unmapped historical municipality +43525,#IgnoreRow,True,#ignore,unmapped historical municipality +43526,#IgnoreRow,True,#ignore,unmapped historical municipality +43527,#IgnoreRow,True,#ignore,unmapped historical municipality +43528,#IgnoreRow,True,#ignore,unmapped historical municipality +43529,#IgnoreRow,True,#ignore,unmapped historical municipality +43530,#IgnoreRow,True,#ignore,unmapped historical municipality +43531,observationAbout,wikidataId/Q1184688 +43532,#IgnoreRow,True,#ignore,unmapped historical municipality +43533,#IgnoreRow,True,#ignore,unmapped historical municipality +44000,observationAbout,wikidataId/Q133924 +44201,observationAbout,wikidataId/Q199609 +44202,observationAbout,wikidataId/Q273880 +44203,observationAbout,wikidataId/Q861221 +44204,observationAbout,wikidataId/Q386968 +44205,observationAbout,wikidataId/Q847546 +44206,observationAbout,wikidataId/Q845792 +44207,observationAbout,wikidataId/Q840580 +44208,observationAbout,wikidataId/Q850508 +44209,observationAbout,wikidataId/Q857281 +44210,observationAbout,wikidataId/Q850359 +44211,observationAbout,wikidataId/Q114147 +44212,observationAbout,wikidataId/Q850366 +44213,observationAbout,wikidataId/Q990455 +44214,observationAbout,wikidataId/Q873572 +44301,#IgnoreRow,True,#ignore,unmapped historical municipality +44302,#IgnoreRow,True,#ignore,unmapped historical municipality +44303,#IgnoreRow,True,#ignore,unmapped historical municipality +44321,#IgnoreRow,True,#ignore,unmapped historical municipality +44322,observationAbout,wikidataId/Q1351003 +44323,#IgnoreRow,True,#ignore,unmapped historical municipality +44324,#IgnoreRow,True,#ignore,unmapped historical municipality +44325,#IgnoreRow,True,#ignore,unmapped historical municipality +44341,observationAbout,wikidataId/Q1184795 +44342,#IgnoreRow,True,#ignore,unmapped historical municipality +44361,#IgnoreRow,True,#ignore,unmapped historical municipality +44362,#IgnoreRow,True,#ignore,unmapped historical municipality +44363,#IgnoreRow,True,#ignore,unmapped historical municipality +44364,#IgnoreRow,True,#ignore,unmapped historical municipality +44381,#IgnoreRow,True,#ignore,unmapped historical municipality +44401,#IgnoreRow,True,#ignore,unmapped historical municipality +44402,#IgnoreRow,True,#ignore,unmapped historical municipality +44403,#IgnoreRow,True,#ignore,unmapped historical municipality +44404,#IgnoreRow,True,#ignore,unmapped historical municipality +44405,#IgnoreRow,True,#ignore,unmapped historical municipality +44406,#IgnoreRow,True,#ignore,unmapped historical municipality +44407,#IgnoreRow,True,#ignore,unmapped historical municipality +44408,#IgnoreRow,True,#ignore,unmapped historical municipality +44421,#IgnoreRow,True,#ignore,unmapped historical municipality +44422,#IgnoreRow,True,#ignore,unmapped historical municipality +44423,#IgnoreRow,True,#ignore,unmapped historical municipality +44424,#IgnoreRow,True,#ignore,unmapped historical municipality +44425,#IgnoreRow,True,#ignore,unmapped historical municipality +44426,#IgnoreRow,True,#ignore,unmapped historical municipality +44427,#IgnoreRow,True,#ignore,unmapped historical municipality +44428,#IgnoreRow,True,#ignore,unmapped historical municipality +44441,#IgnoreRow,True,#ignore,unmapped historical municipality +44442,#IgnoreRow,True,#ignore,unmapped historical municipality +44443,#IgnoreRow,True,#ignore,unmapped historical municipality +44461,observationAbout,wikidataId/Q1184672 +44462,observationAbout,wikidataId/Q1184816 +44481,#IgnoreRow,True,#ignore,unmapped historical municipality +44482,#IgnoreRow,True,#ignore,unmapped historical municipality +44483,#IgnoreRow,True,#ignore,unmapped historical municipality +44484,#IgnoreRow,True,#ignore,unmapped historical municipality +44485,#IgnoreRow,True,#ignore,unmapped historical municipality +44501,#IgnoreRow,True,#ignore,unmapped historical municipality +44502,#IgnoreRow,True,#ignore,unmapped historical municipality +44503,#IgnoreRow,True,#ignore,unmapped historical municipality +44504,#IgnoreRow,True,#ignore,unmapped historical municipality +44521,#IgnoreRow,True,#ignore,unmapped historical municipality +44522,#IgnoreRow,True,#ignore,unmapped historical municipality +45000,observationAbout,wikidataId/Q130300 +45201,observationAbout,wikidataId/Q204308 +45202,observationAbout,wikidataId/Q638301 +45203,observationAbout,wikidataId/Q653389 +45204,observationAbout,wikidataId/Q860669 +45205,observationAbout,wikidataId/Q861283 +45206,observationAbout,wikidataId/Q850388 +45207,observationAbout,wikidataId/Q845800 +45208,observationAbout,wikidataId/Q850479 +45209,observationAbout,wikidataId/Q374701 +45301,#IgnoreRow,True,#ignore,unmapped historical municipality +45302,#IgnoreRow,True,#ignore,unmapped historical municipality +45303,#IgnoreRow,True,#ignore,unmapped historical municipality +45321,#IgnoreRow,True,#ignore,unmapped historical municipality +45322,#IgnoreRow,True,#ignore,unmapped historical municipality +45341,observationAbout,wikidataId/Q769821 +45342,#IgnoreRow,True,#ignore,unmapped historical municipality +45343,#IgnoreRow,True,#ignore,unmapped historical municipality +45344,#IgnoreRow,True,#ignore,unmapped historical municipality +45345,#IgnoreRow,True,#ignore,unmapped historical municipality +45361,observationAbout,wikidataId/Q1206617 +45362,#IgnoreRow,True,#ignore,unmapped historical municipality +45363,#IgnoreRow,True,#ignore,unmapped historical municipality +45381,#IgnoreRow,True,#ignore,unmapped historical municipality +45382,observationAbout,wikidataId/Q1350786 +45383,observationAbout,wikidataId/Q1350822 +45401,observationAbout,wikidataId/Q1206737 +45402,observationAbout,wikidataId/Q1206753 +45403,observationAbout,wikidataId/Q1206732 +45404,observationAbout,wikidataId/Q1350752 +45405,observationAbout,wikidataId/Q964194 +45406,observationAbout,wikidataId/Q1205974 +45421,observationAbout,wikidataId/Q1205839 +45422,#IgnoreRow,True,#ignore,unmapped historical municipality +45423,#IgnoreRow,True,#ignore,unmapped historical municipality +45424,#IgnoreRow,True,#ignore,unmapped historical municipality +45425,#IgnoreRow,True,#ignore,unmapped historical municipality +45426,#IgnoreRow,True,#ignore,unmapped historical municipality +45427,#IgnoreRow,True,#ignore,unmapped historical municipality +45428,#IgnoreRow,True,#ignore,unmapped historical municipality +45429,observationAbout,wikidataId/Q1352167 +45430,observationAbout,wikidataId/Q1205964 +45431,observationAbout,wikidataId/Q1352182 +45441,observationAbout,wikidataId/Q1350939 +45442,observationAbout,wikidataId/Q1206723 +45443,observationAbout,wikidataId/Q1206765 +46000,observationAbout,wikidataId/Q15701 +46201,observationAbout,wikidataId/Q15674 +46202,#IgnoreRow,True,#ignore,unmapped historical municipality +46203,observationAbout,wikidataId/Q819578 +46204,observationAbout,wikidataId/Q548493 +46205,#IgnoreRow,True,#ignore,unmapped historical municipality +46206,observationAbout,wikidataId/Q15732 +46207,#IgnoreRow,True,#ignore,unmapped historical municipality +46208,observationAbout,wikidataId/Q15734 +46209,#IgnoreRow,True,#ignore,unmapped historical municipality +46210,observationAbout,wikidataId/Q19712 +46211,#IgnoreRow,True,#ignore,unmapped historical municipality +46212,#IgnoreRow,True,#ignore,unmapped historical municipality +46213,observationAbout,wikidataId/Q847602 +46214,observationAbout,wikidataId/Q847591 +46215,observationAbout,wikidataId/Q15720 +46216,observationAbout,wikidataId/Q15716 +46217,observationAbout,wikidataId/Q858382 +46218,observationAbout,wikidataId/Q858352 +46219,observationAbout,wikidataId/Q15725 +46220,observationAbout,wikidataId/Q15754 +46221,observationAbout,wikidataId/Q858378 +46222,observationAbout,wikidataId/Q453502 +46223,observationAbout,wikidataId/Q19701 +46224,observationAbout,wikidataId/Q15736 +46225,observationAbout,wikidataId/Q407947 +46301,#IgnoreRow,True,#ignore,unmapped historical municipality +46302,#IgnoreRow,True,#ignore,unmapped historical municipality +46303,observationAbout,wikidataId/Q702628 +46304,observationAbout,wikidataId/Q701079 +46321,#IgnoreRow,True,#ignore,unmapped historical municipality +46322,#IgnoreRow,True,#ignore,unmapped historical municipality +46323,#IgnoreRow,True,#ignore,unmapped historical municipality +46324,#IgnoreRow,True,#ignore,unmapped historical municipality +46341,#IgnoreRow,True,#ignore,unmapped historical municipality +46342,#IgnoreRow,True,#ignore,unmapped historical municipality +46343,#IgnoreRow,True,#ignore,unmapped historical municipality +46344,#IgnoreRow,True,#ignore,unmapped historical municipality +46345,#IgnoreRow,True,#ignore,unmapped historical municipality +46361,#IgnoreRow,True,#ignore,unmapped historical municipality +46362,#IgnoreRow,True,#ignore,unmapped historical municipality +46363,#IgnoreRow,True,#ignore,unmapped historical municipality +46364,#IgnoreRow,True,#ignore,unmapped historical municipality +46365,#IgnoreRow,True,#ignore,unmapped historical municipality +46366,#IgnoreRow,True,#ignore,unmapped historical municipality +46367,#IgnoreRow,True,#ignore,unmapped historical municipality +46368,#IgnoreRow,True,#ignore,unmapped historical municipality +46381,#IgnoreRow,True,#ignore,unmapped historical municipality +46382,#IgnoreRow,True,#ignore,unmapped historical municipality +46383,#IgnoreRow,True,#ignore,unmapped historical municipality +46384,#IgnoreRow,True,#ignore,unmapped historical municipality +46385,#IgnoreRow,True,#ignore,unmapped historical municipality +46386,#IgnoreRow,True,#ignore,unmapped historical municipality +46387,#IgnoreRow,True,#ignore,unmapped historical municipality +46388,#IgnoreRow,True,#ignore,unmapped historical municipality +46389,#IgnoreRow,True,#ignore,unmapped historical municipality +46390,#IgnoreRow,True,#ignore,unmapped historical municipality +46391,#IgnoreRow,True,#ignore,unmapped historical municipality +46392,observationAbout,wikidataId/Q730546 +46401,#IgnoreRow,True,#ignore,unmapped historical municipality +46402,#IgnoreRow,True,#ignore,unmapped historical municipality +46403,#IgnoreRow,True,#ignore,unmapped historical municipality +46404,observationAbout,wikidataId/Q1350888 +46421,#IgnoreRow,True,#ignore,unmapped historical municipality +46441,#IgnoreRow,True,#ignore,unmapped historical municipality +46442,#IgnoreRow,True,#ignore,unmapped historical municipality +46443,#IgnoreRow,True,#ignore,unmapped historical municipality +46444,#IgnoreRow,True,#ignore,unmapped historical municipality +46445,#IgnoreRow,True,#ignore,unmapped historical municipality +46446,#IgnoreRow,True,#ignore,unmapped historical municipality +46447,#IgnoreRow,True,#ignore,unmapped historical municipality +46448,#IgnoreRow,True,#ignore,unmapped historical municipality +46449,#IgnoreRow,True,#ignore,unmapped historical municipality +46450,#IgnoreRow,True,#ignore,unmapped historical municipality +46451,#IgnoreRow,True,#ignore,unmapped historical municipality +46452,observationAbout,wikidataId/Q1350873 +46461,#IgnoreRow,True,#ignore,unmapped historical municipality +46462,#IgnoreRow,True,#ignore,unmapped historical municipality +46463,#IgnoreRow,True,#ignore,unmapped historical municipality +46464,#IgnoreRow,True,#ignore,unmapped historical municipality +46465,#IgnoreRow,True,#ignore,unmapped historical municipality +46466,#IgnoreRow,True,#ignore,unmapped historical municipality +46467,#IgnoreRow,True,#ignore,unmapped historical municipality +46468,observationAbout,wikidataId/Q931439 +46481,#IgnoreRow,True,#ignore,unmapped historical municipality +46482,observationAbout,wikidataId/Q1206619 +46483,#IgnoreRow,True,#ignore,unmapped historical municipality +46484,#IgnoreRow,True,#ignore,unmapped historical municipality +46485,#IgnoreRow,True,#ignore,unmapped historical municipality +46486,#IgnoreRow,True,#ignore,unmapped historical municipality +46487,#IgnoreRow,True,#ignore,unmapped historical municipality +46488,#IgnoreRow,True,#ignore,unmapped historical municipality +46489,#IgnoreRow,True,#ignore,unmapped historical municipality +46490,observationAbout,wikidataId/Q1206156 +46491,observationAbout,wikidataId/Q1208563 +46492,observationAbout,wikidataId/Q1350770 +46501,observationAbout,wikidataId/Q1205885 +46502,observationAbout,wikidataId/Q1350969 +46503,#IgnoreRow,True,#ignore,unmapped historical municipality +46504,#IgnoreRow,True,#ignore,unmapped historical municipality +46505,observationAbout,wikidataId/Q1350724 +46523,observationAbout,wikidataId/Q368508 +46524,observationAbout,wikidataId/Q1205992 +46525,observationAbout,wikidataId/Q1156548 +46526,#IgnoreRow,True,#ignore,unmapped historical municipality +46527,observationAbout,wikidataId/Q1208553 +46528,#IgnoreRow,True,#ignore,unmapped historical municipality +46529,observationAbout,wikidataId/Q1206640 +46530,observationAbout,wikidataId/Q1350985 +46531,observationAbout,wikidataId/Q1350711 +46532,observationAbout,wikidataId/Q1205986 +46533,observationAbout,wikidataId/Q1205996 +46534,observationAbout,wikidataId/Q932423 +46535,observationAbout,wikidataId/Q1206643 +47000,observationAbout,wikidataId/Q766445 +47201,observationAbout,wikidataId/Q181966 +47202,#IgnoreRow,True,#ignore,unmapped historical municipality +47203,#IgnoreRow,True,#ignore,unmapped historical municipality +47205,observationAbout,wikidataId/Q592065 +47206,#IgnoreRow,True,#ignore,unmapped historical municipality +47207,observationAbout,wikidataId/Q838366 +47208,observationAbout,wikidataId/Q695895 +47209,observationAbout,wikidataId/Q117604 +47210,observationAbout,wikidataId/Q860662 +47211,observationAbout,wikidataId/Q328615 +47212,observationAbout,wikidataId/Q371446 +47213,observationAbout,wikidataId/Q694313 +47214,observationAbout,wikidataId/Q714636 +47215,observationAbout,wikidataId/Q874691 +47301,observationAbout,wikidataId/Q1350839 +47302,observationAbout,wikidataId/Q1351188 +47303,observationAbout,wikidataId/Q1202988 +47306,observationAbout,wikidataId/Q1184661 +47308,observationAbout,wikidataId/Q1205928 +47311,observationAbout,wikidataId/Q1202976 +47313,observationAbout,wikidataId/Q1203012 +47314,observationAbout,wikidataId/Q1350738 +47315,observationAbout,wikidataId/Q1193528 +47322,#IgnoreRow,True,#ignore,unmapped historical municipality +47323,#IgnoreRow,True,#ignore,unmapped historical municipality +47324,observationAbout,wikidataId/Q973994 +47325,observationAbout,wikidataId/Q1205469 +47326,observationAbout,wikidataId/Q1351759 +47327,observationAbout,wikidataId/Q1351145 +47328,observationAbout,wikidataId/Q1205406 +47329,observationAbout,wikidataId/Q1206796 +47341,#IgnoreRow,True,#ignore,unmapped historical municipality +47343,#IgnoreRow,True,#ignore,unmapped historical municipality +47344,#IgnoreRow,True,#ignore,unmapped historical municipality +47345,#IgnoreRow,True,#ignore,unmapped historical municipality +47346,#IgnoreRow,True,#ignore,unmapped historical municipality +47347,#IgnoreRow,True,#ignore,unmapped historical municipality +47348,observationAbout,wikidataId/Q1351158 +47349,#IgnoreRow,True,#ignore,unmapped historical municipality +47350,observationAbout,wikidataId/Q1206308 +47351,#IgnoreRow,True,#ignore,unmapped historical municipality +47352,#IgnoreRow,True,#ignore,unmapped historical municipality +47353,observationAbout,wikidataId/Q1351168 +47354,observationAbout,wikidataId/Q145373 +47355,observationAbout,wikidataId/Q1205452 +47356,observationAbout,wikidataId/Q1205897 +47357,observationAbout,wikidataId/Q1351200 +47358,observationAbout,wikidataId/Q1207554 +47359,observationAbout,wikidataId/Q1193315 +47360,observationAbout,wikidataId/Q981332 +47361,observationAbout,wikidataId/Q1206086 +47362,observationAbout,wikidataId/Q1206647 +47371,#IgnoreRow,True,#ignore,unmapped historical municipality +47372,#IgnoreRow,True,#ignore,unmapped historical municipality +47373,#IgnoreRow,True,#ignore,unmapped historical municipality +47374,#IgnoreRow,True,#ignore,unmapped historical municipality +47375,observationAbout,wikidataId/Q1202856 +47381,observationAbout,wikidataId/Q699764 +47382,observationAbout,wikidataId/Q702623 From 60bd9cf9c1637fe8cef45488da81dad4343504ed Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Thu, 22 Oct 2026 06:53:23 +0000 Subject: [PATCH 03/17] validation config changes --- statvar_imports/japan_census/validation_config.json | 1 - 1 file changed, 1 deletion(-) diff --git a/statvar_imports/japan_census/validation_config.json b/statvar_imports/japan_census/validation_config.json index 925a8109fd..d4ce4d56a9 100644 --- a/statvar_imports/japan_census/validation_config.json +++ b/statvar_imports/japan_census/validation_config.json @@ -5,7 +5,6 @@ "rule_id": "check_deleted_records_percent", "description": "Strictly enforce historical deletion average threshold of 0.1%", "validator": "DELETED_RECORDS_PERCENT", - "enabled": false, "params": { "threshold": 0.1 } From 88b2ab88bec3bacf6cd44512abf8e62da4c901cb Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Thu, 22 Oct 2026 07:13:51 +0000 Subject: [PATCH 04/17] gemini code review comments --- statvar_imports/japan_census/process.py | 7 +++++-- statvar_imports/japan_census/process_test.py | 6 ++++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index 60766cd56a..38b46fff49 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -26,6 +26,7 @@ import shutil import subprocess import sys +import tempfile from absl import app from absl import flags @@ -81,6 +82,9 @@ def _expand_input_files(input_data: str) -> list[str]: pattern = pattern.strip() if not pattern: continue + dir_path = os.path.dirname(pattern) + if dir_path and os.path.exists(dir_path) and not os.listdir(dir_path): + raise FileNotFoundError(f'Input directory is empty: {dir_path}') matched = sorted(glob.glob(pattern)) if matched: files.extend(matched) @@ -243,7 +247,7 @@ def main(_) -> None: shard_counters: list[str] = [] futures = [] - with concurrent.futures.ProcessPoolExecutor( + with concurrent.futures.ThreadPoolExecutor( max_workers=max_workers ) as executor: for idx, input_file in enumerate(input_files): @@ -305,5 +309,4 @@ def main(_) -> None: if __name__ == '__main__': - import tempfile app.run(main) diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py index 02193c6566..debdd7fee5 100644 --- a/statvar_imports/japan_census/process_test.py +++ b/statvar_imports/japan_census/process_test.py @@ -45,6 +45,12 @@ def test_expand_input_files_sorts_largest_first(self): expanded = process._expand_input_files(pattern) self.assertEqual(expanded, [large_file, small_file]) + def test_expand_input_files_empty_dir_raises(self): + with tempfile.TemporaryDirectory() as tmp_dir: + pattern = os.path.join(tmp_dir, '*.csv') + with self.assertRaises(FileNotFoundError): + process._expand_input_files(pattern) + def test_merge_csv_files_keeps_single_header(self): with tempfile.TemporaryDirectory() as tmp_dir: p1 = os.path.join(tmp_dir, 'shard_00') From 7f616eee151f40fb29695d3e7fa3c62e82e262bc Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Fri, 23 Oct 2026 08:43:57 +0000 Subject: [PATCH 05/17] Address code review findings for Japan Census import --- statvar_imports/japan_census/README.md | 3 +- .../japan_census/download_script.py | 90 ++++++++++++------- .../japan_census/download_script_test.py | 37 +++++--- statvar_imports/japan_census/process.py | 36 +++++++- statvar_imports/japan_census/process_test.py | 15 ++++ .../japan_census/validation_config.json | 8 +- 6 files changed, 136 insertions(+), 53 deletions(-) diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md index 0ef888246d..683510a5a8 100644 --- a/statvar_imports/japan_census/README.md +++ b/statvar_imports/japan_census/README.md @@ -13,7 +13,8 @@ This project downloads, processes, and imports demographic, employment, educatio * **Employment & Unemployment Rate (National)**: Annual (`1953`–`2025`). * **Student Enrollment (National)**: Annual (`1948`–`2026`). * **Teachers (National & Prefecture)**: Annual (`1975`–`2024`). - * **Deaths & Causes of Death (National & Prefecture)**: Annual (`1975`–`2025`). + * **Deaths (National & Prefecture)**: Annual (`1975`–`2025`). + * **Causes of Death (National & Prefecture)**: Annual (`1975`–`2023`). * **Release Frequency**: Annual (vital, labor, and education statistics) and quinquennial (5-year Population Census). * **Notes**: This dataset covers 101 StatisticalVariables across 1,965 places (`country/JPN`, 47 Prefectures, and 1,917 Municipalities). diff --git a/statvar_imports/japan_census/download_script.py b/statvar_imports/japan_census/download_script.py index bd1671e9b6..887d42665a 100644 --- a/statvar_imports/japan_census/download_script.py +++ b/statvar_imports/japan_census/download_script.py @@ -19,15 +19,19 @@ import os import sys import tempfile +import threading from typing import Any +import requests +from requests.adapters import HTTPAdapter +from urllib3.util.retry import Retry + _MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(1, os.path.join(_MODULE_DIR, '../../util/')) from absl import app from absl import flags from absl import logging -import download_util import file_util _FLAGS = flags.FLAGS @@ -215,14 +219,11 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: status_code = str(result.get('status', '')) error_msg = str(result.get('errorMsg', '')) - if status_code not in ('0', '100'): + if status_code != '0': raise RuntimeError( f'e-Stat API error (status={status_code}): {error_msg}' ) - if status_code == '100': - return [] - stat_data = get_stats.get('STATISTICAL_DATA', {}) class_objs = stat_data.get('CLASS_INF', {}).get('CLASS_OBJ', []) @@ -265,9 +266,47 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: 'isProvisional': str(val_obj.get('@isProvisional', '')), }) + if not parsed_records: + raise RuntimeError('e-Stat API returned 0 data records.') + return parsed_records +_THREAD_LOCAL = threading.local() + + +def _get_http_session() -> requests.Session: + """Returns a thread-local requests.Session configured with automatic retries.""" + session = getattr(_THREAD_LOCAL, 'session', None) + if session is None: + session = requests.Session() + retry_strategy = Retry( + total=3, + connect=3, + read=3, + status=3, + backoff_factor=2.0, + status_forcelist=(429, 500, 502, 503, 504), + allowed_methods=frozenset(['GET']), + raise_on_status=False, + ) + adapter = HTTPAdapter(max_retries=retry_strategy) + session.mount('https://', adapter) + session.mount('http://', adapter) + _THREAD_LOCAL.session = session + return session + + +def _request_estat_json( + url: str, params: dict[str, str], timeout: int = 60 +) -> dict[str, Any]: + """Performs a thread-safe GET request to the e-Stat JSON API with retries.""" + session = _get_http_session() + resp = session.get(url, params=params, timeout=timeout) + resp.raise_for_status() + return resp.json() + + def fetch_indicator_batch( indicators: list[str], regional_rank: str, @@ -297,14 +336,10 @@ def fetch_indicator_batch( len(indicators), indicators[0], ) - response_json = download_util.request_url( + response_json = _request_estat_json( url=ESTAT_JSON_API_URL, params=params, - method='GET', - output='json', timeout=60, - retries=3, - retry_secs=5, ) if not response_json or not isinstance(response_json, dict): raise RuntimeError( @@ -404,22 +439,17 @@ def _load_historical_rows(historical_gcs_path: str) -> list[dict[str, str]]: return [] if historical_gcs_path in _HISTORICAL_CACHE: return _HISTORICAL_CACHE[historical_gcs_path] - try: - logging.info( - 'Reading historical deleted data from GCS: %s', historical_gcs_path - ) - with file_util.FileIO(historical_gcs_path, 'r', encoding='utf-8') as f: - rows = list(csv.DictReader(f)) - _HISTORICAL_CACHE[historical_gcs_path] = rows - return rows - except Exception as e: - logging.warning( - 'Could not read historical deleted data from %s: %s. Proceeding with fresh data only.', - historical_gcs_path, - e, + logging.info( + 'Reading historical deleted data from GCS: %s', historical_gcs_path + ) + with file_util.FileIO(historical_gcs_path, 'r', encoding='utf-8') as f: + rows = list(csv.DictReader(f)) + if not rows: + raise RuntimeError( + f'Historical supplement file is empty: {historical_gcs_path}' ) - _HISTORICAL_CACHE[historical_gcs_path] = [] - return [] + _HISTORICAL_CACHE[historical_gcs_path] = rows + return rows def _apply_historical_supplement( @@ -531,17 +561,17 @@ def download_dataset( if is_census: all_records = _derive_census_population_rows(all_records, census_mode) + if not all_records: + raise RuntimeError( + f'No records retrieved for dataset {os.path.basename(output_filepath)}' + ) + all_records = _apply_historical_supplement( os.path.basename(output_filepath), all_records, historical_gcs_path=historical_gcs_path, ) - if not all_records: - raise RuntimeError( - f'No records retrieved for dataset {os.path.basename(output_filepath)}' - ) - if census_mode == 'municipality': # Split municipality census by year (matching CNS jpn_aa1_aa2_age_gender_{year}.csv) # so multi-core processing can run across all census years in parallel. diff --git a/statvar_imports/japan_census/download_script_test.py b/statvar_imports/japan_census/download_script_test.py index 1969475de6..5835cb9588 100644 --- a/statvar_imports/japan_census/download_script_test.py +++ b/statvar_imports/japan_census/download_script_test.py @@ -110,6 +110,10 @@ def setUp(self): } } + def tearDown(self): + download_script._HISTORICAL_CACHE.clear() + super().tearDown() + def test_parse_estat_json_response_success(self): records = download_script.parse_estat_json_response( self.sample_api_response @@ -126,16 +130,17 @@ def test_parse_estat_json_response_success(self): self.assertEqual(records[0]['value'], '1.9') def test_parse_estat_json_response_error_raises(self): - error_payload = { - 'GET_STATS': { - 'RESULT': { - 'status': '400', - 'errorMsg': 'Too many records.', + for status in ('400', '100'): + error_payload = { + 'GET_STATS': { + 'RESULT': { + 'status': status, + 'errorMsg': 'API error or no data.', + } } } - } - with self.assertRaises(RuntimeError): - download_script.parse_estat_json_response(error_payload) + with self.assertRaises(RuntimeError): + download_script.parse_estat_json_response(error_payload) def test_write_csv_atomically_failure_cleanup(self): with tempfile.TemporaryDirectory() as tmp_dir: @@ -148,9 +153,9 @@ def test_write_csv_atomically_failure_cleanup(self): self.assertFalse(os.path.exists(target_path)) self.assertEqual(os.listdir(tmp_dir), []) - @mock.patch.object(download_script.download_util, 'request_url') - def test_download_dataset_empty_error(self, mock_request_url): - mock_request_url.return_value = { + @mock.patch.object(download_script, '_request_estat_json') + def test_download_dataset_empty_error(self, mock_request_json): + mock_request_json.return_value = { 'GET_STATS': { 'RESULT': { 'status': '100', @@ -168,9 +173,9 @@ def test_download_dataset_empty_error(self, mock_request_url): batch_size=1, ) - @mock.patch.object(download_script.download_util, 'request_url') - def test_download_dataset_writes_csv_atomically(self, mock_request_url): - mock_request_url.return_value = self.sample_api_response + @mock.patch.object(download_script, '_request_estat_json') + def test_download_dataset_writes_csv_atomically(self, mock_request_json): + mock_request_json.return_value = self.sample_api_response with tempfile.TemporaryDirectory() as tmp_dir: out_file = os.path.join(tmp_dir, 'jpn_unemploymentrate.csv') count = download_script.download_dataset( @@ -213,6 +218,10 @@ def test_apply_historical_supplement_from_path(self): self.assertEqual(merged[0]['regionCd'], '01000') self.assertEqual(merged[0]['value'], '2359183') + def test_load_historical_rows_missing_file_raises(self): + with self.assertRaises( Exception ): + download_script._load_historical_rows('/nonexistent/deleted_historical_data.csv') + if __name__ == '__main__': unittest.main() diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index 38b46fff49..464aecc231 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -33,6 +33,9 @@ from absl import logging _MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(1, os.path.join(_MODULE_DIR, '../../util/')) +import file_util + _PROCESSOR_SCRIPT = os.path.abspath( os.path.join(_MODULE_DIR, '../../tools/statvar_importer/stat_var_processor.py') ) @@ -210,9 +213,7 @@ def _merge_mcf_files(shard_prefixes: list[str], suffix: str, final_mcf_path: str if node_line not in seen_nodes: seen_nodes[node_line] = '\n'.join(lines) - if os.path.isdir(final_mcf_path): - shutil.rmtree(final_mcf_path) - elif os.path.exists(final_mcf_path): + if os.path.exists(final_mcf_path): os.remove(final_mcf_path) if not seen_nodes: @@ -223,6 +224,26 @@ def _merge_mcf_files(shard_prefixes: list[str], suffix: str, final_mcf_path: str out_f.write(seen_nodes[node_key] + '\n\n') +def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: + """Copies a gs:// existing_statvar_mcf file once to tmp_dir for all shards.""" + if not existing_statvar_mcf or not existing_statvar_mcf.startswith('gs://'): + return existing_statvar_mcf + local_mcf = os.path.join( + tmp_dir, os.path.basename(existing_statvar_mcf) or 'stat_vars.mcf' + ) + logging.info( + 'Caching %s once to %s for all shard workers...', + existing_statvar_mcf, + local_mcf, + ) + file_util.file_copy(existing_statvar_mcf, local_mcf) + if not os.path.exists(local_mcf) or os.path.getsize(local_mcf) == 0: + raise RuntimeError( + f'Failed to copy existing_statvar_mcf from {existing_statvar_mcf}' + ) + return local_mcf + + def main(_) -> None: input_files = _expand_input_files(_FLAGS.input_data) if not input_files: @@ -243,6 +264,9 @@ def main(_) -> None: ) try: + local_statvar_mcf = _prepare_local_statvar_mcf( + _FLAGS.existing_statvar_mcf, shards_tmp_dir + ) shard_prefixes: list[str] = [] shard_counters: list[str] = [] futures = [] @@ -265,7 +289,7 @@ def main(_) -> None: s_counters, os.path.abspath(_FLAGS.pv_map), os.path.abspath(_FLAGS.config_file), - _FLAGS.existing_statvar_mcf, + local_statvar_mcf, ) ) @@ -275,6 +299,10 @@ def main(_) -> None: # 1. Merge CSV outputs final_csv = f'{output_path}.csv' total_rows = _merge_csv_files(shard_prefixes, final_csv) + if total_rows == 0: + raise RuntimeError( + f'Merged output CSV has 0 rows: {final_csv}' + ) logging.info('Merged %d SVObs rows into %s', total_rows, final_csv) # 2. Generate unified TMCF (normalizing table reference to japan_census_output) diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py index debdd7fee5..855fed941a 100644 --- a/statvar_imports/japan_census/process_test.py +++ b/statvar_imports/japan_census/process_test.py @@ -148,6 +148,21 @@ def test_run_shard_raises_on_nonzero_exit(self, mock_run): existing_statvar_mcf='stat_vars.mcf', ) + @mock.patch.object(process.file_util, 'file_copy') + def test_prepare_local_statvar_mcf_copies_gcs_path(self, mock_file_copy): + with tempfile.TemporaryDirectory() as tmp_dir: + def _fake_copy(src, dst): + with open(dst, 'w', encoding='utf-8') as f: + f.write('Node: dcid:Count_Person\n') + return dst + + mock_file_copy.side_effect = _fake_copy + local_path = process._prepare_local_statvar_mcf( + 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', tmp_dir + ) + self.assertEqual(local_path, os.path.join(tmp_dir, 'stat_vars.mcf')) + mock_file_copy.assert_called_once() + if __name__ == '__main__': unittest.main() diff --git a/statvar_imports/japan_census/validation_config.json b/statvar_imports/japan_census/validation_config.json index d4ce4d56a9..273d0f42e2 100644 --- a/statvar_imports/japan_census/validation_config.json +++ b/statvar_imports/japan_census/validation_config.json @@ -11,11 +11,11 @@ }, { "rule_id": "check_max_date_freshness", - "description": "Verify MaxDate is within allowable publication lag and values are non-negative (and within [0, 100] for rates) for active recurring indicators", + "description": "Verify MaxDate is within allowable publication lag (3 years for annual series, 5 years for quinquennial census) and values are non-negative (and within [0, 100] for rates) for active recurring indicators", "validator": "SQL_VALIDATOR", "params": { - "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, NumPlaces, COUNT(*) OVER () AS total_svs FROM stats WHERE StatVar NOT IN ('Count_Person_100To104Years', 'Count_Person_100To104Years_Male', 'Count_Person_100To104Years_Female', 'Count_Person_105To109Years', 'Count_Person_105To109Years_Male', 'Count_Person_105To109Years_Female', 'Count_Person_110OrMoreYears', 'Count_Person_110OrMoreYears_Male', 'Count_Person_110OrMoreYears_Female')", - "condition": "CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - 5) AND total_svs >= 92 AND NumPlaces >= 1 AND MinValue >= 0 AND (StatVar != 'UnemploymentRate_Person' OR MaxValue <= 100)" + "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, NumPlaces, COUNT(*) OVER () AS total_svs FROM stats WHERE NOT regexp_matches(StatVar, '^Count_Person_1(00To104|05To109|10OrMore)Years')", + "condition": "CAST(MaxDate AS INTEGER) >= (EXTRACT(YEAR FROM CURRENT_DATE) - CASE WHEN regexp_matches(StatVar, '^(Count_(Student|Teacher|Death|MortalityEvent|Person_(Employed|Unemployed|NotInLaborForce))|UnemploymentRate_Person)') THEN 3 ELSE 5 END) AND total_svs >= 92 AND NumPlaces >= 1 AND MinValue >= 0 AND (StatVar != 'UnemploymentRate_Person' OR MaxValue <= 100)" } }, { @@ -23,7 +23,7 @@ "description": "Verify MaxDate is at least the last published census wave (2020) and values are non-negative for discontinued historical centenarian sub-brackets", "validator": "SQL_VALIDATOR", "params": { - "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, COUNT(*) OVER () AS total_svs FROM stats WHERE StatVar IN ('Count_Person_100To104Years', 'Count_Person_100To104Years_Male', 'Count_Person_100To104Years_Female', 'Count_Person_105To109Years', 'Count_Person_105To109Years_Male', 'Count_Person_105To109Years_Female', 'Count_Person_110OrMoreYears', 'Count_Person_110OrMoreYears_Male', 'Count_Person_110OrMoreYears_Female')", + "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, COUNT(*) OVER () AS total_svs FROM stats WHERE regexp_matches(StatVar, '^Count_Person_1(00To104|05To109|10OrMore)Years')", "condition": "CAST(MaxDate AS INTEGER) >= 2020 AND total_svs >= 9 AND MinValue >= 0" } } From 573acabdb99f5367b6bdb42ed0e5febfb9f4f2a5 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Fri, 23 Oct 2026 10:29:50 +0000 Subject: [PATCH 06/17] Address additional code review findings for Japan Census import --- statvar_imports/japan_census/README.md | 5 +- .../japan_census/download_script.py | 103 +++++----- .../japan_census/japan_census_pv_map.csv | 4 +- statvar_imports/japan_census/process.py | 194 ++++++++++++------ statvar_imports/japan_census/process_test.py | 53 +++-- .../japan_census/validation_config.json | 72 +++++++ 6 files changed, 293 insertions(+), 138 deletions(-) diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md index 683510a5a8..747a190706 100644 --- a/statvar_imports/japan_census/README.md +++ b/statvar_imports/japan_census/README.md @@ -35,7 +35,9 @@ The import pipeline queries the e-Stat Statistical Dashboard JSON API in paralle * **Transformation pipeline**: 1. `download_script.py` fetches all 9 dataset groups from the e-Stat JSON API using concurrent workers and atomic file writes into `input_files/`. 2. `process.py` invokes `stat_var_processor.py` in parallel across the 18 input CSVs against `japan_census_pv_map.csv` and `japan_census_metadata.csv`, referencing canonical schema `gs://unresolved_mcf/scripts/statvar/stat_vars.mcf`. - 3. The processor filters non-target regional aggregate rows via `#IgnoreRow`, scales labor force counts reported in units of 10,000 persons to thousands via `#Multiply,10`, resolves place DCIDs, and produces `output_files/japan_census_output.csv` and `output_files/japan_census_output.tmcf`. + 3. The processor filters non-target regional aggregate rows via `#IgnoreRow`, resolves place DCIDs, and produces `output_files/japan_census_output.csv` and `output_files/japan_census_output.tmcf`. To preserve 1:1 historical series continuity (`0` deleted / `0` modified records) with the legacy `Japan_Census` import: + * Labor force counts (`0301010000010010010`, `0301010000020010010`, `0301010000030010010`, reported by e-Stat in units of `10,000 persons`) apply `#Multiply,10` to match the historical scaling (in thousands). + * Indicator `1201040500000010000` (`Number of upper secondary school teachers (full-time)`) maps to `dcid:Count_Teacher_UniversityAndCollegeTeacher` to match the legacy StatVar assignment across `1975`–`2021`. 4. Aggregated processor counters are written to `counters/japan_census_counters.csv`. * **Data Quality Checks**: @@ -43,6 +45,7 @@ The import pipeline queries the e-Stat Statistical Dashboard JSON API in paralle * `check_deleted_records_percent`: Historical deletion threshold `<= 0.1%` (`0` deleted records). * `check_max_date_freshness`: Verifies `MaxDate`, `NumPlaces >= 1`, `MinValue >= 0`, and rate bounds (`MaxValue <= 100` for `UnemploymentRate_Person`) across all 92 active recurring indicators. * `check_max_date_historical_indicators`: Verifies `MaxDate >= 2020` and `MinValue >= 0` across the 9 discontinued centenarian sub-brackets (`100–104`, `105–109`, `110+`). + * `check_max_date_consistent_*`: Scoped `MAX_DATE_CONSISTENT` rules verifying uniform `MaxDate` within each co-released survey group (Population Census, Labor Force & Deaths, Teachers, Students, Causes of Death, and Historical Centenarians). --- diff --git a/statvar_imports/japan_census/download_script.py b/statvar_imports/japan_census/download_script.py index 887d42665a..3a58093c2e 100644 --- a/statvar_imports/japan_census/download_script.py +++ b/statvar_imports/japan_census/download_script.py @@ -95,7 +95,7 @@ def _build_population_5yr_indicators( # RegionalRank: '2' = Country (Japan), '3' = Prefecture, '4' = Municipality. # Cycle: '3' = Calendar Year. DATASET_CONFIGS: dict[str, dict[str, Any]] = { - 'jpn_employement.csv': { + 'jpn_employment.csv': { 'indicators': [ '0301010000010010010', # Employed persons (Both sexes) '0301010000020010010', # Unemployed persons (Both sexes) @@ -178,9 +178,8 @@ def _build_population_5yr_indicators( } -def _extract_class_lookup( - class_objs: list[dict[str, Any]] | dict[str, Any], target_id: str -) -> dict[str, str]: +def _extract_class_lookup(class_objs: list[dict[str, Any]] | dict[str, Any], + target_id: str) -> dict[str, str]: """Extracts a code-to-name dictionary from e-Stat CLASS_INF.CLASS_OBJ.""" if isinstance(class_objs, dict): class_objs = [class_objs] @@ -190,9 +189,8 @@ def _extract_class_lookup( if isinstance(classes, dict): classes = [classes] return { - str(c.get('@code', '')): str( - c.get('@name') if '@name' in c else c.get('$', '') - ) + str(c.get('@code', '')): + str(c.get('@name') if '@name' in c else c.get('$', '')) for c in classes if isinstance(c, dict) } @@ -212,7 +210,8 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: RuntimeError: If the API returned an error status code or invalid payload. """ if not isinstance(data, dict) or 'GET_STATS' not in data: - raise RuntimeError('Invalid JSON response from e-Stat API: missing GET_STATS.') + raise RuntimeError( + 'Invalid JSON response from e-Stat API: missing GET_STATS.') get_stats = data['GET_STATS'] result = get_stats.get('RESULT', {}) @@ -221,8 +220,7 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: if status_code != '0': raise RuntimeError( - f'e-Stat API error (status={status_code}): {error_msg}' - ) + f'e-Stat API error (status={status_code}): {error_msg}') stat_data = get_stats.get('STATISTICAL_DATA', {}) class_objs = stat_data.get('CLASS_INF', {}).get('CLASS_OBJ', []) @@ -245,9 +243,8 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: unit_cd = str(val_obj.get('@unit', '')) reg_cd = str(val_obj.get('@regionCode', '')) time_cd = str(val_obj.get('@time', '')) - year_str = time_map.get( - time_cd, time_cd[:4] if len(time_cd) >= 4 else time_cd - ) + year_str = time_map.get(time_cd, + time_cd[:4] if len(time_cd) >= 4 else time_cd) parsed_records.append({ 'year': year_str, @@ -297,9 +294,9 @@ def _get_http_session() -> requests.Session: return session -def _request_estat_json( - url: str, params: dict[str, str], timeout: int = 60 -) -> dict[str, Any]: +def _request_estat_json(url: str, + params: dict[str, str], + timeout: int = 60) -> dict[str, Any]: """Performs a thread-safe GET request to the e-Stat JSON API with retries.""" session = _get_http_session() resp = session.get(url, params=params, timeout=timeout) @@ -348,9 +345,8 @@ def fetch_indicator_batch( return parse_estat_json_response(response_json) -def _derive_census_population_rows( - records: list[dict[str, str]], mode: str -) -> list[dict[str, str]]: +def _derive_census_population_rows(records: list[dict[str, str]], + mode: str) -> list[dict[str, str]]: """Filters to 5-year census years and computes exact unrounded broad age sums. At National (RegionalRank=2) and Prefecture (RegionalRank=3) levels, direct @@ -379,8 +375,7 @@ def _derive_census_population_rows( else: max_5yr_suffix = 230 if mode == 'national' else 170 allowed_5yr = { - f'0201010{g}20000010{i:03d}' - for g in ('0', '1', '2') + f'0201010{g}20000010{i:03d}' for g in ('0', '1', '2') for i in range(10, max_5yr_suffix + 10, 10) } allowed_totals = { @@ -396,14 +391,16 @@ def _derive_census_population_rows( # Derive exact Census counts for <15, 15-64, 65+, and (for National/Prefecture) 85+ from 5-year brackets. broad_Specs = [ - ('10000010010', 'Population (under 15)', [f'{i:03d}' for i in (10, 20, 30)], False), - ('10000010020', 'Population (15-64)', [f'{i:03d}' for i in range(40, 140, 10)], False), - ('10000010030', 'Population (65 and over)', [f'{i:03d}' for i in range(140, 210, 10)], True), + ('10000010010', 'Population (under 15)', + [f'{i:03d}' for i in (10, 20, 30)], False), + ('10000010020', 'Population (15-64)', + [f'{i:03d}' for i in range(40, 140, 10)], False), + ('10000010030', 'Population (65 and over)', + [f'{i:03d}' for i in range(140, 210, 10)], True), ] if mode != 'municipality': - broad_Specs.append( - ('40000010070', 'Population (85 and over)', [f'{i:03d}' for i in (180, 190, 200)], True) - ) + broad_Specs.append(('40000010070', 'Population (85 and over)', + [f'{i:03d}' for i in (180, 190, 200)], True)) for reg_cd, yr in sorted(reg_meta.keys()): base = reg_meta[(reg_cd, yr)] @@ -439,15 +436,13 @@ def _load_historical_rows(historical_gcs_path: str) -> list[dict[str, str]]: return [] if historical_gcs_path in _HISTORICAL_CACHE: return _HISTORICAL_CACHE[historical_gcs_path] - logging.info( - 'Reading historical deleted data from GCS: %s', historical_gcs_path - ) + logging.info('Reading historical deleted data from GCS: %s', + historical_gcs_path) with file_util.FileIO(historical_gcs_path, 'r', encoding='utf-8') as f: rows = list(csv.DictReader(f)) if not rows: raise RuntimeError( - f'Historical supplement file is empty: {historical_gcs_path}' - ) + f'Historical supplement file is empty: {historical_gcs_path}') _HISTORICAL_CACHE[historical_gcs_path] = rows return rows @@ -466,9 +461,13 @@ def _apply_historical_supplement( for r in records: by_key[(r['regionCd'], r['year'], r['indicatorCd'])] = r + norm_target = dataset_filename.replace('jpn_employement.csv', + 'jpn_employment.csv') added_or_updated = 0 for s in hist_rows: - if s['dataset'] != dataset_filename: + norm_src = s['dataset'].replace('jpn_employement.csv', + 'jpn_employment.csv') + if norm_src != norm_target: continue key = (s['regionCd'], s['year'], s['indicatorCd']) if key in by_key: @@ -501,22 +500,24 @@ def _apply_historical_supplement( return list(by_key.values()) -def _write_csv_atomically(output_filepath: str, rows: list[dict[str, str]]) -> None: +def _write_csv_atomically(output_filepath: str, rows: list[dict[str, + str]]) -> None: """Writes rows atomically to output_filepath and verifies non-empty output.""" output_dir = os.path.dirname(output_filepath) os.makedirs(output_dir, exist_ok=True) - fd, tmp_path = tempfile.mkstemp( - prefix='.tmp_jpn_', suffix='.csv', dir=output_dir - ) + fd, tmp_path = tempfile.mkstemp(prefix='.tmp_jpn_', + suffix='.csv', + dir=output_dir) try: with os.fdopen(fd, 'w', encoding='utf-8', newline='') as f: - writer = csv.DictWriter(f, fieldnames=CSV_HEADER, lineterminator='\n') + writer = csv.DictWriter(f, + fieldnames=CSV_HEADER, + lineterminator='\n') writer.writeheader() writer.writerows(rows) if not (os.path.exists(tmp_path) and os.path.getsize(tmp_path) > 0): raise RuntimeError( - f'Atomic write failed: {tmp_path} is empty or missing.' - ) + f'Atomic write failed: {tmp_path} is empty or missing.') os.replace(tmp_path, output_filepath) except Exception: if os.path.exists(tmp_path): @@ -538,11 +539,12 @@ def download_dataset( tasks: list[tuple[list[str], str]] = [] for rank in regional_ranks: for i in range(0, len(indicators), batch_size): - tasks.append((indicators[i : i + batch_size], rank)) + tasks.append((indicators[i:i + batch_size], rank)) all_records: list[dict[str, str]] = [] if max_workers > 1 and len(tasks) > 1: - with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: + with concurrent.futures.ThreadPoolExecutor( + max_workers=max_workers) as executor: futures = [ executor.submit(fetch_indicator_batch, batch, rank) for batch, rank in tasks @@ -555,8 +557,7 @@ def download_dataset( fetch_indicator_batch( indicators=batch, regional_rank=rank, - ) - ) + )) if is_census: all_records = _derive_census_population_rows(all_records, census_mode) @@ -583,15 +584,14 @@ def download_dataset( if os.path.exists(output_filepath): os.remove(output_filepath) for yr, yr_rows in sorted(by_year.items()): - yr_filepath = os.path.join(output_dir, f'jpn_aa1_aa2_age_gender_{yr}.csv') + yr_filepath = os.path.join(output_dir, + f'jpn_aa1_aa2_age_gender_{yr}.csv') _write_csv_atomically(yr_filepath, yr_rows) logging.info('Saved %d rows to %s', len(yr_rows), yr_filepath) return len(all_records) _write_csv_atomically(output_filepath, all_records) - logging.info( - 'Saved %d rows to %s', len(all_records), output_filepath - ) + logging.info('Saved %d rows to %s', len(all_records), output_filepath) return len(all_records) @@ -628,11 +628,8 @@ def download_all( def main(argv: list[str]) -> None: if len(argv) > 1: raise app.UsageError('Too many command-line arguments.') - output_dir = ( - _FLAGS.output_dir - if os.path.isabs(_FLAGS.output_dir) - else os.path.join(_MODULE_DIR, _FLAGS.output_dir) - ) + output_dir = (_FLAGS.output_dir if os.path.isabs(_FLAGS.output_dir) else + os.path.join(_MODULE_DIR, _FLAGS.output_dir)) try: download_all( output_dir, diff --git a/statvar_imports/japan_census/japan_census_pv_map.csv b/statvar_imports/japan_census/japan_census_pv_map.csv index 981a72f168..479cac8bfa 100644 --- a/statvar_imports/japan_census/japan_census_pv_map.csv +++ b/statvar_imports/japan_census/japan_census_pv_map.csv @@ -13,8 +13,8 @@ regionRank,#ignore,regional rank metadata,value,{@Data} isSeasonal,#ignore,seasonal adjustment flag,value,{@Data} isProvisional,#ignore,provisional flag,value,{@Data} -,#IgnoreRow,True,#ignore,suppressed or missing value -0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employmentStatus,Employed,#Multiply,10 -0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employmentStatus,Unemployed,#Multiply,10 +0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employmentStatus,Employed,employment,Employed,#Multiply,10 +0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employmentStatus,Unemployed,employment,Unemployed,#Multiply,10 0301010000030010010,dcid,Count_Person_NotInLaborForce,populationType,Person,measuredProperty,count,employmentStatus,BLS_NotInLaborForce,age,Years16Onwards,#Multiply,10 0301010000020020010,populationType,Person,measuredProperty,unemploymentRate 1201050300000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,PrimarySchool diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index 464aecc231..6d7a33f358 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -37,8 +37,8 @@ import file_util _PROCESSOR_SCRIPT = os.path.abspath( - os.path.join(_MODULE_DIR, '../../tools/statvar_importer/stat_var_processor.py') -) + os.path.join(_MODULE_DIR, + '../../tools/statvar_importer/stat_var_processor.py')) _FLAGS = flags.FLAGS flags.DEFINE_string( @@ -117,49 +117,86 @@ def _run_shard( f'--output_counters={shard_counters_path}', f'--existing_statvar_mcf={existing_statvar_mcf}', ] - logging.info( - '[Shard %02d] Starting %s', shard_idx, os.path.basename(input_file) - ) + logging.info('[Shard %02d] Starting %s', shard_idx, + os.path.basename(input_file)) proc = subprocess.run(cmd, capture_output=True, text=True) if proc.returncode != 0: raise RuntimeError( f'Shard {shard_idx} ({input_file}) failed with exit code {proc.returncode}:\n' - f'{proc.stderr[-2000:]}' - ) - logging.info( - '[Shard %02d] Finished %s', shard_idx, os.path.basename(input_file) - ) + f'{proc.stderr[-2000:]}') + logging.info('[Shard %02d] Finished %s', shard_idx, + os.path.basename(input_file)) return shard_idx, input_file, shard_output_prefix, shard_counters_path +def _atomic_write_text(target_path: str, content: str) -> None: + """Writes text content atomically to target_path via a temp file + os.replace.""" + target_dir = os.path.dirname(os.path.abspath(target_path)) + os.makedirs(target_dir, exist_ok=True) + fd, tmp_path = tempfile.mkstemp(prefix='.tmp_proc_', dir=target_dir) + try: + with os.fdopen(fd, 'w', encoding='utf-8', newline='') as out_f: + out_f.write(content) + os.replace(tmp_path, target_path) + except Exception: + if os.path.exists(tmp_path): + os.remove(tmp_path) + raise + + def _merge_csv_files(shard_prefixes: list[str], final_csv_path: str) -> int: - """Merges shard CSV files into final_csv_path, keeping a single header.""" + """Merges shard CSV files atomically into final_csv_path, keeping a single header.""" total_rows = 0 wrote_header = False - with open(final_csv_path, 'w', encoding='utf-8', newline='') as out_f: - for prefix in shard_prefixes: - shard_csv = f'{prefix}.csv' - if not os.path.exists(shard_csv): - continue - with open(shard_csv, 'r', encoding='utf-8') as in_f: - header = in_f.readline() - if not header: + target_dir = os.path.dirname(os.path.abspath(final_csv_path)) + os.makedirs(target_dir, exist_ok=True) + fd, tmp_path = tempfile.mkstemp(prefix='.tmp_merge_', + suffix='.csv', + dir=target_dir) + try: + with os.fdopen(fd, 'w', encoding='utf-8', newline='') as out_f: + for prefix in shard_prefixes: + shard_csv = f'{prefix}.csv' + if not os.path.exists(shard_csv): continue - if not wrote_header: - out_f.write(header) - wrote_header = True - for line in in_f: - out_f.write(line) - total_rows += 1 + with open(shard_csv, 'r', encoding='utf-8') as in_f: + header = in_f.readline() + if not header: + continue + if not wrote_header: + out_f.write(header) + wrote_header = True + for line in in_f: + out_f.write(line) + total_rows += 1 + os.replace(tmp_path, final_csv_path) + except Exception: + if os.path.exists(tmp_path): + os.remove(tmp_path) + raise return total_rows -def _merge_counters(shard_counter_files: list[str], final_counters_path: str) -> None: +def _counter_aggregation_mode(key: str) -> str: + """Returns how a stat_var_processor counter key should be aggregated across shards.""" + lower = key.lower() + if lower == 'existing-statvars-from-mcf' or lower.startswith('existing-'): + return 'max' + if 'start_time' in lower: + return 'min' + if 'end_time' in lower or 'mem' in lower: + return 'max' + if lower.endswith('_rate') or '_rate_' in lower: + return 'avg' + return 'sum' + + +def _merge_counters(shard_counter_files: list[str], + final_counters_path: str) -> None: """Aggregates numeric counters across all shards into final_counters_path.""" if not final_counters_path: return - os.makedirs(os.path.dirname(os.path.abspath(final_counters_path)), exist_ok=True) - totals: dict[str, float] = defaultdict(float) + values_by_key: dict[str, list[float]] = defaultdict(list) is_int_map: dict[str, bool] = defaultdict(lambda: True) for cf in shard_counter_files: @@ -176,24 +213,37 @@ def _merge_counters(shard_counter_files: list[str], final_counters_path: str) -> k, v_str = parts[0].strip('"'), parts[1].strip() try: if '.' in v_str: - totals[k] += float(v_str) + values_by_key[k].append(float(v_str)) is_int_map[k] = False else: - totals[k] += int(v_str) + values_by_key[k].append(float(int(v_str))) except ValueError: continue - with open(final_counters_path, 'w', encoding='utf-8') as out_f: - out_f.write('key,value\n') - for k in sorted(totals.keys()): - val = totals[k] - if is_int_map[k]: - out_f.write(f'{k},{int(val)}\n') - else: - out_f.write(f'{k},{val:.2f}\n') - - -def _merge_mcf_files(shard_prefixes: list[str], suffix: str, final_mcf_path: str) -> None: + lines = ['key,value\n'] + for k in sorted(values_by_key.keys()): + vals = values_by_key[k] + mode = _counter_aggregation_mode(k) + if mode == 'max': + val = max(vals) + elif mode == 'min': + val = min(vals) + elif mode == 'avg': + val = sum(vals) / len(vals) + is_int_map[k] = False + else: + val = sum(vals) + + if is_int_map[k]: + lines.append(f'{k},{int(val)}\n') + else: + lines.append(f'{k},{val:.2f}\n') + + _atomic_write_text(final_counters_path, ''.join(lines)) + + +def _merge_mcf_files(shard_prefixes: list[str], suffix: str, + final_mcf_path: str) -> None: """Deduplicates MCF nodes by Node line across shards and writes final_mcf_path.""" seen_nodes: dict[str, str] = {} for prefix in shard_prefixes: @@ -206,10 +256,13 @@ def _merge_mcf_files(shard_prefixes: list[str], suffix: str, final_mcf_path: str block = block.strip() if not block: continue - lines = [l for l in block.splitlines() if not l.strip().startswith('#')] + lines = [ + l for l in block.splitlines() if not l.strip().startswith('#') + ] if not lines: continue - node_line = next((l for l in lines if l.startswith('Node:')), lines[0]) + node_line = next((l for l in lines if l.startswith('Node:')), + lines[0]) if node_line not in seen_nodes: seen_nodes[node_line] = '\n'.join(lines) @@ -219,9 +272,9 @@ def _merge_mcf_files(shard_prefixes: list[str], suffix: str, final_mcf_path: str if not seen_nodes: return - with open(final_mcf_path, 'w', encoding='utf-8') as out_f: - for node_key in sorted(seen_nodes.keys()): - out_f.write(seen_nodes[node_key] + '\n\n') + body = ''.join( + seen_nodes[node_key] + '\n\n' for node_key in sorted(seen_nodes.keys())) + _atomic_write_text(final_mcf_path, body) def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: @@ -229,8 +282,8 @@ def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: if not existing_statvar_mcf or not existing_statvar_mcf.startswith('gs://'): return existing_statvar_mcf local_mcf = os.path.join( - tmp_dir, os.path.basename(existing_statvar_mcf) or 'stat_vars.mcf' - ) + tmp_dir, + os.path.basename(existing_statvar_mcf) or 'stat_vars.mcf') logging.info( 'Caching %s once to %s for all shard workers...', existing_statvar_mcf, @@ -239,12 +292,12 @@ def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: file_util.file_copy(existing_statvar_mcf, local_mcf) if not os.path.exists(local_mcf) or os.path.getsize(local_mcf) == 0: raise RuntimeError( - f'Failed to copy existing_statvar_mcf from {existing_statvar_mcf}' - ) + f'Failed to copy existing_statvar_mcf from {existing_statvar_mcf}') return local_mcf -def main(_) -> None: +def _run_pipeline() -> None: + """Executes the parallel sharded stat_var_processor pipeline and merges outputs.""" input_files = _expand_input_files(_FLAGS.input_data) if not input_files: raise RuntimeError(f'No input files matched: {_FLAGS.input_data}') @@ -255,8 +308,8 @@ def main(_) -> None: shards_tmp_dir = tempfile.mkdtemp(prefix='.shards_', dir=output_dir) max_workers = max( - 1, min(_FLAGS.parallelism, len(input_files), os.cpu_count() or 1) - ) + 1, min(_FLAGS.parallelism, len(input_files), + os.cpu_count() or 1)) logging.info( 'Processing %d input files across %d parallel workers...', len(input_files), @@ -265,19 +318,19 @@ def main(_) -> None: try: local_statvar_mcf = _prepare_local_statvar_mcf( - _FLAGS.existing_statvar_mcf, shards_tmp_dir - ) + _FLAGS.existing_statvar_mcf, shards_tmp_dir) shard_prefixes: list[str] = [] shard_counters: list[str] = [] futures = [] with concurrent.futures.ThreadPoolExecutor( - max_workers=max_workers - ) as executor: + max_workers=max_workers) as executor: for idx, input_file in enumerate(input_files): base = os.path.splitext(os.path.basename(input_file))[0] - s_prefix = os.path.join(shards_tmp_dir, f'shard_{idx:02d}_{base}') - s_counters = os.path.join(shards_tmp_dir, f'counters_{idx:02d}_{base}.csv') + s_prefix = os.path.join(shards_tmp_dir, + f'shard_{idx:02d}_{base}') + s_counters = os.path.join(shards_tmp_dir, + f'counters_{idx:02d}_{base}.csv') shard_prefixes.append(s_prefix) shard_counters.append(s_counters) futures.append( @@ -290,8 +343,7 @@ def main(_) -> None: os.path.abspath(_FLAGS.pv_map), os.path.abspath(_FLAGS.config_file), local_statvar_mcf, - ) - ) + )) for fut in concurrent.futures.as_completed(futures): fut.result() @@ -300,9 +352,7 @@ def main(_) -> None: final_csv = f'{output_path}.csv' total_rows = _merge_csv_files(shard_prefixes, final_csv) if total_rows == 0: - raise RuntimeError( - f'Merged output CSV has 0 rows: {final_csv}' - ) + raise RuntimeError(f'Merged output CSV has 0 rows: {final_csv}') logging.info('Merged %d SVObs rows into %s', total_rows, final_csv) # 2. Generate unified TMCF (normalizing table reference to japan_census_output) @@ -314,12 +364,12 @@ def main(_) -> None: shard_table = os.path.basename(prefix) with open(shard_tmcf, 'r', encoding='utf-8') as f: tmcf_text = f.read().replace(shard_table, table_name) - with open(final_tmcf, 'w', encoding='utf-8') as out_f: - out_f.write(tmcf_text) + _atomic_write_text(final_tmcf, tmcf_text) break # 3. Merge StatVar MCF and Schema MCF files - _merge_mcf_files(shard_prefixes, '_stat_vars.mcf', f'{output_path}_stat_vars.mcf') + _merge_mcf_files(shard_prefixes, '_stat_vars.mcf', + f'{output_path}_stat_vars.mcf') _merge_mcf_files( shard_prefixes, '_stat_vars_schema.mcf', @@ -328,7 +378,8 @@ def main(_) -> None: # 4. Merge Counters if _FLAGS.output_counters: - _merge_counters(shard_counters, os.path.abspath(_FLAGS.output_counters)) + _merge_counters(shard_counters, + os.path.abspath(_FLAGS.output_counters)) logging.info('Merged counters into %s', _FLAGS.output_counters) finally: @@ -336,5 +387,12 @@ def main(_) -> None: shutil.rmtree(shards_tmp_dir) +def main(_) -> None: + try: + _run_pipeline() + except Exception as e: + logging.fatal('%s', e, exc_info=True) + + if __name__ == '__main__': app.run(main) diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py index 855fed941a..a1f06107bb 100644 --- a/statvar_imports/japan_census/process_test.py +++ b/statvar_imports/japan_census/process_test.py @@ -58,13 +58,11 @@ def test_merge_csv_files_keeps_single_header(self): with open(f'{p1}.csv', 'w', encoding='utf-8') as f: f.write( 'observationAbout,observationDate,value,variableMeasured\n' - 'dcid:country/JPN,2020,100,dcid:Count_Person\n' - ) + 'dcid:country/JPN,2020,100,dcid:Count_Person\n') with open(f'{p2}.csv', 'w', encoding='utf-8') as f: f.write( 'observationAbout,observationDate,value,variableMeasured\n' - 'dcid:country/JPN,2025,95,dcid:Count_Person\n' - ) + 'dcid:country/JPN,2025,95,dcid:Count_Person\n') final_csv = os.path.join(tmp_dir, 'merged.csv') total_rows = process._merge_csv_files([p1, p2], final_csv) @@ -85,9 +83,23 @@ def test_merge_counters_aggregates_int_and_float(self): c1 = os.path.join(tmp_dir, 'c1.csv') c2 = os.path.join(tmp_dir, 'c2.csv') with open(c1, 'w', encoding='utf-8') as f: - f.write('key,value\nsvobs-added,100\nelapsed_time,1.25\n') + f.write('key,value\n' + 'svobs-added,100\n' + 'elapsed_time,1.25\n' + 'existing-statvars-from-mcf,12896\n' + 'process_mem,1.50\n' + 'processing_rate,500.00\n' + 'start_time,1000\n' + 'end_time,1010\n') with open(c2, 'w', encoding='utf-8') as f: - f.write('key,value\nsvobs-added,250\nelapsed_time,2.50\n') + f.write('key,value\n' + 'svobs-added,250\n' + 'elapsed_time,2.50\n' + 'existing-statvars-from-mcf,12896\n' + 'process_mem,2.25\n' + 'processing_rate,700.00\n' + 'start_time,1005\n' + 'end_time,1025\n') final_counters = os.path.join(tmp_dir, 'merged_counters.csv') process._merge_counters([c1, c2], final_counters) @@ -98,6 +110,11 @@ def test_merge_counters_aggregates_int_and_float(self): [ 'key,value', 'elapsed_time,3.75', + 'end_time,1025', + 'existing-statvars-from-mcf,12896', + 'process_mem,2.25', + 'processing_rate,600.00', + 'start_time,1000', 'svobs-added,350', ], ) @@ -113,12 +130,10 @@ def test_merge_mcf_files_deduplicates_and_skips_empty(self): self.assertFalse(os.path.exists(final_mcf)) # 2. Non-empty case with duplicate node across shards - node_block = ( - '# Auto generated comment\n\n' - 'Node: dcid:Count_Person\n' - 'typeOf: dcid:StatisticalVariable\n' - 'populationType: dcid:Person\n' - ) + node_block = ('# Auto generated comment\n\n' + 'Node: dcid:Count_Person\n' + 'typeOf: dcid:StatisticalVariable\n' + 'populationType: dcid:Person\n') with open(f'{p1}_stat_vars.mcf', 'w', encoding='utf-8') as f: f.write(node_block) with open(f'{p2}_stat_vars.mcf', 'w', encoding='utf-8') as f: @@ -151,6 +166,7 @@ def test_run_shard_raises_on_nonzero_exit(self, mock_run): @mock.patch.object(process.file_util, 'file_copy') def test_prepare_local_statvar_mcf_copies_gcs_path(self, mock_file_copy): with tempfile.TemporaryDirectory() as tmp_dir: + def _fake_copy(src, dst): with open(dst, 'w', encoding='utf-8') as f: f.write('Node: dcid:Count_Person\n') @@ -158,11 +174,20 @@ def _fake_copy(src, dst): mock_file_copy.side_effect = _fake_copy local_path = process._prepare_local_statvar_mcf( - 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', tmp_dir - ) + 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', tmp_dir) self.assertEqual(local_path, os.path.join(tmp_dir, 'stat_vars.mcf')) mock_file_copy.assert_called_once() + @mock.patch.object(process.logging, 'fatal') + @mock.patch.object(process, + '_run_pipeline', + side_effect=RuntimeError('Pipeline failed')) + def test_main_calls_logging_fatal_on_error(self, mock_run_pipeline, + mock_fatal): + process.main([]) + mock_run_pipeline.assert_called_once() + mock_fatal.assert_called_once() + if __name__ == '__main__': unittest.main() diff --git a/statvar_imports/japan_census/validation_config.json b/statvar_imports/japan_census/validation_config.json index 273d0f42e2..f924a0a03f 100644 --- a/statvar_imports/japan_census/validation_config.json +++ b/statvar_imports/japan_census/validation_config.json @@ -26,6 +26,78 @@ "query": "SELECT StatVar, MaxDate, MinValue, MaxValue, COUNT(*) OVER () AS total_svs FROM stats WHERE regexp_matches(StatVar, '^Count_Person_1(00To104|05To109|10OrMore)Years')", "condition": "CAST(MaxDate AS INTEGER) >= 2020 AND total_svs >= 9 AND MinValue >= 0" } + }, + { + "rule_id": "check_max_date_consistent_census", + "description": "Ensure all 75 active Population Census indicators share a consistent MaxDate", + "validator": "MAX_DATE_CONSISTENT", + "scope": { + "variables": { + "regex": [ + "^Count_Person($|_(Male|Female|Upto14|0To4|5To9|10To14|15To19|20To24|25To29|30To34|35To39|40To44|45To49|50To54|55To59|60To64|15To64|65To69|70To74|75To79|80To84|65OrMore|85To89|90To94|95To99|85OrMore))" + ] + } + } + }, + { + "rule_id": "check_max_date_consistent_labor_and_deaths", + "description": "Ensure annual Labor Force Survey and Vital Statistics total death indicators share a consistent MaxDate", + "validator": "MAX_DATE_CONSISTENT", + "scope": { + "variables": { + "regex": [ + "^(Count_(Death|Person_(Employed|Unemployed|NotInLaborForce))|UnemploymentRate_Person)" + ] + } + } + }, + { + "rule_id": "check_max_date_consistent_teachers", + "description": "Ensure School Basic Survey teacher indicators share a consistent MaxDate", + "validator": "MAX_DATE_CONSISTENT", + "scope": { + "variables": { + "regex": [ + "^Count_Teacher_" + ] + } + } + }, + { + "rule_id": "check_max_date_consistent_students", + "description": "Ensure School Basic Survey student enrollment indicators share a consistent MaxDate", + "validator": "MAX_DATE_CONSISTENT", + "scope": { + "variables": { + "regex": [ + "^Count_Student_" + ] + } + } + }, + { + "rule_id": "check_max_date_consistent_causes_of_death", + "description": "Ensure Vital Statistics cause-of-death indicators share a consistent MaxDate", + "validator": "MAX_DATE_CONSISTENT", + "scope": { + "variables": { + "regex": [ + "^Count_MortalityEvent_" + ] + } + } + }, + { + "rule_id": "check_max_date_consistent_historical_centenarians", + "description": "Ensure historical centenarian sub-brackets share a consistent MaxDate", + "validator": "MAX_DATE_CONSISTENT", + "scope": { + "variables": { + "regex": [ + "^Count_Person_1(00To104|05To109|10OrMore)Years" + ] + } + } } ] } \ No newline at end of file From d8c1e750215aad830c629c2024ad2a2e8dd87382 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Fri, 23 Oct 2026 10:34:38 +0000 Subject: [PATCH 07/17] Format download_script_test.py with yapf --- .../japan_census/download_script_test.py | 37 ++++++++++++------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/statvar_imports/japan_census/download_script_test.py b/statvar_imports/japan_census/download_script_test.py index 5835cb9588..4f94388ea8 100644 --- a/statvar_imports/japan_census/download_script_test.py +++ b/statvar_imports/japan_census/download_script_test.py @@ -66,10 +66,17 @@ def setUp(self): }, }, { - '@id': 'time', + '@id': + 'time', 'CLASS': [ - {'@code': '1953CY00', '@name': '1953'}, - {'@code': '1954CY00', '@name': '1954'}, + { + '@code': '1953CY00', + '@name': '1953' + }, + { + '@code': '1954CY00', + '@name': '1954' + }, ], }, ] @@ -116,13 +123,11 @@ def tearDown(self): def test_parse_estat_json_response_success(self): records = download_script.parse_estat_json_response( - self.sample_api_response - ) + self.sample_api_response) self.assertEqual(len(records), 2) self.assertEqual(records[0]['indicatorCd'], '0301010000020020010') - self.assertEqual( - records[0]['indicatorName'], 'Unemployment rate (Both sexes)' - ) + self.assertEqual(records[0]['indicatorName'], + 'Unemployment rate (Both sexes)') self.assertEqual(records[0]['unitName'], '%') self.assertEqual(records[0]['regionCd'], '00000') self.assertEqual(records[0]['regionName'], 'Japan') @@ -148,8 +153,9 @@ def test_write_csv_atomically_failure_cleanup(self): with mock.patch.object(os.path, 'getsize', return_value=0): with self.assertRaises(RuntimeError): download_script._write_csv_atomically( - target_path, [{'year': '2020'}] - ) + target_path, [{ + 'year': '2020' + }]) self.assertFalse(os.path.exists(target_path)) self.assertEqual(os.listdir(tmp_dir), []) @@ -189,7 +195,8 @@ def test_download_dataset_writes_csv_atomically(self, mock_request_json): with open(out_file, 'r', encoding='utf-8') as f: reader = csv.reader(f) header = next(reader) - self.assertEqual(header[:4], ['year', 'indicatorCd', 'regionCd', 'value']) + self.assertEqual(header[:4], + ['year', 'indicatorCd', 'regionCd', 'value']) rows = list(reader) self.assertEqual(len(rows), 2) self.assertEqual(rows[1][0], '1954') @@ -200,7 +207,8 @@ def test_apply_historical_supplement_from_path(self): hist_csv = os.path.join(tmp_dir, 'deleted_historical_data.csv') with open(hist_csv, 'w', encoding='utf-8', newline='') as f: writer = csv.writer(f) - writer.writerow(['dataset', 'year', 'indicatorCd', 'regionCd', 'value']) + writer.writerow( + ['dataset', 'year', 'indicatorCd', 'regionCd', 'value']) writer.writerow([ 'jpn_aa1_gender_age_total_population.csv', '1920', @@ -219,8 +227,9 @@ def test_apply_historical_supplement_from_path(self): self.assertEqual(merged[0]['value'], '2359183') def test_load_historical_rows_missing_file_raises(self): - with self.assertRaises( Exception ): - download_script._load_historical_rows('/nonexistent/deleted_historical_data.csv') + with self.assertRaises(Exception): + download_script._load_historical_rows( + '/nonexistent/deleted_historical_data.csv') if __name__ == '__main__': From edc537f565a029c4d2a8abaf53097194f0c21344 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Fri, 23 Oct 2026 12:50:28 +0000 Subject: [PATCH 08/17] Use canonical BLS_Employed/BLS_Unemployed PVs and clean stale CSVs --- .../japan_census/download_script.py | 22 ++++++++++++++++++ .../japan_census/japan_census_pv_map.csv | 4 ++-- statvar_imports/japan_census/process.py | 23 +++++++++++++++---- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/statvar_imports/japan_census/download_script.py b/statvar_imports/japan_census/download_script.py index 3a58093c2e..408639ab9e 100644 --- a/statvar_imports/japan_census/download_script.py +++ b/statvar_imports/japan_census/download_script.py @@ -595,6 +595,26 @@ def download_dataset( return len(all_records) +def _cleanup_stale_csvs(output_dir: str) -> None: + """Removes any stale CSV files in output_dir not produced by DATASET_CONFIGS.""" + expected_fixed = { + fn for fn, cfg in DATASET_CONFIGS.items() + if cfg.get('census_mode') != 'municipality' + } + for entry in os.listdir(output_dir): + if not entry.endswith('.csv'): + continue + if entry in expected_fixed: + continue + if entry.startswith('jpn_aa1_aa2_age_gender_') and entry[ + len('jpn_aa1_aa2_age_gender_'):-4].isdigit(): + continue + stale_path = os.path.join(output_dir, entry) + if os.path.isfile(stale_path): + logging.info('Removing stale CSV file: %s', stale_path) + os.remove(stale_path) + + def download_all( output_dir: str, max_workers: int = 8, @@ -602,6 +622,7 @@ def download_all( ) -> None: """Downloads all configured Japan Census datasets into output_dir.""" os.makedirs(output_dir, exist_ok=True) + _cleanup_stale_csvs(output_dir) total_rows = 0 for filename, cfg in DATASET_CONFIGS.items(): output_filepath = os.path.join(output_dir, filename) @@ -617,6 +638,7 @@ def download_all( historical_gcs_path=historical_gcs_path, ) total_rows += count + _cleanup_stale_csvs(output_dir) logging.info( 'Successfully downloaded all %d datasets (%d total rows) to %s', len(DATASET_CONFIGS), diff --git a/statvar_imports/japan_census/japan_census_pv_map.csv b/statvar_imports/japan_census/japan_census_pv_map.csv index 479cac8bfa..c8f4ce3424 100644 --- a/statvar_imports/japan_census/japan_census_pv_map.csv +++ b/statvar_imports/japan_census/japan_census_pv_map.csv @@ -13,8 +13,8 @@ regionRank,#ignore,regional rank metadata,value,{@Data} isSeasonal,#ignore,seasonal adjustment flag,value,{@Data} isProvisional,#ignore,provisional flag,value,{@Data} -,#IgnoreRow,True,#ignore,suppressed or missing value -0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employmentStatus,Employed,employment,Employed,#Multiply,10 -0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employmentStatus,Unemployed,employment,Unemployed,#Multiply,10 +0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employment,BLS_Employed,#Multiply,10 +0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employment,BLS_Unemployed,#Multiply,10 0301010000030010010,dcid,Count_Person_NotInLaborForce,populationType,Person,measuredProperty,count,employmentStatus,BLS_NotInLaborForce,age,Years16Onwards,#Multiply,10 0301010000020020010,populationType,Person,measuredProperty,unemploymentRate 1201050300000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,PrimarySchool diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index 6d7a33f358..53cebc5c06 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -145,9 +145,10 @@ def _atomic_write_text(target_path: str, content: str) -> None: def _merge_csv_files(shard_prefixes: list[str], final_csv_path: str) -> int: - """Merges shard CSV files atomically into final_csv_path, keeping a single header.""" + """Merges shard CSV files atomically into final_csv_path, deduplicating rows across shards.""" total_rows = 0 wrote_header = False + seen_lines: set[str] = set() target_dir = os.path.dirname(os.path.abspath(final_csv_path)) os.makedirs(target_dir, exist_ok=True) fd, tmp_path = tempfile.mkstemp(prefix='.tmp_merge_', @@ -167,6 +168,9 @@ def _merge_csv_files(shard_prefixes: list[str], final_csv_path: str) -> int: out_f.write(header) wrote_header = True for line in in_f: + if line in seen_lines: + continue + seen_lines.add(line) out_f.write(line) total_rows += 1 os.replace(tmp_path, final_csv_path) @@ -178,9 +182,16 @@ def _merge_csv_files(shard_prefixes: list[str], final_csv_path: str) -> int: def _counter_aggregation_mode(key: str) -> str: - """Returns how a stat_var_processor counter key should be aggregated across shards.""" + """Returns how a stat_var_processor counter key should be aggregated across shards. + + Note: Per-shard '*-unique-*' cardinality counters (except output-svobs-unique-typeOf, + which is always 1 across shards) are summed across shards because exact set union + cardinality cannot be reconstructed from per-shard integer counts alone. + """ lower = key.lower() - if lower == 'existing-statvars-from-mcf' or lower.startswith('existing-'): + if (lower == 'existing-statvars-from-mcf' or + lower.startswith('existing-') or + lower.endswith('output-svobs-unique-typeof')): return 'max' if 'start_time' in lower: return 'min' @@ -363,9 +374,13 @@ def _run_pipeline() -> None: if os.path.exists(shard_tmcf): shard_table = os.path.basename(prefix) with open(shard_tmcf, 'r', encoding='utf-8') as f: - tmcf_text = f.read().replace(shard_table, table_name) + tmcf_text = f.read().replace(f'C:{shard_table}->', + f'C:{table_name}->') _atomic_write_text(final_tmcf, tmcf_text) break + if not os.path.exists(final_tmcf): + raise RuntimeError( + f'Failed to generate unified TMCF file: {final_tmcf}') # 3. Merge StatVar MCF and Schema MCF files _merge_mcf_files(shard_prefixes, '_stat_vars.mcf', From 4ff6b9ad42b9430efc01138eeeb7863b38e9752f Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Fri, 23 Oct 2026 13:38:07 +0000 Subject: [PATCH 09/17] Normalize both E: and C: table prefixes in unified TMCF --- statvar_imports/japan_census/process.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index 53cebc5c06..a0180893ea 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -374,8 +374,10 @@ def _run_pipeline() -> None: if os.path.exists(shard_tmcf): shard_table = os.path.basename(prefix) with open(shard_tmcf, 'r', encoding='utf-8') as f: - tmcf_text = f.read().replace(f'C:{shard_table}->', - f'C:{table_name}->') + tmcf_text = (f.read().replace(f'E:{shard_table}->', + f'E:{table_name}->').replace( + f'C:{shard_table}->', + f'C:{table_name}->')) _atomic_write_text(final_tmcf, tmcf_text) break if not os.path.exists(final_tmcf): From 63d7ad06b23307c67816ed14bbf2819ef77ffce4 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Sat, 24 Oct 2026 08:25:01 +0000 Subject: [PATCH 10/17] Download raw data as-is without preprocessing in download_script.py --- statvar_imports/japan_census/README.md | 10 +- .../japan_census/download_script.py | 301 +++------------- .../japan_census/download_script_test.py | 35 +- statvar_imports/japan_census/process.py | 324 ++++++++++++++++-- statvar_imports/japan_census/process_test.py | 124 ++++++- 5 files changed, 485 insertions(+), 309 deletions(-) diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md index 747a190706..e2e9232e49 100644 --- a/statvar_imports/japan_census/README.md +++ b/statvar_imports/japan_census/README.md @@ -22,19 +22,19 @@ This project downloads, processes, and imports demographic, employment, educatio ## 2. Preprocessing Steps -The import pipeline queries the e-Stat Statistical Dashboard JSON API in parallel, merges static pre-1975/archival census records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv` so no historical observations are lost, and processes all 18 input files in parallel via `process.py`. +The import pipeline queries the e-Stat Statistical Dashboard JSON API in parallel to download raw datasets as-is, and then `process.py` preprocesses Census broad age sums, merges historical archival records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv`, and runs `stat_var_processor.py` across shards in parallel. * **Files**: - * `download_script.py`: Queries the e-Stat Statistical Dashboard API (`https://dashboard.e-stat.go.jp/api/1.0/Json/getData`), computes exact broad age-group sums (`<15`, `15–64`, `65+`, `85+`) from 5-year census brackets, merges historical records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv` (`--historical_gcs_path`), and writes 18 standardized CSV files into `input_files/`. - * `process.py`: Multi-process wrapper around `stat_var_processor.py` that processes all `input_files/*.csv` shards in parallel (`--parallelism=2`) and deterministically merges the resulting `.csv`, `.tmcf`, `.mcf`, and `counters.csv` outputs into `output_files/`. + * `download_script.py`: Queries the e-Stat Statistical Dashboard API (`https://dashboard.e-stat.go.jp/api/1.0/Json/getData`) and writes the 9 raw CSV dataset files as-is into `input_files/` without preprocessing. + * `process.py`: Preprocesses raw input files (computes exact broad age-group sums for Census datasets, merges historical records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv`, and stages municipality census files by year), runs `stat_var_processor.py` across shards in parallel (`--parallelism=2`), and deterministically merges the resulting `.csv`, `.tmcf`, `.mcf`, and `counters.csv` outputs into `output_files/`. * `japan_census_metadata.csv`: Configuration file for `stat_var_processor.py` specifying output columns, header rows, mapped columns, and provenance URL. * `japan_census_pv_map.csv`: Property-value mapping file mapping e-Stat indicator codes (`indicatorCd`) and 5-digit region codes (`regionCd`) to Data Commons StatisticalVariables and place DCIDs (`country/JPN` and `wikidataId/Q...`). * `validation_config.json`: Configuration defining historical deletion (`<= 0.1%`), value bounds, and date freshness validation rules across all 101 StatisticalVariables. * `test_data/`: Sample input data (`sample_input.csv`) and expected output (`sample_expected_output.csv`) for testing. * **Transformation pipeline**: - 1. `download_script.py` fetches all 9 dataset groups from the e-Stat JSON API using concurrent workers and atomic file writes into `input_files/`. - 2. `process.py` invokes `stat_var_processor.py` in parallel across the 18 input CSVs against `japan_census_pv_map.csv` and `japan_census_metadata.csv`, referencing canonical schema `gs://unresolved_mcf/scripts/statvar/stat_vars.mcf`. + 1. `download_script.py` fetches all 9 raw datasets from the e-Stat JSON API as-is using concurrent workers and atomic file writes into `input_files/`. + 2. `process.py` preprocesses the raw input CSVs in a temporary staging directory (deriving broad age sums for Census datasets, applying historical data supplements, and splitting municipality census data by year) and invokes `stat_var_processor.py` in parallel across the staged input CSVs against `japan_census_pv_map.csv` and `japan_census_metadata.csv`, referencing canonical schema `gs://unresolved_mcf/scripts/statvar/stat_vars.mcf`. 3. The processor filters non-target regional aggregate rows via `#IgnoreRow`, resolves place DCIDs, and produces `output_files/japan_census_output.csv` and `output_files/japan_census_output.tmcf`. To preserve 1:1 historical series continuity (`0` deleted / `0` modified records) with the legacy `Japan_Census` import: * Labor force counts (`0301010000010010010`, `0301010000020010010`, `0301010000030010010`, reported by e-Stat in units of `10,000 persons`) apply `#Multiply,10` to match the historical scaling (in thousands). * Indicator `1201040500000010000` (`Number of upper secondary school teachers (full-time)`) maps to `dcid:Count_Teacher_UniversityAndCollegeTeacher` to match the legacy StatVar assignment across `1975`–`2021`. diff --git a/statvar_imports/japan_census/download_script.py b/statvar_imports/japan_census/download_script.py index 408639ab9e..d27b3f27e9 100644 --- a/statvar_imports/japan_census/download_script.py +++ b/statvar_imports/japan_census/download_script.py @@ -12,7 +12,11 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""Download script for Japan Census and socio-economic datasets from e-Stat Dashboard API.""" +"""Download script for Japan Census and socio-economic datasets from e-Stat Dashboard API. + +Downloads raw dataset files as-is from the official e-Stat Dashboard API into input_files/ +without any preprocessing or business logic derivations. +""" import concurrent.futures import csv @@ -27,12 +31,10 @@ from urllib3.util.retry import Retry _MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) -sys.path.insert(1, os.path.join(_MODULE_DIR, '../../util/')) from absl import app from absl import flags from absl import logging -import file_util _FLAGS = flags.FLAGS flags.DEFINE_string( @@ -45,11 +47,6 @@ 8, 'Maximum number of parallel threads for e-Stat API requests.', ) -flags.DEFINE_string( - 'historical_gcs_path', - 'gs://unresolved_mcf/country/japan/deleted_historical_data.csv', - 'GCS path to the deleted historical data CSV file.', -) ESTAT_JSON_API_URL = 'https://dashboard.e-stat.go.jp/api/1.0/Json/getData' @@ -102,8 +99,8 @@ def _build_population_5yr_indicators( '0301010000030010010', # Not in labour force (Both sexes) ], 'regional_ranks': ['2'], - 'batch_size': 3, - 'is_census': False, + 'batch_size': + 3, }, 'jpn_unemploymentrate.csv': { 'indicators': [ @@ -111,7 +108,6 @@ def _build_population_5yr_indicators( ], 'regional_ranks': ['2'], 'batch_size': 1, - 'is_census': False, }, 'jpn_student.csv': { 'indicators': [ @@ -121,18 +117,18 @@ def _build_population_5yr_indicators( '1201050800000010000', # University students and graduate school students ], 'regional_ranks': ['2'], - 'batch_size': 4, - 'is_census': False, + 'batch_size': + 4, }, 'jpn_aa1_teacher.csv': { 'indicators': [ '1201040100000010000', # Teachers of kindergartens '1201040300000010000', # Teachers of elementary schools - '1201040500000010000', # Teachers of senior high schools (mapped to Count_Teacher_UniversityAndCollegeTeacher) + '1201040500000010000', # Teachers of senior high schools ], 'regional_ranks': ['2', '3'], - 'batch_size': 3, - 'is_census': False, + 'batch_size': + 3, }, 'jpn_aa1_health.csv': { 'indicators': [ @@ -141,8 +137,8 @@ def _build_population_5yr_indicators( '1505130000000010070', # Deaths by causes (suicide) ], 'regional_ranks': ['2', '3'], - 'batch_size': 3, - 'is_census': False, + 'batch_size': + 3, }, 'jpn_aa1_death.csv': { 'indicators': [ @@ -151,29 +147,23 @@ def _build_population_5yr_indicators( '0203020200000010000', # Deaths (Female) ], 'regional_ranks': ['2', '3'], - 'batch_size': 3, - 'is_census': False, + 'batch_size': + 3, }, 'jpn_gender_age_total_population.csv': { 'indicators': _build_population_5yr_indicators(), 'regional_ranks': ['2'], 'batch_size': 5, - 'is_census': True, - 'census_mode': 'national', }, 'jpn_aa1_gender_age_total_population.csv': { 'indicators': _build_population_5yr_indicators(), 'regional_ranks': ['3'], 'batch_size': 5, - 'is_census': True, - 'census_mode': 'prefecture', }, 'jpn_aa1_aa2_age_gender.csv': { 'indicators': _build_population_5yr_indicators(), 'regional_ranks': ['4'], 'batch_size': 3, - 'is_census': True, - 'census_mode': 'municipality', }, } @@ -190,9 +180,8 @@ def _extract_class_lookup(class_objs: list[dict[str, Any]] | dict[str, Any], classes = [classes] return { str(c.get('@code', '')): - str(c.get('@name') if '@name' in c else c.get('$', '')) - for c in classes - if isinstance(c, dict) + str(c.get('@name') if '@name' in c else c.get('$', '')) + for c in classes if isinstance(c, dict) } return {} @@ -247,20 +236,34 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: time_cd[:4] if len(time_cd) >= 4 else time_cd) parsed_records.append({ - 'year': year_str, - 'indicatorCd': ind_cd, - 'regionCd': reg_cd, - 'value': str(val_obj.get('$', '')), - 'indicatorName': indicator_map.get(ind_cd, ''), - 'unitCd': unit_cd, - 'unitName': unit_map.get(unit_cd, ''), - 'statCd': str(val_obj.get('@stat', '')), - 'regionName': region_map.get(reg_cd, ''), - 'timeCd': time_cd, - 'cycle': str(val_obj.get('@cycle', '')), - 'regionRank': str(val_obj.get('@regionRank', '')), - 'isSeasonal': str(val_obj.get('@isSeasonal', '')), - 'isProvisional': str(val_obj.get('@isProvisional', '')), + 'year': + year_str, + 'indicatorCd': + ind_cd, + 'regionCd': + reg_cd, + 'value': + str(val_obj.get('$', '')), + 'indicatorName': + indicator_map.get(ind_cd, ''), + 'unitCd': + unit_cd, + 'unitName': + unit_map.get(unit_cd, ''), + 'statCd': + str(val_obj.get('@stat', '')), + 'regionName': + region_map.get(reg_cd, ''), + 'timeCd': + time_cd, + 'cycle': + str(val_obj.get('@cycle', '')), + 'regionRank': + str(val_obj.get('@regionRank', '')), + 'isSeasonal': + str(val_obj.get('@isSeasonal', '')), + 'isProvisional': + str(val_obj.get('@isProvisional', '')), }) if not parsed_records: @@ -345,161 +348,6 @@ def fetch_indicator_batch( return parse_estat_json_response(response_json) -def _derive_census_population_rows(records: list[dict[str, str]], - mode: str) -> list[dict[str, str]]: - """Filters to 5-year census years and computes exact unrounded broad age sums. - - At National (RegionalRank=2) and Prefecture (RegionalRank=3) levels, direct - broad-age indicators (0201010*100000100*) report rounded annual Population - Estimates (in thousands), whereas summing the 5-year Census age brackets - (0201010*20000010* + 0201010*30000011000) yields the exact unrounded Population - Census counts for <15, 15-64, 65+, and 85+. - """ - census_records = [ - r for r in records if r['year'].isdigit() and int(r['year']) % 5 == 0 - ] - - by_reg_yr_ind: dict[tuple[str, str, str], dict[str, str]] = {} - reg_meta: dict[tuple[str, str], dict[str, str]] = {} - for r in census_records: - key = (r['regionCd'], r['year'], r['indicatorCd']) - by_reg_yr_ind[key] = r - reg_meta[(r['regionCd'], r['year'])] = r - - output_rows: list[dict[str, str]] = [] - # Keep total population (Both sexes, Male, Female) and 5-year brackets: - # National keeps 010..230 (0-4 through 110+); Prefecture keeps 010..170 (0-4 through 80-84); - # Municipality keeps only the 3 totals + 9 broad age sums (<15, 15-64, 65+ for Both/Male/Female = 12 StatVars). - if mode == 'municipality': - allowed_5yr: set[str] = set() - else: - max_5yr_suffix = 230 if mode == 'national' else 170 - allowed_5yr = { - f'0201010{g}20000010{i:03d}' for g in ('0', '1', '2') - for i in range(10, max_5yr_suffix + 10, 10) - } - allowed_totals = { - '0201010000000010000', - '0201010100000010000', - '0201010200000010000', - } - - for r in census_records: - ind = r['indicatorCd'] - if ind in allowed_totals or ind in allowed_5yr: - output_rows.append(r) - - # Derive exact Census counts for <15, 15-64, 65+, and (for National/Prefecture) 85+ from 5-year brackets. - broad_Specs = [ - ('10000010010', 'Population (under 15)', - [f'{i:03d}' for i in (10, 20, 30)], False), - ('10000010020', 'Population (15-64)', - [f'{i:03d}' for i in range(40, 140, 10)], False), - ('10000010030', 'Population (65 and over)', - [f'{i:03d}' for i in range(140, 210, 10)], True), - ] - if mode != 'municipality': - broad_Specs.append(('40000010070', 'Population (85 and over)', - [f'{i:03d}' for i in (180, 190, 200)], True)) - - for reg_cd, yr in sorted(reg_meta.keys()): - base = reg_meta[(reg_cd, yr)] - for g in ('0', '1', '2'): - for target_tail, desc, suffixes, include_100plus in broad_Specs: - part_codes = [f'0201010{g}20000010{s}' for s in suffixes] - if include_100plus: - part_codes.append(f'0201010{g}30000011000') - vals: list[int] = [] - valid = True - for c in part_codes: - rec = by_reg_yr_ind.get((reg_cd, yr, c)) - if not rec or not rec['value'].lstrip('-').isdigit(): - valid = False - break - vals.append(int(rec['value'])) - if valid: - new_row = dict(base) - new_row['indicatorCd'] = f'0201010{g}{target_tail}' - new_row['indicatorName'] = desc - new_row['value'] = str(sum(vals)) - output_rows.append(new_row) - - return output_rows - - -_HISTORICAL_CACHE: dict[str, list[dict[str, str]]] = {} - - -def _load_historical_rows(historical_gcs_path: str) -> list[dict[str, str]]: - """Loads and caches historical rows from GCS or local CSV.""" - if not historical_gcs_path: - return [] - if historical_gcs_path in _HISTORICAL_CACHE: - return _HISTORICAL_CACHE[historical_gcs_path] - logging.info('Reading historical deleted data from GCS: %s', - historical_gcs_path) - with file_util.FileIO(historical_gcs_path, 'r', encoding='utf-8') as f: - rows = list(csv.DictReader(f)) - if not rows: - raise RuntimeError( - f'Historical supplement file is empty: {historical_gcs_path}') - _HISTORICAL_CACHE[historical_gcs_path] = rows - return rows - - -def _apply_historical_supplement( - dataset_filename: str, - records: list[dict[str, str]], - historical_gcs_path: str = '', -) -> list[dict[str, str]]: - """Merges pre-1975 historical Census and exact archival rows from GCS CSV.""" - hist_rows = _load_historical_rows(historical_gcs_path) - if not hist_rows: - return records - - by_key: dict[tuple[str, str, str], dict[str, str]] = {} - for r in records: - by_key[(r['regionCd'], r['year'], r['indicatorCd'])] = r - - norm_target = dataset_filename.replace('jpn_employement.csv', - 'jpn_employment.csv') - added_or_updated = 0 - for s in hist_rows: - norm_src = s['dataset'].replace('jpn_employement.csv', - 'jpn_employment.csv') - if norm_src != norm_target: - continue - key = (s['regionCd'], s['year'], s['indicatorCd']) - if key in by_key: - by_key[key]['value'] = s['value'] - else: - by_key[key] = { - 'year': s['year'], - 'indicatorCd': s['indicatorCd'], - 'regionCd': s['regionCd'], - 'value': s['value'], - 'indicatorName': '', - 'unitCd': '', - 'unitName': '', - 'statCd': '', - 'regionName': '', - 'timeCd': f"{s['year']}CY00", - 'cycle': '3', - 'regionRank': '2' if s['regionCd'] == '00000' else '3', - 'isSeasonal': '1', - 'isProvisional': '0', - } - added_or_updated += 1 - - if added_or_updated: - logging.info( - 'Applied %d historical supplement rows to %s', - added_or_updated, - dataset_filename, - ) - return list(by_key.values()) - - def _write_csv_atomically(output_filepath: str, rows: list[dict[str, str]]) -> None: """Writes rows atomically to output_filepath and verifies non-empty output.""" @@ -530,12 +378,9 @@ def download_dataset( indicators: list[str], regional_ranks: list[str], batch_size: int, - is_census: bool = False, - census_mode: str = '', max_workers: int = 1, - historical_gcs_path: str = '', ) -> int: - """Downloads all batches for a dataset and writes atomically to output_filepath.""" + """Downloads all batches for a dataset and writes as-is atomically to output_filepath.""" tasks: list[tuple[list[str], str]] = [] for rank in regional_ranks: for i in range(0, len(indicators), batch_size): @@ -559,37 +404,11 @@ def download_dataset( regional_rank=rank, )) - if is_census: - all_records = _derive_census_population_rows(all_records, census_mode) - if not all_records: raise RuntimeError( f'No records retrieved for dataset {os.path.basename(output_filepath)}' ) - all_records = _apply_historical_supplement( - os.path.basename(output_filepath), - all_records, - historical_gcs_path=historical_gcs_path, - ) - - if census_mode == 'municipality': - # Split municipality census by year (matching CNS jpn_aa1_aa2_age_gender_{year}.csv) - # so multi-core processing can run across all census years in parallel. - output_dir = os.path.dirname(output_filepath) - by_year: dict[str, list[dict[str, str]]] = {} - for r in all_records: - by_year.setdefault(r['year'], []).append(r) - # Remove any old unsplit jpn_aa1_aa2_age_gender.csv if present - if os.path.exists(output_filepath): - os.remove(output_filepath) - for yr, yr_rows in sorted(by_year.items()): - yr_filepath = os.path.join(output_dir, - f'jpn_aa1_aa2_age_gender_{yr}.csv') - _write_csv_atomically(yr_filepath, yr_rows) - logging.info('Saved %d rows to %s', len(yr_rows), yr_filepath) - return len(all_records) - _write_csv_atomically(output_filepath, all_records) logging.info('Saved %d rows to %s', len(all_records), output_filepath) return len(all_records) @@ -597,30 +416,22 @@ def download_dataset( def _cleanup_stale_csvs(output_dir: str) -> None: """Removes any stale CSV files in output_dir not produced by DATASET_CONFIGS.""" - expected_fixed = { - fn for fn, cfg in DATASET_CONFIGS.items() - if cfg.get('census_mode') != 'municipality' - } + expected_files = set(DATASET_CONFIGS.keys()) for entry in os.listdir(output_dir): if not entry.endswith('.csv'): continue - if entry in expected_fixed: - continue - if entry.startswith('jpn_aa1_aa2_age_gender_') and entry[ - len('jpn_aa1_aa2_age_gender_'):-4].isdigit(): - continue - stale_path = os.path.join(output_dir, entry) - if os.path.isfile(stale_path): - logging.info('Removing stale CSV file: %s', stale_path) - os.remove(stale_path) + if entry not in expected_files: + stale_path = os.path.join(output_dir, entry) + if os.path.isfile(stale_path): + logging.info('Removing stale CSV file: %s', stale_path) + os.remove(stale_path) def download_all( output_dir: str, max_workers: int = 8, - historical_gcs_path: str = '', ) -> None: - """Downloads all configured Japan Census datasets into output_dir.""" + """Downloads all configured Japan Census datasets as-is into output_dir.""" os.makedirs(output_dir, exist_ok=True) _cleanup_stale_csvs(output_dir) total_rows = 0 @@ -632,10 +443,7 @@ def download_all( indicators=cfg['indicators'], regional_ranks=cfg['regional_ranks'], batch_size=cfg['batch_size'], - is_census=cfg.get('is_census', False), - census_mode=cfg.get('census_mode', ''), max_workers=max_workers, - historical_gcs_path=historical_gcs_path, ) total_rows += count _cleanup_stale_csvs(output_dir) @@ -656,7 +464,6 @@ def main(argv: list[str]) -> None: download_all( output_dir, max_workers=_FLAGS.max_workers, - historical_gcs_path=_FLAGS.historical_gcs_path, ) except Exception as e: logging.fatal('%s', e, exc_info=True) diff --git a/statvar_imports/japan_census/download_script_test.py b/statvar_imports/japan_census/download_script_test.py index 4f94388ea8..3536ffd274 100644 --- a/statvar_imports/japan_census/download_script_test.py +++ b/statvar_imports/japan_census/download_script_test.py @@ -67,7 +67,7 @@ def setUp(self): }, { '@id': - 'time', + 'time', 'CLASS': [ { '@code': '1953CY00', @@ -117,10 +117,6 @@ def setUp(self): } } - def tearDown(self): - download_script._HISTORICAL_CACHE.clear() - super().tearDown() - def test_parse_estat_json_response_success(self): records = download_script.parse_estat_json_response( self.sample_api_response) @@ -202,35 +198,6 @@ def test_download_dataset_writes_csv_atomically(self, mock_request_json): self.assertEqual(rows[1][0], '1954') self.assertEqual(rows[1][3], '2.3') - def test_apply_historical_supplement_from_path(self): - with tempfile.TemporaryDirectory() as tmp_dir: - hist_csv = os.path.join(tmp_dir, 'deleted_historical_data.csv') - with open(hist_csv, 'w', encoding='utf-8', newline='') as f: - writer = csv.writer(f) - writer.writerow( - ['dataset', 'year', 'indicatorCd', 'regionCd', 'value']) - writer.writerow([ - 'jpn_aa1_gender_age_total_population.csv', - '1920', - '0201010000000010000', - '01000', - '2359183', - ]) - merged = download_script._apply_historical_supplement( - 'jpn_aa1_gender_age_total_population.csv', - [], - historical_gcs_path=hist_csv, - ) - self.assertEqual(len(merged), 1) - self.assertEqual(merged[0]['year'], '1920') - self.assertEqual(merged[0]['regionCd'], '01000') - self.assertEqual(merged[0]['value'], '2359183') - - def test_load_historical_rows_missing_file_raises(self): - with self.assertRaises(Exception): - download_script._load_historical_rows( - '/nonexistent/deleted_historical_data.csv') - if __name__ == '__main__': unittest.main() diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index a0180893ea..0ff6eab77d 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -12,15 +12,17 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""Parallel wrapper for running stat_var_processor.py across multiple input CSVs. +"""Parallel processor for Japan Census datasets. -Spawns concurrent worker processes (one per input CSV file), merges all output -CSVs, MCFs, and counters into the unified output_path, and preserves full -stat_var_processor CLI flag compatibility. +Preprocesses raw input CSV files (computes exact broad age-group sums for Census +datasets, merges historical records from GCS CSV, and splits municipality census +shards by year), runs stat_var_processor.py concurrently across shards, and merges +output CSV, TMCF, MCFs, and counters into unified output files. """ from collections import defaultdict import concurrent.futures +import csv import glob import os import shutil @@ -71,12 +73,278 @@ 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', 'Path to existing StatVar MCF file.', ) +flags.DEFINE_string( + 'historical_gcs_path', + 'gs://unresolved_mcf/country/japan/deleted_historical_data.csv', + 'GCS path to the deleted historical data CSV file.', +) flags.DEFINE_integer( 'parallelism', 2, 'Maximum number of concurrent stat_var_processor worker processes.', ) +CSV_HEADER = [ + 'year', + 'indicatorCd', + 'regionCd', + 'value', + 'indicatorName', + 'unitCd', + 'unitName', + 'statCd', + 'regionName', + 'timeCd', + 'cycle', + 'regionRank', + 'isSeasonal', + 'isProvisional', +] + + +def _derive_census_population_rows(records: list[dict[str, str]], + mode: str) -> list[dict[str, str]]: + """Filters to 5-year census years and computes exact unrounded broad age sums. + + At National (RegionalRank=2) and Prefecture (RegionalRank=3) levels, direct + broad-age indicators (0201010*100000100*) report rounded annual Population + Estimates (in thousands), whereas summing the 5-year Census age brackets + (0201010*20000010* + 0201010*30000011000) yields the exact unrounded Population + Census counts for <15, 15-64, 65+, and 85+. + """ + census_records = [ + r for r in records if r['year'].isdigit() and int(r['year']) % 5 == 0 + ] + + by_reg_yr_ind: dict[tuple[str, str, str], dict[str, str]] = {} + reg_meta: dict[tuple[str, str], dict[str, str]] = {} + for r in census_records: + key = (r['regionCd'], r['year'], r['indicatorCd']) + by_reg_yr_ind[key] = r + reg_meta[(r['regionCd'], r['year'])] = r + + output_rows: list[dict[str, str]] = [] + # Keep total population (Both sexes, Male, Female) and 5-year brackets: + # National keeps 010..230 (0-4 through 110+); Prefecture keeps 010..170 (0-4 through 80-84); + # Municipality keeps only the 3 totals + 9 broad age sums (<15, 15-64, 65+ for Both/Male/Female = 12 StatVars). + if mode == 'municipality': + allowed_5yr: set[str] = set() + else: + max_5yr_suffix = 230 if mode == 'national' else 170 + allowed_5yr = { + f'0201010{g}20000010{i:03d}' + for g in ('0', '1', '2') + for i in range(10, max_5yr_suffix + 10, 10) + } + allowed_totals = { + '0201010000000010000', + '0201010100000010000', + '0201010200000010000', + } + + for r in census_records: + ind = r['indicatorCd'] + if ind in allowed_totals or ind in allowed_5yr: + output_rows.append(r) + + # Derive exact Census counts for <15, 15-64, 65+, and (for National/Prefecture) 85+ from 5-year brackets. + broad_specs = [ + ('10000010010', 'Population (under 15)', + [f'{i:03d}' for i in (10, 20, 30)], False), + ('10000010020', 'Population (15-64)', + [f'{i:03d}' for i in range(40, 140, 10)], False), + ('10000010030', 'Population (65 and over)', + [f'{i:03d}' for i in range(140, 210, 10)], True), + ] + if mode != 'municipality': + broad_specs.append(('40000010070', 'Population (85 and over)', + [f'{i:03d}' for i in (180, 190, 200)], True)) + + for reg_cd, yr in sorted(reg_meta.keys()): + base = reg_meta[(reg_cd, yr)] + for g in ('0', '1', '2'): + for target_tail, desc, suffixes, include_100plus in broad_specs: + part_codes = [f'0201010{g}20000010{s}' for s in suffixes] + if include_100plus: + part_codes.append(f'0201010{g}30000011000') + vals: list[int] = [] + valid = True + for c in part_codes: + rec = by_reg_yr_ind.get((reg_cd, yr, c)) + if not rec or not rec['value'].lstrip('-').isdigit(): + valid = False + break + vals.append(int(rec['value'])) + if valid: + new_row = dict(base) + new_row['indicatorCd'] = f'0201010{g}{target_tail}' + new_row['indicatorName'] = desc + new_row['value'] = str(sum(vals)) + output_rows.append(new_row) + + return output_rows + + +_HISTORICAL_CACHE: dict[str, list[dict[str, str]]] = {} + + +def _load_historical_rows(historical_gcs_path: str) -> list[dict[str, str]]: + """Loads and caches historical rows from GCS or local CSV.""" + if not historical_gcs_path: + return [] + if historical_gcs_path in _HISTORICAL_CACHE: + return _HISTORICAL_CACHE[historical_gcs_path] + logging.info('Reading historical deleted data: %s', historical_gcs_path) + with file_util.FileIO(historical_gcs_path, 'r', encoding='utf-8') as f: + rows = list(csv.DictReader(f)) + if not rows: + raise RuntimeError( + f'Historical supplement file is empty: {historical_gcs_path}') + _HISTORICAL_CACHE[historical_gcs_path] = rows + return rows + + +def _apply_historical_supplement( + dataset_filename: str, + records: list[dict[str, str]], + historical_gcs_path: str = '', +) -> list[dict[str, str]]: + """Merges pre-1975 historical Census and exact archival rows from GCS CSV.""" + if not historical_gcs_path: + return records + hist_rows = _load_historical_rows(historical_gcs_path) + if not hist_rows: + return records + + by_key: dict[tuple[str, str, str], dict[str, str]] = {} + for r in records: + by_key[(r['regionCd'], r['year'], r['indicatorCd'])] = r + + norm_target = dataset_filename.replace('jpn_employement.csv', + 'jpn_employment.csv') + added_or_updated = 0 + for s in hist_rows: + norm_src = s['dataset'].replace('jpn_employement.csv', + 'jpn_employment.csv') + if norm_src != norm_target: + continue + key = (s['regionCd'], s['year'], s['indicatorCd']) + if key in by_key: + by_key[key]['value'] = s['value'] + else: + by_key[key] = { + 'year': s['year'], + 'indicatorCd': s['indicatorCd'], + 'regionCd': s['regionCd'], + 'value': s['value'], + 'indicatorName': '', + 'unitCd': '', + 'unitName': '', + 'statCd': '', + 'regionName': '', + 'timeCd': f"{s['year']}CY00", + 'cycle': '3', + 'regionRank': '2' if s['regionCd'] == '00000' else '3', + 'isSeasonal': '1', + 'isProvisional': '0', + } + added_or_updated += 1 + + if added_or_updated: + logging.info( + 'Applied %d historical supplement rows to %s', + added_or_updated, + dataset_filename, + ) + return list(by_key.values()) + + +def _write_staged_csv( + filepath: str, + rows: list[dict[str, str]], + fieldnames: list[str], +) -> None: + """Writes preprocessed rows to a staged CSV file.""" + if not rows: + return + all_keys = list(fieldnames) + for r in rows: + for k in r.keys(): + if k not in all_keys: + all_keys.append(k) + with open(filepath, 'w', encoding='utf-8', newline='') as f: + writer = csv.DictWriter(f, fieldnames=all_keys, lineterminator='\n') + writer.writeheader() + writer.writerows(rows) + + +def _prepare_shard_inputs( + input_files: list[str], + staging_dir: str, + historical_gcs_path: str = '', +) -> list[str]: + """Preprocesses raw input files into staged shard CSV files. + + Applies Census broad age derivations, historical data supplement merging, + and municipality year-splitting in the temporary staging directory. + """ + for filepath in input_files: + if not os.path.exists(filepath): + continue + filename = os.path.basename(filepath) + with open(filepath, 'r', encoding='utf-8') as f: + reader = csv.DictReader(f) + rows = list(reader) + fieldnames = list(reader.fieldnames or CSV_HEADER) + + if not rows: + continue + + if filename == 'jpn_gender_age_total_population.csv': + rows = _derive_census_population_rows(rows, mode='national') + rows = _apply_historical_supplement(filename, rows, + historical_gcs_path) + out_file = os.path.join(staging_dir, filename) + _write_staged_csv(out_file, rows, fieldnames) + elif filename == 'jpn_aa1_gender_age_total_population.csv': + rows = _derive_census_population_rows(rows, mode='prefecture') + rows = _apply_historical_supplement(filename, rows, + historical_gcs_path) + out_file = os.path.join(staging_dir, filename) + _write_staged_csv(out_file, rows, fieldnames) + elif filename == 'jpn_aa1_aa2_age_gender.csv': + rows = _derive_census_population_rows(rows, mode='municipality') + rows = _apply_historical_supplement(filename, rows, + historical_gcs_path) + by_year: dict[str, list[dict[str, str]]] = {} + for r in rows: + by_year.setdefault(r['year'], []).append(r) + for yr, yr_rows in sorted(by_year.items()): + yr_file = os.path.join(staging_dir, + f'jpn_aa1_aa2_age_gender_{yr}.csv') + _write_staged_csv(yr_file, yr_rows, fieldnames) + elif (filename.startswith('jpn_aa1_aa2_age_gender_') + and filename.endswith('.csv')): + has_5yr = any('20000010' in r.get('indicatorCd', '') + for r in rows[:100]) + if has_5yr: + rows = _derive_census_population_rows(rows, + mode='municipality') + rows = _apply_historical_supplement(filename, rows, + historical_gcs_path) + out_file = os.path.join(staging_dir, filename) + _write_staged_csv(out_file, rows, fieldnames) + else: + rows = _apply_historical_supplement(filename, rows, + historical_gcs_path) + out_file = os.path.join(staging_dir, filename) + _write_staged_csv(out_file, rows, fieldnames) + + return sorted( + glob.glob(os.path.join(staging_dir, '*.csv')), + key=lambda p: -os.path.getsize(p), + ) + def _expand_input_files(input_data: str) -> list[str]: """Expands comma-separated paths or glob patterns into sorted file paths.""" @@ -93,7 +361,6 @@ def _expand_input_files(input_data: str) -> list[str]: files.extend(matched) elif os.path.exists(pattern): files.append(pattern) - # Sort largest files first so long-running shards start immediately. return sorted(set(files), key=lambda p: -os.path.getsize(p)) @@ -189,9 +456,8 @@ def _counter_aggregation_mode(key: str) -> str: cardinality cannot be reconstructed from per-shard integer counts alone. """ lower = key.lower() - if (lower == 'existing-statvars-from-mcf' or - lower.startswith('existing-') or - lower.endswith('output-svobs-unique-typeof')): + if (lower == 'existing-statvars-from-mcf' or lower.startswith('existing-') + or lower.endswith('output-svobs-unique-typeof')): return 'max' if 'start_time' in lower: return 'min' @@ -283,14 +549,15 @@ def _merge_mcf_files(shard_prefixes: list[str], suffix: str, if not seen_nodes: return - body = ''.join( - seen_nodes[node_key] + '\n\n' for node_key in sorted(seen_nodes.keys())) + body = ''.join(seen_nodes[node_key] + '\n\n' + for node_key in sorted(seen_nodes.keys())) _atomic_write_text(final_mcf_path, body) def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: """Copies a gs:// existing_statvar_mcf file once to tmp_dir for all shards.""" - if not existing_statvar_mcf or not existing_statvar_mcf.startswith('gs://'): + if not existing_statvar_mcf or not existing_statvar_mcf.startswith( + 'gs://'): return existing_statvar_mcf local_mcf = os.path.join( tmp_dir, @@ -309,8 +576,8 @@ def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: def _run_pipeline() -> None: """Executes the parallel sharded stat_var_processor pipeline and merges outputs.""" - input_files = _expand_input_files(_FLAGS.input_data) - if not input_files: + raw_input_files = _expand_input_files(_FLAGS.input_data) + if not raw_input_files: raise RuntimeError(f'No input files matched: {_FLAGS.input_data}') output_path = os.path.abspath(_FLAGS.output_path) @@ -318,16 +585,29 @@ def _run_pipeline() -> None: os.makedirs(output_dir, exist_ok=True) shards_tmp_dir = tempfile.mkdtemp(prefix='.shards_', dir=output_dir) - max_workers = max( - 1, min(_FLAGS.parallelism, len(input_files), - os.cpu_count() or 1)) - logging.info( - 'Processing %d input files across %d parallel workers...', - len(input_files), - max_workers, - ) try: + staging_dir = os.path.join(shards_tmp_dir, 'staged_inputs') + os.makedirs(staging_dir, exist_ok=True) + shard_input_files = _prepare_shard_inputs( + raw_input_files, + staging_dir=staging_dir, + historical_gcs_path=_FLAGS.historical_gcs_path, + ) + if not shard_input_files: + raise RuntimeError( + f'No staged input files generated from: {raw_input_files}') + + max_workers = max( + 1, + min(_FLAGS.parallelism, len(shard_input_files), + os.cpu_count() or 1)) + logging.info( + 'Processing %d staged input files across %d parallel workers...', + len(shard_input_files), + max_workers, + ) + local_statvar_mcf = _prepare_local_statvar_mcf( _FLAGS.existing_statvar_mcf, shards_tmp_dir) shard_prefixes: list[str] = [] @@ -336,7 +616,7 @@ def _run_pipeline() -> None: with concurrent.futures.ThreadPoolExecutor( max_workers=max_workers) as executor: - for idx, input_file in enumerate(input_files): + for idx, input_file in enumerate(shard_input_files): base = os.path.splitext(os.path.basename(input_file))[0] s_prefix = os.path.join(shards_tmp_dir, f'shard_{idx:02d}_{base}') diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py index a1f06107bb..a1fcb15592 100644 --- a/statvar_imports/japan_census/process_test.py +++ b/statvar_imports/japan_census/process_test.py @@ -14,6 +14,7 @@ # limitations under the License. """Unit tests for Japan Census process.py.""" +import csv import os import sys import tempfile @@ -32,6 +33,10 @@ class ProcessTest(unittest.TestCase): + def tearDown(self): + process._HISTORICAL_CACHE.clear() + super().tearDown() + def test_expand_input_files_sorts_largest_first(self): with tempfile.TemporaryDirectory() as tmp_dir: small_file = os.path.join(tmp_dir, 'a_small.csv') @@ -175,9 +180,126 @@ def _fake_copy(src, dst): mock_file_copy.side_effect = _fake_copy local_path = process._prepare_local_statvar_mcf( 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', tmp_dir) - self.assertEqual(local_path, os.path.join(tmp_dir, 'stat_vars.mcf')) + self.assertEqual(local_path, os.path.join(tmp_dir, + 'stat_vars.mcf')) mock_file_copy.assert_called_once() + def test_apply_historical_supplement_from_path(self): + with tempfile.TemporaryDirectory() as tmp_dir: + hist_csv = os.path.join(tmp_dir, 'deleted_historical_data.csv') + with open(hist_csv, 'w', encoding='utf-8', newline='') as f: + writer = csv.writer(f) + writer.writerow( + ['dataset', 'year', 'indicatorCd', 'regionCd', 'value']) + writer.writerow([ + 'jpn_aa1_gender_age_total_population.csv', + '1920', + '0201010000000010000', + '01000', + '2359183', + ]) + merged = process._apply_historical_supplement( + 'jpn_aa1_gender_age_total_population.csv', + [], + historical_gcs_path=hist_csv, + ) + self.assertEqual(len(merged), 1) + self.assertEqual(merged[0]['year'], '1920') + self.assertEqual(merged[0]['regionCd'], '01000') + self.assertEqual(merged[0]['value'], '2359183') + + def test_load_historical_rows_missing_file_raises(self): + with self.assertRaises(Exception): + process._load_historical_rows( + '/nonexistent/deleted_historical_data.csv') + + def test_derive_census_population_rows(self): + sample_records = [ + # 2020 (Census year) Total population + { + 'year': '2020', + 'regionCd': '00000', + 'indicatorCd': '0201010000000010000', + 'value': '126146099', + }, + # 2020 0-4, 5-9, 10-14 (<15 bracket) + { + 'year': '2020', + 'regionCd': '00000', + 'indicatorCd': '0201010020000010010', + 'value': '4830000', + }, + { + 'year': '2020', + 'regionCd': '00000', + 'indicatorCd': '0201010020000010020', + 'value': '5130000', + }, + { + 'year': '2020', + 'regionCd': '00000', + 'indicatorCd': '0201010020000010030', + 'value': '5360000', + }, + # 2021 (non-census year) + { + 'year': '2021', + 'regionCd': '00000', + 'indicatorCd': '0201010000000010000', + 'value': '125502000', + }, + ] + derived = process._derive_census_population_rows(sample_records, + mode='national') + # Non-census year 2021 should be filtered out + years = {r['year'] for r in derived} + self.assertEqual(years, {'2020'}) + # Should contain total and derived <15 sum (4830000 + 5130000 + 5360000 = 15320000) + under_15 = [ + r for r in derived if r['indicatorCd'] == '0201010010000010010' + ] + self.assertEqual(len(under_15), 1) + self.assertEqual(under_15[0]['value'], '15320000') + + def test_prepare_shard_inputs_stages_files(self): + with tempfile.TemporaryDirectory() as tmp_dir: + input_dir = os.path.join(tmp_dir, 'inputs') + staging_dir = os.path.join(tmp_dir, 'staged') + os.makedirs(input_dir, exist_ok=True) + os.makedirs(staging_dir, exist_ok=True) + + muni_file = os.path.join(input_dir, 'jpn_aa1_aa2_age_gender.csv') + with open(muni_file, 'w', encoding='utf-8', newline='') as f: + writer = csv.DictWriter(f, fieldnames=process.CSV_HEADER) + writer.writeheader() + writer.writerow({ + 'year': '2020', + 'indicatorCd': '0201010000000010000', + 'regionCd': '01100', + 'value': '1961996', + }) + writer.writerow({ + 'year': '2015', + 'indicatorCd': '0201010000000010000', + 'regionCd': '01100', + 'value': '1952356', + }) + + staged = process._prepare_shard_inputs( + [muni_file], + staging_dir=staging_dir, + ) + # Should have split into two year shards: 2015 and 2020 + self.assertEqual(len(staged), 2) + staged_names = {os.path.basename(p) for p in staged} + self.assertEqual( + staged_names, + { + 'jpn_aa1_aa2_age_gender_2015.csv', + 'jpn_aa1_aa2_age_gender_2020.csv' + }, + ) + @mock.patch.object(process.logging, 'fatal') @mock.patch.object(process, '_run_pipeline', From 3e184d24f086e24f690a172b07f4590cbb63a69c Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Sat, 24 Oct 2026 19:04:37 +0000 Subject: [PATCH 11/17] Keep live API values when applying historical supplement in process.py --- statvar_imports/japan_census/process.py | 45 ++++++++++---------- statvar_imports/japan_census/process_test.py | 24 ++++++++--- 2 files changed, 41 insertions(+), 28 deletions(-) diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index 0ff6eab77d..f4388da427 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -222,7 +222,7 @@ def _apply_historical_supplement( norm_target = dataset_filename.replace('jpn_employement.csv', 'jpn_employment.csv') - added_or_updated = 0 + added = 0 for s in hist_rows: norm_src = s['dataset'].replace('jpn_employement.csv', 'jpn_employment.csv') @@ -230,30 +230,29 @@ def _apply_historical_supplement( continue key = (s['regionCd'], s['year'], s['indicatorCd']) if key in by_key: - by_key[key]['value'] = s['value'] - else: - by_key[key] = { - 'year': s['year'], - 'indicatorCd': s['indicatorCd'], - 'regionCd': s['regionCd'], - 'value': s['value'], - 'indicatorName': '', - 'unitCd': '', - 'unitName': '', - 'statCd': '', - 'regionName': '', - 'timeCd': f"{s['year']}CY00", - 'cycle': '3', - 'regionRank': '2' if s['regionCd'] == '00000' else '3', - 'isSeasonal': '1', - 'isProvisional': '0', - } - added_or_updated += 1 - - if added_or_updated: + continue + by_key[key] = { + 'year': s['year'], + 'indicatorCd': s['indicatorCd'], + 'regionCd': s['regionCd'], + 'value': s['value'], + 'indicatorName': '', + 'unitCd': '', + 'unitName': '', + 'statCd': '', + 'regionName': '', + 'timeCd': f"{s['year']}CY00", + 'cycle': '3', + 'regionRank': '2' if s['regionCd'] == '00000' else '3', + 'isSeasonal': '1', + 'isProvisional': '0', + } + added += 1 + + if added: logging.info( 'Applied %d historical supplement rows to %s', - added_or_updated, + added, dataset_filename, ) return list(by_key.values()) diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py index a1fcb15592..84a657b290 100644 --- a/statvar_imports/japan_census/process_test.py +++ b/statvar_imports/japan_census/process_test.py @@ -198,15 +198,29 @@ def test_apply_historical_supplement_from_path(self): '01000', '2359183', ]) + writer.writerow([ + 'jpn_aa1_gender_age_total_population.csv', + '2020', + '0201010000000010000', + '01000', + '9999999', + ]) + live_rows = [{ + 'year': '2020', + 'indicatorCd': '0201010000000010000', + 'regionCd': '01000', + 'value': '5224614', + }] merged = process._apply_historical_supplement( 'jpn_aa1_gender_age_total_population.csv', - [], + live_rows, historical_gcs_path=hist_csv, ) - self.assertEqual(len(merged), 1) - self.assertEqual(merged[0]['year'], '1920') - self.assertEqual(merged[0]['regionCd'], '01000') - self.assertEqual(merged[0]['value'], '2359183') + by_yr = {r['year']: r for r in merged} + self.assertEqual(len(by_yr), 2) + self.assertEqual(by_yr['1920']['value'], '2359183') + # Live value must be preserved over historical supplement value + self.assertEqual(by_yr['2020']['value'], '5224614') def test_load_historical_rows_missing_file_raises(self): with self.assertRaises(Exception): From 9ef5b2e7b016239a115ee7b61ed315e094e7f976 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Mon, 26 Oct 2026 07:39:51 +0000 Subject: [PATCH 12/17] Update cron_schedule to weekly (0 5 * * 1) in manifest.json and README.md --- statvar_imports/japan_census/README.md | 4 ++-- statvar_imports/japan_census/manifest.json | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md index e2e9232e49..86cad01d74 100644 --- a/statvar_imports/japan_census/README.md +++ b/statvar_imports/japan_census/README.md @@ -51,10 +51,10 @@ The import pipeline queries the e-Stat Statistical Dashboard JSON API in paralle ## 3. Automated Import -This import is configured for automated monthly execution in `manifest.json`. +This import is configured for automated weekly execution in `manifest.json`. ### Automated Steps -1. The automated job triggers on cron schedule `0 5 1 * *` configured in `manifest.json`. +1. The automated job triggers on cron schedule `0 5 * * 1` configured in `manifest.json`. 2. `download_script.py` queries the e-Stat Dashboard API and populates `input_files/*.csv`. 3. `process.py` runs `stat_var_processor.py` with `--parallelism=2` to generate `output_files/japan_census_output.csv` and `output_files/japan_census_output.tmcf`. 4. Validation rules in `validation_config.json` are evaluated before publication. diff --git a/statvar_imports/japan_census/manifest.json b/statvar_imports/japan_census/manifest.json index 323b5acaeb..823963afe7 100644 --- a/statvar_imports/japan_census/manifest.json +++ b/statvar_imports/japan_census/manifest.json @@ -23,7 +23,7 @@ "counters/*.csv", "validation_config.json" ], - "cron_schedule": "0 5 1 * *", + "cron_schedule": "0 5 * * 1", "validation_config_file": "validation_config.json", "resource_limits": { "cpu": 8, From 6b1cb3f7235afba50498ae54f7c1717e33d2d949 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Mon, 26 Oct 2026 12:36:16 +0000 Subject: [PATCH 13/17] Address code review feedback for Japan_Census import --- statvar_imports/japan_census/README.md | 7 ++- .../japan_census/download_script.py | 62 +++++++------------ .../japan_census/download_script_test.py | 2 +- .../japan_census/japan_census_pv_map.csv | 4 +- statvar_imports/japan_census/process.py | 44 ++++++++----- statvar_imports/japan_census/process_test.py | 12 +++- 6 files changed, 69 insertions(+), 62 deletions(-) diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md index 86cad01d74..c41759c2bf 100644 --- a/statvar_imports/japan_census/README.md +++ b/statvar_imports/japan_census/README.md @@ -84,14 +84,15 @@ python3 process.py \ --parallelism=2 ``` -### Run Sample Integration Test +### Run Sample Integration Test (Offline) ```bash -python3 ../../tools/statvar_importer/stat_var_processor.py \ +python3 process.py \ --input_data=test_data/sample_input.csv \ --pv_map=japan_census_pv_map.csv \ --config_file=japan_census_metadata.csv \ - --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \ + --historical_gcs_path="" \ + --existing_statvar_mcf="" \ --output_path=test_data/sample_expected_output ``` diff --git a/statvar_imports/japan_census/download_script.py b/statvar_imports/japan_census/download_script.py index d27b3f27e9..75611229be 100644 --- a/statvar_imports/japan_census/download_script.py +++ b/statvar_imports/japan_census/download_script.py @@ -99,8 +99,7 @@ def _build_population_5yr_indicators( '0301010000030010010', # Not in labour force (Both sexes) ], 'regional_ranks': ['2'], - 'batch_size': - 3, + 'batch_size': 3, }, 'jpn_unemploymentrate.csv': { 'indicators': [ @@ -117,8 +116,7 @@ def _build_population_5yr_indicators( '1201050800000010000', # University students and graduate school students ], 'regional_ranks': ['2'], - 'batch_size': - 4, + 'batch_size': 4, }, 'jpn_aa1_teacher.csv': { 'indicators': [ @@ -127,8 +125,7 @@ def _build_population_5yr_indicators( '1201040500000010000', # Teachers of senior high schools ], 'regional_ranks': ['2', '3'], - 'batch_size': - 3, + 'batch_size': 3, }, 'jpn_aa1_health.csv': { 'indicators': [ @@ -137,8 +134,7 @@ def _build_population_5yr_indicators( '1505130000000010070', # Deaths by causes (suicide) ], 'regional_ranks': ['2', '3'], - 'batch_size': - 3, + 'batch_size': 3, }, 'jpn_aa1_death.csv': { 'indicators': [ @@ -147,8 +143,7 @@ def _build_population_5yr_indicators( '0203020200000010000', # Deaths (Female) ], 'regional_ranks': ['2', '3'], - 'batch_size': - 3, + 'batch_size': 3, }, 'jpn_gender_age_total_population.csv': { 'indicators': _build_population_5yr_indicators(), @@ -180,8 +175,9 @@ def _extract_class_lookup(class_objs: list[dict[str, Any]] | dict[str, Any], classes = [classes] return { str(c.get('@code', '')): - str(c.get('@name') if '@name' in c else c.get('$', '')) - for c in classes if isinstance(c, dict) + str(c.get('@name') if '@name' in c else c.get('$', '')) + for c in classes + if isinstance(c, dict) } return {} @@ -236,34 +232,20 @@ def parse_estat_json_response(data: dict[str, Any]) -> list[dict[str, str]]: time_cd[:4] if len(time_cd) >= 4 else time_cd) parsed_records.append({ - 'year': - year_str, - 'indicatorCd': - ind_cd, - 'regionCd': - reg_cd, - 'value': - str(val_obj.get('$', '')), - 'indicatorName': - indicator_map.get(ind_cd, ''), - 'unitCd': - unit_cd, - 'unitName': - unit_map.get(unit_cd, ''), - 'statCd': - str(val_obj.get('@stat', '')), - 'regionName': - region_map.get(reg_cd, ''), - 'timeCd': - time_cd, - 'cycle': - str(val_obj.get('@cycle', '')), - 'regionRank': - str(val_obj.get('@regionRank', '')), - 'isSeasonal': - str(val_obj.get('@isSeasonal', '')), - 'isProvisional': - str(val_obj.get('@isProvisional', '')), + 'year': year_str, + 'indicatorCd': ind_cd, + 'regionCd': reg_cd, + 'value': str(val_obj.get('$', '')), + 'indicatorName': indicator_map.get(ind_cd, ''), + 'unitCd': unit_cd, + 'unitName': unit_map.get(unit_cd, ''), + 'statCd': str(val_obj.get('@stat', '')), + 'regionName': region_map.get(reg_cd, ''), + 'timeCd': time_cd, + 'cycle': str(val_obj.get('@cycle', '')), + 'regionRank': str(val_obj.get('@regionRank', '')), + 'isSeasonal': str(val_obj.get('@isSeasonal', '')), + 'isProvisional': str(val_obj.get('@isProvisional', '')), }) if not parsed_records: diff --git a/statvar_imports/japan_census/download_script_test.py b/statvar_imports/japan_census/download_script_test.py index 3536ffd274..7fc2a1b143 100644 --- a/statvar_imports/japan_census/download_script_test.py +++ b/statvar_imports/japan_census/download_script_test.py @@ -67,7 +67,7 @@ def setUp(self): }, { '@id': - 'time', + 'time', 'CLASS': [ { '@code': '1953CY00', diff --git a/statvar_imports/japan_census/japan_census_pv_map.csv b/statvar_imports/japan_census/japan_census_pv_map.csv index c8f4ce3424..68abda48de 100644 --- a/statvar_imports/japan_census/japan_census_pv_map.csv +++ b/statvar_imports/japan_census/japan_census_pv_map.csv @@ -13,8 +13,8 @@ regionRank,#ignore,regional rank metadata,value,{@Data} isSeasonal,#ignore,seasonal adjustment flag,value,{@Data} isProvisional,#ignore,provisional flag,value,{@Data} -,#IgnoreRow,True,#ignore,suppressed or missing value -0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employment,BLS_Employed,#Multiply,10 -0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employment,BLS_Unemployed,#Multiply,10 +0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employment,BLS_Employed,employmentStatus,BLS_Employed,#Multiply,10 +0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employment,BLS_Unemployed,employmentStatus,BLS_Unemployed,#Multiply,10 0301010000030010010,dcid,Count_Person_NotInLaborForce,populationType,Person,measuredProperty,count,employmentStatus,BLS_NotInLaborForce,age,Years16Onwards,#Multiply,10 0301010000020020010,populationType,Person,measuredProperty,unemploymentRate 1201050300000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,PrimarySchool diff --git a/statvar_imports/japan_census/process.py b/statvar_imports/japan_census/process.py index f4388da427..4f3f3860af 100644 --- a/statvar_imports/japan_census/process.py +++ b/statvar_imports/japan_census/process.py @@ -132,8 +132,7 @@ def _derive_census_population_rows(records: list[dict[str, str]], else: max_5yr_suffix = 230 if mode == 'national' else 170 allowed_5yr = { - f'0201010{g}20000010{i:03d}' - for g in ('0', '1', '2') + f'0201010{g}20000010{i:03d}' for g in ('0', '1', '2') for i in range(10, max_5yr_suffix + 10, 10) } allowed_totals = { @@ -204,6 +203,20 @@ def _load_historical_rows(historical_gcs_path: str) -> list[dict[str, str]]: return rows +_HISTORICAL_SUPPLEMENT_DATASETS = frozenset({ + 'jpn_gender_age_total_population.csv', + 'jpn_aa1_gender_age_total_population.csv', + 'jpn_aa1_aa2_age_gender.csv', + 'jpn_employment.csv', + 'jpn_employement.csv', + 'jpn_unemploymentrate.csv', + 'jpn_student.csv', + 'jpn_aa1_teacher.csv', + 'jpn_aa1_death.csv', + 'jpn_aa1_health.csv', +}) + + def _apply_historical_supplement( dataset_filename: str, records: list[dict[str, str]], @@ -212,6 +225,11 @@ def _apply_historical_supplement( """Merges pre-1975 historical Census and exact archival rows from GCS CSV.""" if not historical_gcs_path: return records + is_aa2_shard = (dataset_filename.startswith('jpn_aa1_aa2_age_gender_') and + dataset_filename.endswith('.csv')) + if (dataset_filename not in _HISTORICAL_SUPPLEMENT_DATASETS and + not is_aa2_shard): + return records hist_rows = _load_historical_rows(historical_gcs_path) if not hist_rows: return records @@ -322,13 +340,11 @@ def _prepare_shard_inputs( yr_file = os.path.join(staging_dir, f'jpn_aa1_aa2_age_gender_{yr}.csv') _write_staged_csv(yr_file, yr_rows, fieldnames) - elif (filename.startswith('jpn_aa1_aa2_age_gender_') - and filename.endswith('.csv')): - has_5yr = any('20000010' in r.get('indicatorCd', '') - for r in rows[:100]) + elif (filename.startswith('jpn_aa1_aa2_age_gender_') and + filename.endswith('.csv')): + has_5yr = any('20000010' in r.get('indicatorCd', '') for r in rows) if has_5yr: - rows = _derive_census_population_rows(rows, - mode='municipality') + rows = _derive_census_population_rows(rows, mode='municipality') rows = _apply_historical_supplement(filename, rows, historical_gcs_path) out_file = os.path.join(staging_dir, filename) @@ -455,8 +471,9 @@ def _counter_aggregation_mode(key: str) -> str: cardinality cannot be reconstructed from per-shard integer counts alone. """ lower = key.lower() - if (lower == 'existing-statvars-from-mcf' or lower.startswith('existing-') - or lower.endswith('output-svobs-unique-typeof')): + if (lower == 'existing-statvars-from-mcf' or + lower.startswith('existing-') or + lower.endswith('output-svobs-unique-typeof')): return 'max' if 'start_time' in lower: return 'min' @@ -548,15 +565,14 @@ def _merge_mcf_files(shard_prefixes: list[str], suffix: str, if not seen_nodes: return - body = ''.join(seen_nodes[node_key] + '\n\n' - for node_key in sorted(seen_nodes.keys())) + body = ''.join( + seen_nodes[node_key] + '\n\n' for node_key in sorted(seen_nodes.keys())) _atomic_write_text(final_mcf_path, body) def _prepare_local_statvar_mcf(existing_statvar_mcf: str, tmp_dir: str) -> str: """Copies a gs:// existing_statvar_mcf file once to tmp_dir for all shards.""" - if not existing_statvar_mcf or not existing_statvar_mcf.startswith( - 'gs://'): + if not existing_statvar_mcf or not existing_statvar_mcf.startswith('gs://'): return existing_statvar_mcf local_mcf = os.path.join( tmp_dir, diff --git a/statvar_imports/japan_census/process_test.py b/statvar_imports/japan_census/process_test.py index 84a657b290..98ccd2c3a9 100644 --- a/statvar_imports/japan_census/process_test.py +++ b/statvar_imports/japan_census/process_test.py @@ -180,8 +180,7 @@ def _fake_copy(src, dst): mock_file_copy.side_effect = _fake_copy local_path = process._prepare_local_statvar_mcf( 'gs://unresolved_mcf/scripts/statvar/stat_vars.mcf', tmp_dir) - self.assertEqual(local_path, os.path.join(tmp_dir, - 'stat_vars.mcf')) + self.assertEqual(local_path, os.path.join(tmp_dir, 'stat_vars.mcf')) mock_file_copy.assert_called_once() def test_apply_historical_supplement_from_path(self): @@ -227,6 +226,15 @@ def test_load_historical_rows_missing_file_raises(self): process._load_historical_rows( '/nonexistent/deleted_historical_data.csv') + def test_apply_historical_supplement_skips_unknown_dataset(self): + rows = [{'year': '2020', 'indicatorCd': '0201010000000010000'}] + result = process._apply_historical_supplement( + 'sample_input.csv', + rows, + historical_gcs_path='gs://nonexistent/deleted_historical_data.csv', + ) + self.assertEqual(result, rows) + def test_derive_census_population_rows(self): sample_records = [ # 2020 (Census year) Total population From 788b0aaa46869b8ded3263f26d4211d932537dd5 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Tue, 27 Oct 2026 08:09:30 +0000 Subject: [PATCH 14/17] Keep employment PV mapping aligned with stat_vars.mcf to prevent duplicate statvar MCF output --- statvar_imports/japan_census/japan_census_pv_map.csv | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/statvar_imports/japan_census/japan_census_pv_map.csv b/statvar_imports/japan_census/japan_census_pv_map.csv index 68abda48de..c8f4ce3424 100644 --- a/statvar_imports/japan_census/japan_census_pv_map.csv +++ b/statvar_imports/japan_census/japan_census_pv_map.csv @@ -13,8 +13,8 @@ regionRank,#ignore,regional rank metadata,value,{@Data} isSeasonal,#ignore,seasonal adjustment flag,value,{@Data} isProvisional,#ignore,provisional flag,value,{@Data} -,#IgnoreRow,True,#ignore,suppressed or missing value -0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employment,BLS_Employed,employmentStatus,BLS_Employed,#Multiply,10 -0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employment,BLS_Unemployed,employmentStatus,BLS_Unemployed,#Multiply,10 +0301010000010010010,dcid,Count_Person_Employed,populationType,Person,measuredProperty,count,employment,BLS_Employed,#Multiply,10 +0301010000020010010,dcid,Count_Person_Unemployed,populationType,Person,measuredProperty,count,employment,BLS_Unemployed,#Multiply,10 0301010000030010010,dcid,Count_Person_NotInLaborForce,populationType,Person,measuredProperty,count,employmentStatus,BLS_NotInLaborForce,age,Years16Onwards,#Multiply,10 0301010000020020010,populationType,Person,measuredProperty,unemploymentRate 1201050300000010000,populationType,Student,measuredProperty,count,schoolGradeLevel,PrimarySchool From 71923a0553ebba8f7c9a2e80f8db93cb22ffa34f Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Sat, 31 Oct 2026 12:03:52 +0000 Subject: [PATCH 15/17] Add __init__.py, sample_expected_output.tmcf, and remove default rule in validation_config.json --- statvar_imports/japan_census/__init__.py | 13 +++++++++++++ .../test_data/sample_expected_output.tmcf | 6 ++++++ statvar_imports/japan_census/validation_config.json | 8 -------- 3 files changed, 19 insertions(+), 8 deletions(-) create mode 100644 statvar_imports/japan_census/__init__.py create mode 100644 statvar_imports/japan_census/test_data/sample_expected_output.tmcf diff --git a/statvar_imports/japan_census/__init__.py b/statvar_imports/japan_census/__init__.py new file mode 100644 index 0000000000..f8cb6d9d20 --- /dev/null +++ b/statvar_imports/japan_census/__init__.py @@ -0,0 +1,13 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/statvar_imports/japan_census/test_data/sample_expected_output.tmcf b/statvar_imports/japan_census/test_data/sample_expected_output.tmcf new file mode 100644 index 0000000000..d515c397cd --- /dev/null +++ b/statvar_imports/japan_census/test_data/sample_expected_output.tmcf @@ -0,0 +1,6 @@ +Node: E:sample_expected_output->E0 +observationAbout: C:sample_expected_output->observationAbout +observationDate: C:sample_expected_output->observationDate +value: C:sample_expected_output->value +variableMeasured: C:sample_expected_output->variableMeasured +typeOf: dcs:StatVarObservation diff --git a/statvar_imports/japan_census/validation_config.json b/statvar_imports/japan_census/validation_config.json index f924a0a03f..3aad35ee00 100644 --- a/statvar_imports/japan_census/validation_config.json +++ b/statvar_imports/japan_census/validation_config.json @@ -1,14 +1,6 @@ { "schema_version": "1.0", "rules": [ - { - "rule_id": "check_deleted_records_percent", - "description": "Strictly enforce historical deletion average threshold of 0.1%", - "validator": "DELETED_RECORDS_PERCENT", - "params": { - "threshold": 0.1 - } - }, { "rule_id": "check_max_date_freshness", "description": "Verify MaxDate is within allowable publication lag (3 years for annual series, 5 years for quinquennial census) and values are non-negative (and within [0, 100] for rates) for active recurring indicators", From a3e302c3710221a88d027eda65d0d5ab54968db5 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Sat, 31 Oct 2026 12:13:41 +0000 Subject: [PATCH 16/17] Keep __init__.py empty --- statvar_imports/japan_census/__init__.py | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/statvar_imports/japan_census/__init__.py b/statvar_imports/japan_census/__init__.py index f8cb6d9d20..e69de29bb2 100644 --- a/statvar_imports/japan_census/__init__.py +++ b/statvar_imports/japan_census/__init__.py @@ -1,13 +0,0 @@ -# Copyright 2026 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. From 16340f90f325e17ea858aeccf681d70a26973240 Mon Sep 17 00:00:00 2001 From: goutamabrol Date: Sat, 31 Oct 2026 12:16:52 +0000 Subject: [PATCH 17/17] Update README.md for test_data tmcf and validation_config --- statvar_imports/japan_census/README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/statvar_imports/japan_census/README.md b/statvar_imports/japan_census/README.md index c41759c2bf..a5d158e399 100644 --- a/statvar_imports/japan_census/README.md +++ b/statvar_imports/japan_census/README.md @@ -29,8 +29,8 @@ The import pipeline queries the e-Stat Statistical Dashboard JSON API in paralle * `process.py`: Preprocesses raw input files (computes exact broad age-group sums for Census datasets, merges historical records from `gs://unresolved_mcf/country/japan/deleted_historical_data.csv`, and stages municipality census files by year), runs `stat_var_processor.py` across shards in parallel (`--parallelism=2`), and deterministically merges the resulting `.csv`, `.tmcf`, `.mcf`, and `counters.csv` outputs into `output_files/`. * `japan_census_metadata.csv`: Configuration file for `stat_var_processor.py` specifying output columns, header rows, mapped columns, and provenance URL. * `japan_census_pv_map.csv`: Property-value mapping file mapping e-Stat indicator codes (`indicatorCd`) and 5-digit region codes (`regionCd`) to Data Commons StatisticalVariables and place DCIDs (`country/JPN` and `wikidataId/Q...`). - * `validation_config.json`: Configuration defining historical deletion (`<= 0.1%`), value bounds, and date freshness validation rules across all 101 StatisticalVariables. - * `test_data/`: Sample input data (`sample_input.csv`) and expected output (`sample_expected_output.csv`) for testing. + * `validation_config.json`: Configuration defining value bounds and date freshness validation rules across all 101 StatisticalVariables. + * `test_data/`: Sample input data (`sample_input.csv`) and expected outputs (`sample_expected_output.csv`, `sample_expected_output.tmcf`) for testing. * **Transformation pipeline**: 1. `download_script.py` fetches all 9 raw datasets from the e-Stat JSON API as-is using concurrent workers and atomic file writes into `input_files/`. @@ -41,8 +41,7 @@ The import pipeline queries the e-Stat Statistical Dashboard JSON API in paralle 4. Aggregated processor counters are written to `counters/japan_census_counters.csv`. * **Data Quality Checks**: - * Automated validation via `tools/import_validation/runner.py` evaluates the output against `validation_config.json`, enforcing: - * `check_deleted_records_percent`: Historical deletion threshold `<= 0.1%` (`0` deleted records). + * Automated validation via `tools/import_validation/runner.py` evaluates the output against `validation_config.json` (alongside the default `<= 0.1%` historical deletion threshold), enforcing: * `check_max_date_freshness`: Verifies `MaxDate`, `NumPlaces >= 1`, `MinValue >= 0`, and rate bounds (`MaxValue <= 100` for `UnemploymentRate_Person`) across all 92 active recurring indicators. * `check_max_date_historical_indicators`: Verifies `MaxDate >= 2020` and `MinValue >= 0` across the 9 discontinued centenarian sub-brackets (`100–104`, `105–109`, `110+`). * `check_max_date_consistent_*`: Scoped `MAX_DATE_CONSISTENT` rules verifying uniform `MaxDate` within each co-released survey group (Population Census, Labor Force & Deaths, Teachers, Students, Causes of Death, and Historical Centenarians).