Skip to content

fixed script WorldBank_IDS - #2291

Open
niveditasing wants to merge 2 commits into
datacommonsorg:masterfrom
niveditasing:WorldBankIds_fix
Open

niveditasing wants to merge 2 commits into
datacommonsorg:masterfrom
niveditasing:WorldBankIds_fix

Conversation

@niveditasing

@niveditasing niveditasing commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes two failures in the WorldBank_IDS (statvar_imports/world_bank/worldbank_ids:WorldBank_IDS) auto-refresh import:

  1. Download failure (HTTP 524 origin timeout / AttributeError: 'NoneType' object has no attribute 'rename'):

    • Fetching all 63 years (1970–current_year + 6) for an indicator in a single World Bank API call intermittently triggers Cloudflare HTTP 524 timeouts, causing bblocks (get_indicator_data) to return None and crash in DebtIDS._get_indicator (data.rename(...)).
    • Updated download.py to fetch each indicator individually with bounded retries and exponential backoff (MAX_RETRIES = 3), and fall back to 20-year chunks (YEAR_CHUNK_SIZE = 20) if full-range requests still fail.
    • Removed the unused WorldBankData import from download.py.
  2. Validation failure (check_lint_error_count / Sanity_InvalidChars_dcid on input0):

    • Upstream World Bank IDS data contains two counterpart_area values (Africa Finance Corporation and Uganda) that were missing from InputlendingEntity.csv.
    • In currency (input0), configs/currency_pvmap.csv fell back to lendingEntity: {Data}, generating a provisional schema node Node: dcid:Africa Finance Corporation (containing spaces) in currency_stat_vars_schema.mcf and failing genmcf lint validation (Sanity_InvalidChars_dcid).
    • In interest, principal, and disbursed (input1–input3), rows for these two unmapped lenders lacked a lendingEntity property and collided with World StatVars (error-statvar-with-dup-svobs).
    • Added Africa Finance Corporation -> Africa_Finance_Corporation and Uganda -> country/UGA to InputlendingEntity.csv.

Files Changed

  • statvar_imports/world_bank/worldbank_ids/download.py
  • statvar_imports/world_bank/worldbank_ids/InputlendingEntity.csv

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds new entries to the InputlendingEntity.csv mapping and refactors the World Bank data download script to implement a robust retry mechanism with exponential backoff and a fallback to year-based chunking. The feedback recommends narrowing down the exception handling in both retry loops from a broad Exception to specific transient errors, such as requests.exceptions.RequestException, to prevent masking programming or configuration bugs.

Comment thread statvar_imports/world_bank/worldbank_ids/download.py
Comment thread statvar_imports/world_bank/worldbank_ids/download.py
@niveditasing niveditasing changed the title fixed script fixed script WorldBank_IDS Oct 20, 2026
@balit-raibot
balit-raibot self-requested a review October 23, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants