[jm-rivera]

Data Commons Platform 1.1.6 deletes the Cloud Run preprocessing job, and dcp-tools 1.1.0 (ONEcampaign/dcp-tools#197) starts the ingestion workflow without its name. KnowledgeGraphResource loses prep_job_name, definitions.py stops reading DCP_PREP_JOB_NAME, and .env.example drops it. The resource builds dcp-tools settings from its own values only, through a subclass whose settings_customise_sources keeps init values, so environment variables such as GCS_INPUT_FOLDER_PATH no longer override it.

ingest_bundle in src/dfo_pipelines/data_commons/ingestion.py calls check_ingestion_workflow_contract before uploading, so a stack older than 1.1.6 receives nothing and the run fails with WorkflowContractError. It then starts the workflow, watches the execution for up to 120 s at 5 s intervals, and fails the run on FAILED or CANCELLED with the execution's error. Each of the 11 ingestion assets records execution_name metadata beside dagster/uri. The GCS client gets project=settings.gcp_project_id. The export passes the source release date through add_provenance's imported_source_release_date keyword. Tests share one knowledge_graph fixture in tests/conftest.py and compare provenance as parsed properties. The README's Ingestion section covers the watch, the metadata and the roles/workflows.viewer requirement.

The branch requires dcp-tools>=1.1.0 and declares pydantic-settings, which resources.py imports directly. The lock gains datacommons-admin 1.1.6 and datacommons-db, and setuptools drops from 83.0.0 to 80.0.0 under datacommons-db's cap. Merge after DevelopmentFinanceObservatory/infra#61 applies on dev, then remove DCP_PREP_JOB_NAME from the Dagster deployment.

dcp-tools 1.1.0 leaves logging to the importing application, so its INFO lines reach a run log only if the deployed code location captures the root or dcp_tools logger, as .dagster_home/dagster.yaml does locally. The execution name is in asset metadata either way.

Testing: after uv sync --locked against dcp-tools 1.1.0 from PyPI, pytest passes 444 of 444, ruff check, ruff format --check, ty check and sqlfluff lint are clean, dg check defs loads every definition, and dbt build on the CI fixtures passes 1138 nodes. No ingestion was run.

[tillywoodfield]

Thanks @jm-rivera! I've released dcp-tools 1.2.0, which includes changes that enable us to drop a few things from this PR. Please could this PR move to dcp-tools>=1.2.0 and keep to what the upgrade needs, with the rest as follow-up PRs, so each is easier to review?

For the upgrade itself:

• Require dcp-tools>=1.2.0.
• Drop prep_job_name from KnowledgeGraphResource, DCP_PREP_JOB_NAME from definitions.py and .env.example, and the test arguments, with the shared knowledge_graph fixture since those lines change anyway.
• Drop local_path=Path("."), gcs_output_folder_path="" and their comment from KnowledgeGraphResource.settings, since both are optional in 1.2.0.
• Call check_ingestion_workflow_contract before the sync, so a bundle never uploads to a stack older than 1.1.6.
• Add the README's line on roles/workflows.viewer, which the contract check needs.

For follow-ups:

• Watching the execution and recording execution_name metadata on the 11 assets. On 1.2.0, watch_execution_start becomes a call to wait_for_ingestion(settings, execution_name, timeout=watch_seconds, poll_interval=poll_seconds). It returns the terminal state, including FAILED and CANCELLED, and raises TimeoutError when the watch runs out, which ingest_bundle can treat as the execution still running. That drops the sleep and clock parameters and the fake-clock tests.
• Passing the source release date through add_provenance's imported_source_release_date, with the provenance test that compares parsed properties.
• Passing project=settings.gcp_project_id to get_gcs_client.
• _ResourceKGSettings, which shuts out environment variables as well as .env, and the pydantic-settings dependency it brings. KGSettings ignores undeclared .env variables in 1.2.0, so only the environment part is left to decide.