feat(dcp): move the dcp stack to Data Commons Platform 1.1.6 by jm-rivera · Pull Request #61 · DevelopmentFinanceObservatory/infra

jm-rivera · 2026-10-23T13:36:22Z

Moves the dcp stack to Data Commons Platform 1.1.6. infra/stacks/dcp/main.tf, variables.tf and outputs.tf are upstream's 1.1.6 root files with the module source pinned to ?ref=v1.1.6. dcp_version defaults to 1.1.6, selecting the 1.1.6 ingestion images, the ingestion-1.1.6 Dataflow template and the new rollback template. Preprocessing becomes a Cloud Batch job per ingestion with cpu and memory in milliCPU and MiB (8000, 32768), batch.googleapis.com joins the enabled APIs, and the ingestion_prep_job_name output goes with the Cloud Run job. The serving account loses its project-wide roles and gets read-only Spanner. The PR below this one covers the only DFO dependency on those roles. dfo.tf grants the ingestion workflow service account roles/workflows.viewer, which dcp-tools 1.1.0 needs to read the workflow definition before it starts an execution. In infra/stacks/foundation/main.tf, ingestion operators get roles/batch.jobsViewer, and the applier drops roles/vpcaccess.admin and roles/compute.networkViewer, which nothing in 1.1.6 uses with DFO's settings.

The justfile pins CLI 1.1.6. seed-db is gone, init-db drops --init-only, and init-db and migrate-db run under the caller's ADC, which needs roles/spanner.databaseAdmin and roles/spanner.databaseUser. docs/development.md and infra/dfo-dev/dcp/dfo.README.md match.

Merge to dev only after the PR below has applied in both environments (a dev to main release carries it to prod). Once this is on dev, every release carries 1.1.6 to prod.

Before this merges, the synced dcp image runs on dfo-dev under 1.1.5 and passes the smoke checks: /core/api/version, the site, /v2/node paging, SDMX and the /prototype routes. Cutover, per environment, dev first. First confirm the environment runs a dcp image built after the website v3.5.13 sync, because 1.1.6 stops setting USE_SPANNER_GRAPH and older images skip the Spanner graph without it. Read the digest from infra/<env>/dcp/image.auto.tfvars, find its dfo-<tier>-<sha> tag with gcloud artifacts docker images list <image path> --include-tags --filter="version=<digest>", and check git merge-base --is-ancestor 50e4a777e <sha> in the dcp repo. Freeze ingestion from merge until migrate-db has run. Confirm the ingestion workflow has no ACTIVE executions and no lock is held, and take an on-demand Spanner backup, since version retention is 24 hours. Prod keeps stateless_deletion_protection = true, so right before the release that carries this PR, run terraform state rm 'module.stack.module.ingestion_preprocessing_job[0].google_cloud_run_v2_job.dc_data_job' in infra/dfo-prod/dcp. The apply then removes the job's IAM bindings and service account and leaves the job, which gcloud run jobs delete <namespace>-dc-ingestion-preprocessing-job --region europe-west1 --project prod-fin-observatory removes afterwards. Dev runs without protection and needs neither step. After the apply, run just migrate-db (one migration creates the NodeEmbeddingModel remote model). Check the site, SDMX and /v2/node paging, run one small ingestion, and rotate the Data Commons API key, which 1.1.5 preprocessing logged. Ingestion through dcp-tools needs dcp-tools 1.1.0 (ONEcampaign/dcp-tools#197, datacommons-admin 1.1.6) on a 1.1.6 stack, so DevelopmentFinanceObservatory/pipelines#39 moves to it in the same window, after this applies on dev.

Testing: terraform fmt -check -recursive on the dcp and foundation stacks. terraform init -backend=false && terraform validate on dereferenced copies of the dev and prod dcp and foundation directories, clean with no undeclared-variable warnings. just --dry-run init-db and migrate-db print the 1.1.6 commands. Not applied.

infra/stacks/dcp/main.tf, variables.tf and outputs.tf are upstream's 1.1.6 root files with the module ref pinned to v1.1.6. dcp_version defaults to 1.1.6, which picks the 1.1.6 ingestion images, the ingestion-1.1.6 Dataflow template and the new rollback template. Preprocessing runs as a Cloud Batch job with cpu and memory in milliCPU and MiB, so batch.googleapis.com joins the enabled APIs. The ingestion_prep_job_name output goes, since the Cloud Run preprocessing job no longer exists. The justfile pins CLI 1.1.6. seed-db is removed, init-db drops --init-only, and init-db and migrate-db run under the caller's application default credentials, which need roles/spanner.databaseAdmin and roles/spanner.databaseUser. docs/development.md and the dfo-dev dcp README state the same.

Preprocessing runs as a Cloud Batch job, so ingestion operators get roles/batch.jobsViewer beside roles/run.viewer, which still covers the postprocessing job's executions. The 1.1.6 module has no VPC Access connector and no google_compute_network data source with DFO's settings, so the applier drops roles/vpcaccess.admin and roles/compute.networkViewer. The storage_artifacts_bucket_enable_versioning setting goes from both environments: the module's bucket has count 0 here, so it set nothing.

dcp-tools 1.1.0 reads the ingestion workflow's definition before it starts an execution and refuses a workflow older than DCP 1.1.6. It runs as the ingestion workflow service account when callers impersonate it, and the module grants that account only roles/workflows.invoker, which lacks workflows.workflows.get. The account gets roles/workflows.viewer.

github-actions · 2026-10-23T18:14:55Z

Terraform Plan — `foundation`

Plan: 0 to add, 0 to change, 2 to destroy.

Init	Validate	Plan
pass	pass	pass

Show plan

google_project_service.apis["cloudresourcemanager.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/cloudresourcemanager.googleapis.com]
data.google_project.this: Reading...
google_project_service.apis["drive.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/drive.googleapis.com]
google_project_service.apis["iamcredentials.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/iamcredentials.googleapis.com]
google_project_service.apis["docs.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/docs.googleapis.com]
google_project_service.apis["iam.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/iam.googleapis.com]
google_project_iam_custom_role.applier_transfer_configs: Refreshing state... [id=projects/dev-fin-observatory/roles/dfoInfraApplierTransferConfigs]
google_project_service.apis["artifactregistry.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/artifactregistry.googleapis.com]
google_project_service.apis["workflows.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/workflows.googleapis.com]
google_project_service.apis["sts.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/sts.googleapis.com]
google_project_service.apis["sheets.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/sheets.googleapis.com]
data.google_project.this: Read complete after 1s [id=projects/dev-fin-observatory]
google_iam_workload_identity_pool.github: Refreshing state... [id=projects/dev-fin-observatory/locations/global/workloadIdentityPools/github-pool]
google_artifact_registry_repository.dcp_build_cache: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/dfo-dcp-build-cache]
google_artifact_registry_repository.website: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/dfo-website-staging]
module.ai_deployer.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.dcp_deployer.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dcp-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.apis_deployer.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-apis-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.docs_sync.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-docs-sync@dev-fin-observatory.iam.gserviceaccount.com]
module.site_qa_deployer[0].google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com]
google_artifact_registry_repository.datacommons: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/datacommons]
google_project_service_identity.workflows: Refreshing state... [id=projects/dev-fin-observatory/services/workflows.googleapis.com]
module.website_deployer.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-website-staging-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.planner.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_service_account.this: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.site_qa_deployer[0].google_project_iam_member.project_roles["roles/firebasehosting.admin"]: Refreshing state... [id=dev-fin-observatory/roles/firebasehosting.admin/serviceAccount:dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.site_qa_deployer[0].google_project_iam_member.project_roles["roles/firebaserules.admin"]: Refreshing state... [id=dev-fin-observatory/roles/firebaserules.admin/serviceAccount:dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.site_qa_deployer[0].google_project_iam_member.project_roles["roles/serviceusage.serviceUsageConsumer"]: Refreshing state... [id=dev-fin-observatory/roles/serviceusage.serviceUsageConsumer/serviceAccount:dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.site_qa_deployer[0].google_project_iam_member.project_roles["roles/datastore.indexAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/datastore.indexAdmin/serviceAccount:dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_deployer.google_artifact_registry_repository_iam_member.repository_roles["northamerica-northeast1:datacommons:roles/artifactregistry.writer"]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/datacommons/roles/artifactregistry.writer/serviceAccount:dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.apis_deployer.google_artifact_registry_repository_iam_member.repository_roles["northamerica-northeast1:datacommons:roles/artifactregistry.writer"]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/datacommons/roles/artifactregistry.writer/serviceAccount:dfo-apis-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.website_deployer.google_artifact_registry_repository_iam_member.repository_roles["northamerica-northeast1:dfo-website-staging:roles/artifactregistry.writer"]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/dfo-website-staging/roles/artifactregistry.writer/serviceAccount:dfo-website-staging-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.planner.google_project_iam_member.project_roles["roles/serviceusage.serviceUsageConsumer"]: Refreshing state... [id=dev-fin-observatory/roles/serviceusage.serviceUsageConsumer/serviceAccount:dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com]
module.planner.google_project_iam_member.project_roles["roles/iam.securityReviewer"]: Refreshing state... [id=dev-fin-observatory/roles/iam.securityReviewer/serviceAccount:dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com]
module.planner.google_storage_bucket_iam_member.state_bucket[0]: Refreshing state... [id=b/tf-state-dfo-dev-dev-fin-observatory/roles/storage.objectViewer/serviceAccount:dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com/state-bucket-object-prefixes//(resource.type == "storage.googleapis.com/Bucket" && resource.name == "projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory") || (resource.type == "storage.googleapis.com/Object" && (resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/foundation/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/ai/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/apis/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/website/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/dcp/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/platform/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/pipelines/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/edge/") || resource.name.startsWith("projects/_/buckets/tf-state-dfo-dev-dev-fin-observatory/objects/terraform/state/dfo-dev/telemetry/")))]
module.planner.google_project_iam_member.project_roles["roles/viewer"]: Refreshing state... [id=dev-fin-observatory/roles/viewer/serviceAccount:dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com]
google_secret_manager_secret_iam_member.dc_key_planner: Refreshing state... [id=projects/dev-fin-observatory/secrets/dc-api-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/compute.securityAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/compute.securityAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/compute.networkAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/compute.networkAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/iam.serviceAccountAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/iam.serviceAccountAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/artifactregistry.admin"]: Refreshing state... [id=dev-fin-observatory/roles/artifactregistry.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/bigquery.dataOwner"]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.dataOwner/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/dns.admin"]: Refreshing state... [id=dev-fin-observatory/roles/dns.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/bigquery.jobUser"]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.jobUser/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/monitoring.notificationChannelEditor"]: Refreshing state... [id=dev-fin-observatory/roles/monitoring.notificationChannelEditor/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/cloudtasks.queueAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/cloudtasks.queueAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/certificatemanager.editor"]: Refreshing state... [id=dev-fin-observatory/roles/certificatemanager.editor/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/secretmanager.admin"]: Refreshing state... [id=dev-fin-observatory/roles/secretmanager.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/monitoring.alertPolicyEditor"]: Refreshing state... [id=dev-fin-observatory/roles/monitoring.alertPolicyEditor/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/compute.loadBalancerAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/compute.loadBalancerAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/serviceusage.apiKeysAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/serviceusage.apiKeysAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/compute.networkViewer"]: Refreshing state... [id=dev-fin-observatory/roles/compute.networkViewer/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/cloudsql.admin"]: Refreshing state... [id=dev-fin-observatory/roles/cloudsql.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/vpcaccess.admin"]: Refreshing state... [id=dev-fin-observatory/roles/vpcaccess.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/logging.configWriter"]: Refreshing state... [id=dev-fin-observatory/roles/logging.configWriter/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/iam.roleAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/iam.roleAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/bigquery.connectionAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.connectionAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/resourcemanager.projectIamAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/resourcemanager.projectIamAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/iam.serviceAccountUser"]: Refreshing state... [id=dev-fin-observatory/roles/iam.serviceAccountUser/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/spanner.admin"]: Refreshing state... [id=dev-fin-observatory/roles/spanner.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/redis.admin"]: Refreshing state... [id=dev-fin-observatory/roles/redis.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/cloudscheduler.admin"]: Refreshing state... [id=dev-fin-observatory/roles/cloudscheduler.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/workflows.admin"]: Refreshing state... [id=dev-fin-observatory/roles/workflows.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/bigquery.resourceAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.resourceAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/serviceusage.serviceUsageAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/serviceusage.serviceUsageAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/run.admin"]: Refreshing state... [id=dev-fin-observatory/roles/run.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/iam.workloadIdentityPoolAdmin"]: Refreshing state... [id=dev-fin-observatory/roles/iam.workloadIdentityPoolAdmin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/storage.admin"]: Refreshing state... [id=dev-fin-observatory/roles/storage.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
google_iam_workload_identity_pool_provider.github: Refreshing state... [id=projects/dev-fin-observatory/locations/global/workloadIdentityPools/github-pool/providers/github]
google_project_iam_member.applier_transfer_configs: Refreshing state... [id=dev-fin-observatory/projects/dev-fin-observatory/roles/dfoInfraApplierTransferConfigs/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
google_service_account_iam_member.applier_impersonators["***"]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountTokenCreator/***]
module.dcp_deployer.google_artifact_registry_repository_iam_member.repository_roles["northamerica-northeast1:dfo-dcp-build-cache:roles/artifactregistry.writer"]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/dfo-dcp-build-cache/roles/artifactregistry.writer/serviceAccount:dfo-dcp-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.dcp_deployer.google_artifact_registry_repository_iam_member.repository_roles["northamerica-northeast1:datacommons:roles/artifactregistry.writer"]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/repositories/datacommons/roles/artifactregistry.writer/serviceAccount:dfo-dcp-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.docs_sync.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-docs-sync@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_env_ref/1401634043:docsync:refs/heads/dev]
module.site_qa_deployer[0].google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_ref/1406139622:refs/heads/main]
module.website_deployer.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-website-staging-deployer@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_env_ref/1401633828:dev:refs/heads/dev]
module.apis_deployer.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-apis-deployer@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_env_ref/1401633649:dev:refs/heads/dev]
module.dcp_deployer.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dcp-deployer@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_env_ref/1431071058:dev:refs/heads/dev]
module.applier.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_env_ref/1401633456:dev:refs/heads/dev]
module.ai_deployer.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repo_env_ref/1401633727:dev:refs/heads/dev]
module.planner.google_service_account_iam_member.wif: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/principalSet://iam.googleapis.com/projects/41292283861/locations/global/workloadIdentityPools/github-pool/attribute.repository/DevelopmentFinanceObservatory/infra]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # module.applier.google_project_iam_member.project_roles["roles/compute.networkViewer"] will be destroyed
  # (because key ["roles/compute.networkViewer"] is not in for_each map)
  - resource "google_project_iam_member" "project_roles" {
      - etag    = "BwZehXyq/mo=" -> null
      - id      = "dev-fin-observatory/roles/compute.networkViewer/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com" -> null
      - member  = "serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com" -> null
      - project = "dev-fin-observatory" -> null
      - role    = "roles/compute.networkViewer" -> null
    }

  # module.applier.google_project_iam_member.project_roles["roles/vpcaccess.admin"] will be destroyed
  # (because key ["roles/vpcaccess.admin"] is not in for_each map)
  - resource "google_project_iam_member" "project_roles" {
      - etag    = "BwZehXyq/mo=" -> null
      - id      = "dev-fin-observatory/roles/vpcaccess.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com" -> null
      - member  = "serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com" -> null
      - project = "dev-fin-observatory" -> null
      - role    = "roles/vpcaccess.admin" -> null
    }

Plan: 0 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

_{Stack foundation · 7da02cf · @jm-rivera · run}

github-actions · 2026-10-23T18:40:25Z

Terraform Apply — `foundation`

Apply complete! Resources: 0 added, 0 changed, 2 destroyed.

Init	Validate	Apply
pass	pass	pass

Show apply output

module.applier.google_project_iam_member.project_roles["roles/vpcaccess.admin"]: Destroying... [id=dev-fin-observatory/roles/vpcaccess.admin/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/compute.networkViewer"]: Destroying... [id=dev-fin-observatory/roles/compute.networkViewer/serviceAccount:dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com]
module.applier.google_project_iam_member.project_roles["roles/vpcaccess.admin"]: Destruction complete after 9s
module.applier.google_project_iam_member.project_roles["roles/compute.networkViewer"]: Destruction complete after 9s

Apply complete! Resources: 0 added, 0 changed, 2 destroyed.

Outputs:

ai_deployer_service_account = "dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com"
apis_deployer_service_account = "dfo-apis-deployer@dev-fin-observatory.iam.gserviceaccount.com"
applier_service_account = "dfo-infra-applier@dev-fin-observatory.iam.gserviceaccount.com"
artifact_registry_datacommons_repository_id = "datacommons"
artifact_registry_dcp_build_cache_repository_id = "dfo-dcp-build-cache"
artifact_registry_website_repository_id = "dfo-website-staging"
dcp_deployer_service_account = "dfo-dcp-deployer@dev-fin-observatory.iam.gserviceaccount.com"
docs_sync_service_account = "dfo-docs-sync@dev-fin-observatory.iam.gserviceaccount.com"
planner_service_account = "dfo-infra-planner@dev-fin-observatory.iam.gserviceaccount.com"
project_id = "dev-fin-observatory"
region = "northamerica-northeast1"
site_qa_deployer_service_account = "dfo-site-qa-deployer@dev-fin-observatory.iam.gserviceaccount.com"
website_deployer_service_account = "dfo-website-staging-deployer@dev-fin-observatory.iam.gserviceaccount.com"
workload_identity_pool_id = "github-pool"
workload_identity_provider = "projects/41292283861/locations/global/workloadIdentityPools/github-pool/providers/github"

_{Stack foundation · b52ff45 · @jm-rivera · run}

github-actions · 2026-10-23T18:44:02Z

Terraform Apply — `dcp`

Apply complete! Resources: 16 added, 4 changed, 16 destroyed.

Init	Validate	Apply
pass	pass	pass

Show apply output

null_resource.search_api_gemini_key: Destroying... [id=7603058370291887627]
null_resource.search_api_gemini_key: Destruction complete after 0s
ephemeral.random_password.presentation_migrator: Opening...
ephemeral.random_password.datasets_migrator: Opening...
ephemeral.random_password.datasets_migrator: Opening complete after 0s
ephemeral.random_password.presentation_migrator: Opening complete after 0s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/workflows.invoker"]: Destroying... [id=dev-fin-observatory/roles/workflows.invoker/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_developer[0]: Destroying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job/roles/run.developer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.google_service_account_iam_member.workflows_token_creator[0]: Destroying... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountTokenCreator/serviceAccount:service-41292283861@gcp-sa-workflows.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/redis.editor"]: Destroying... [id=dev-fin-observatory/roles/redis.editor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_invoker[0]: Destroying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job/roles/run.invoker/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_viewer[0]: Destroying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job/roles/run.viewer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/iam.serviceAccountUser"]: Destroying... [id=dev-fin-observatory/roles/iam.serviceAccountUser/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_project_iam_member.bq_sa_spanner_viewer[0]: Destroying... [id=dev-fin-observatory/roles/spanner.viewer/serviceAccount:bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/spanner.databaseUser"]: Destroying... [id=dev-fin-observatory/roles/spanner.databaseUser/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/secretmanager.secretAccessor"]: Destroying... [id=dev-fin-observatory/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.google_service_account_iam_member.workflows_token_creator[0]: Destruction complete after 4s
ephemeral.random_password.datasets_migrator: Closing...
ephemeral.random_password.datasets_migrator: Closing complete after 0s
module.stack.module.spanner[0].google_spanner_database_iam_member.spanner_reader[0]: Destroying... [id=dev-fin-observatory/dfo-dev-dc-instance/dfo-dev-dc-db/roles/spanner.databaseUser/serviceAccount:bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/secretmanager.secretAccessor"]: Destruction complete after 8s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/compute.networkViewer"]: Destroying... [id=dev-fin-observatory/roles/compute.networkViewer/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/redis.editor"]: Destruction complete after 8s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/storage.objectViewer"]: Destroying... [id=dev-fin-observatory/roles/storage.objectViewer/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_project_iam_member.bq_sa_spanner_viewer[0]: Destruction complete after 8s
ephemeral.random_password.presentation_migrator: Closing...
ephemeral.random_password.presentation_migrator: Closing complete after 0s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/spanner.databaseUser"]: Destruction complete after 8s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/workflows.invoker"]: Destruction complete after 9s
module.stack.module.spanner[0].google_spanner_database_iam_member.spanner_reader[0]: Destruction complete after 5s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/iam.serviceAccountUser"]: Destruction complete after 9s
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_developer[0]: Still destroying... [id=projects/dev-fin-observatory/locations/...in-observatory.iam.gserviceaccount.com, 00m10s elapsed]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_invoker[0]: Still destroying... [id=projects/dev-fin-observatory/locations/...in-observatory.iam.gserviceaccount.com, 00m10s elapsed]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_viewer[0]: Still destroying... [id=projects/dev-fin-observatory/locations/...in-observatory.iam.gserviceaccount.com, 00m10s elapsed]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_developer[0]: Destruction complete after 13s
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_viewer[0]: Destruction complete after 13s
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_invoker[0]: Destruction complete after 13s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/storage.objectViewer"]: Destruction complete after 8s
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/compute.networkViewer"]: Destruction complete after 9s
module.stack.module.ingestion_preprocessing_job[0].google_cloud_run_v2_job.dc_data_job: Destroying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job]
module.stack.module.ingestion_preprocessing_job[0].google_cloud_run_v2_job.dc_data_job: Still destroying... [id=projects/dev-fin-observatory/locations/...dfo-dev-dc-ingestion-preprocessing-job, 00m10s elapsed]
module.stack.module.ingestion_preprocessing_job[0].google_cloud_run_v2_job.dc_data_job: Destruction complete after 10s
google_project_service.apis["vpcaccess.googleapis.com"]: Destroying... [id=dev-fin-observatory/vpcaccess.googleapis.com]
google_project_service.apis["vpcaccess.googleapis.com"]: Destruction complete after 0s
google_project_service.apis["batch.googleapis.com"]: Creating...
google_project_service.apis["batch.googleapis.com"]: Still creating... [00m10s elapsed]
google_project_service.apis["batch.googleapis.com"]: Still creating... [00m20s elapsed]
google_project_service.apis["batch.googleapis.com"]: Creation complete after 22s [id=dev-fin-observatory/batch.googleapis.com]
module.stack.terraform_data.redis_network_validation: Creating...
module.stack.terraform_data.redis_network_validation: Creation complete after 0s [id=83d25509-7164-17cf-b8ca-63cd2e9fd694]
module.stack.terraform_data.worker_subnet_validation: Creating...
module.stack.module.network.terraform_data.existing_network_config_validation: Creating...
module.stack.terraform_data.worker_subnet_validation: Creation complete after 0s [id=f62e9abe-37be-bc80-94e4-d18fc10e68fd]
module.stack.module.network.terraform_data.existing_network_config_validation: Creation complete after 0s [id=5bc94028-ec04-7bf9-075d-2848dbc02766]
module.stack.module.ingestion_dataflow.data.google_project.project: Reading...
module.stack.module.spanner[0].data.google_bigquery_default_service_account.bq_sa: Reading...
module.stack.module.ingestion_preprocessing_job[0].google_project_iam_member.preprocessing_batch_agent: Creating...
module.stack.module.ingestion_preprocessing_job[0].google_project_iam_member.preprocessing_log_writer: Creating...
module.stack.module.ingestion_helper_service.google_cloud_run_v2_service.ingestion_helper[0]: Modifying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-ingestion-helper]
module.stack.module.ingestion_postprocessing_job[0].google_cloud_run_v2_job.dc_postprocessing_job: Modifying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-postprocessing-job]
module.stack.module.ingestion_dataflow.data.google_project.project: Read complete after 1s [id=projects/dev-fin-observatory]
module.stack.module.ingestion_dataflow.google_service_account_iam_member.workflows_token_creator[0]: Creating...
module.stack.module.spanner[0].data.google_bigquery_default_service_account.bq_sa: Read complete after 1s [id=bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_project_iam_member.bq_sa_spanner_viewer[0]: Creating...
module.stack.module.spanner[0].google_spanner_database_iam_member.spanner_reader[0]: Creating...
module.stack.module.ingestion_dataflow.google_service_account_iam_member.workflows_token_creator[0]: Creation complete after 4s [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountTokenCreator/serviceAccount:service-41292283861@gcp-sa-workflows.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_spanner_database_iam_member.spanner_reader[0]: Creation complete after 4s [id=dev-fin-observatory/dfo-dev-dc-instance/dfo-dev-dc-db/roles/spanner.databaseUser/serviceAccount:bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_project_iam_member.bq_sa_spanner_viewer[0]: Creation complete after 8s [id=dev-fin-observatory/roles/spanner.viewer/serviceAccount:bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.ingestion_preprocessing_job[0].google_project_iam_member.preprocessing_batch_agent: Creation complete after 9s [id=dev-fin-observatory/roles/batch.agentReporter/serviceAccount:dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_preprocessing_job[0].google_project_iam_member.preprocessing_log_writer: Creation complete after 9s [id=dev-fin-observatory/roles/logging.logWriter/serviceAccount:dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_secret_manager_secret_iam_member.serving_secret_accessor["MAPS_API_KEY"]: Creating...
module.stack.module.datacommons_services[0].google_secret_manager_secret_iam_member.serving_secret_accessor["DC_API_KEY"]: Creating...
module.stack.google_storage_bucket_iam_member.serving_bucket_access[0]: Creating...
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/spanner.databaseReader"]: Creating...
module.stack.module.ingestion_helper_service.google_cloud_run_v2_service.ingestion_helper[0]: Still modifying... [id=projects/dev-fin-observatory/locations/...1/services/dfo-dev-dc-ingestion-helper, 00m10s elapsed]
module.stack.module.ingestion_postprocessing_job[0].google_cloud_run_v2_job.dc_postprocessing_job: Still modifying... [id=projects/dev-fin-observatory/locations/...fo-dev-dc-ingestion-postprocessing-job, 00m10s elapsed]
module.stack.module.ingestion_postprocessing_job[0].google_cloud_run_v2_job.dc_postprocessing_job: Modifications complete after 10s [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-postprocessing-job]
module.stack.google_storage_bucket_iam_member.serving_bucket_access[0]: Creation complete after 4s [id=b/dfo-dev-dc-artifacts-dev-fin-observatory/roles/storage.objectViewer/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_secret_manager_secret_iam_member.serving_secret_accessor["DC_API_KEY"]: Creation complete after 4s [id=projects/dev-fin-observatory/secrets/dfo-dev-dc-api-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_secret_manager_secret_iam_member.serving_secret_accessor["MAPS_API_KEY"]: Creation complete after 5s [id=projects/dev-fin-observatory/secrets/dfo-dev-dc-google-maps-api-key-f1c75504/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/spanner.databaseReader"]: Creation complete after 8s [id=dev-fin-observatory/roles/spanner.databaseReader/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_helper_service.google_cloud_run_v2_service.ingestion_helper[0]: Still modifying... [id=projects/dev-fin-observatory/locations/...1/services/dfo-dev-dc-ingestion-helper, 00m21s elapsed]
module.stack.module.ingestion_helper_service.google_cloud_run_v2_service.ingestion_helper[0]: Still modifying... [id=projects/dev-fin-observatory/locations/...1/services/dfo-dev-dc-ingestion-helper, 00m31s elapsed]
module.stack.module.ingestion_helper_service.google_cloud_run_v2_service.ingestion_helper[0]: Modifications complete after 31s [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-ingestion-helper]
google_project_iam_member.ingestion_workflow_sa_viewer[0]: Creating...
module.stack.google_project_iam_member.workflow_batch_editor[0]: Creating...
module.stack.module.ingestion_workflow.google_workflows_workflow.ingestion_orchestrator[0]: Modifying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/workflows/dfo-dev-dc-ingestion-workflow]
module.stack.google_project_iam_member.workflow_batch_editor[0]: Creation complete after 8s [id=dev-fin-observatory/roles/batch.jobsEditor/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
google_project_iam_member.ingestion_workflow_sa_viewer[0]: Creation complete after 8s [id=dev-fin-observatory/roles/workflows.viewer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_workflow.google_workflows_workflow.ingestion_orchestrator[0]: Still modifying... [id=projects/dev-fin-observatory/locations/...orkflows/dfo-dev-dc-ingestion-workflow, 00m10s elapsed]
module.stack.module.ingestion_workflow.google_workflows_workflow.ingestion_orchestrator[0]: Modifications complete after 10s [id=projects/dev-fin-observatory/locations/northamerica-northeast1/workflows/dfo-dev-dc-ingestion-workflow]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Modifying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m10s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m20s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m30s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m40s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Modifications complete after 41s [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service]
null_resource.search_api_gemini_key: Creating...
null_resource.search_api_gemini_key: Provisioning with 'local-exec'...
null_resource.search_api_gemini_key (local-exec): Executing: ["/bin/sh" "-c" "\"./../../../scripts/set-search-api-gemini-key.sh\" \\\n  dev-fin-observatory \\\n  northamerica-northeast1 \\\n  dfo-dev-dc-datacommons-service \\\n  dfo-dev-gemini-api-key\n"]
null_resource.search_api_gemini_key: Still creating... [00m10s elapsed]
null_resource.search_api_gemini_key: Still creating... [00m20s elapsed]
null_resource.search_api_gemini_key: Still creating... [00m30s elapsed]
null_resource.search_api_gemini_key (local-exec): GEMINI_API_KEY set on dfo-dev-dc-datacommons-service from projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key
null_resource.search_api_gemini_key: Creation complete after 33s [id=3525530315236628214]

Apply complete! Resources: 16 added, 4 changed, 16 destroyed.

Outputs:

datacommons_service_name = "dfo-dev-dc-datacommons-service"
datacommons_service_service_account_email = "dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com"
datacommons_service_url = "https://dfo-dev-dc-datacommons-service-zms5lj4zhq-nn.a.run.app"
datacommons_services_image = "northamerica-northeast1-docker.pkg.dev/dev-fin-observatory/datacommons/dfo-services-image@sha256:8848e98280dd52db10cc8e935feefbd4d74f7b4cd4f5effabf3cba54be2ef23f"
datasets_database_name = "datasets"
datasets_gc_db_user = "dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam"
datasets_gc_service_account_email = "dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam.gserviceaccount.com"
datasets_instance_connection_name = "dev-fin-observatory:northamerica-northeast1:dfo-dev-platform-postgres"
datasets_migrator_secret_id = "projects/dev-fin-observatory/secrets/dfo-dev-datasets-migrator-password"
datasets_migrator_secret_name = "dfo-dev-datasets-migrator-password"
datasets_pinned_bucket_name = "dev-fin-observatory-datasets-pinned"
datasets_working_bucket_name = "dev-fin-observatory-datasets-working"
ingestion_artifacts_path = "ingestion/internal"
ingestion_bucket_url = "gs://dfo-dev-dc-artifacts-dev-fin-observatory"
ingestion_dataflow_template_gcs_path = "gs://datcom-templates/templates/flex/ingestion-1.1.6.json"
ingestion_helper_image = "gcr.io/datcom-ci/datacommons-ingestion-helper:1.1.6"
ingestion_input_path = "ingestion/input"
ingestion_postprocessing_image = "gcr.io/datcom-ci/datacommons-aggregation-helper:1.1.6"
ingestion_preprocessing_image = "gcr.io/datcom-ci/datacommons-data:1.1.6"
ingestion_service_url = "https://dfo-dev-dc-ingestion-helper-zms5lj4zhq-nn.a.run.app"
ingestion_workflow_id = "projects/dev-fin-observatory/locations/northamerica-northeast1/workflows/dfo-dev-dc-ingestion-workflow"
ingestion_workflow_name = "dfo-dev-dc-ingestion-workflow"
ingestion_workflow_service_account_email = "dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com"
presentation_database_name = "presentation"
presentation_migrator_secret_id = "projects/dev-fin-observatory/secrets/dfo-dev-presentation-migrator-password"
presentation_migrator_secret_name = "dfo-dev-presentation-migrator-password"
project_id = "dev-fin-observatory"
region = "northamerica-northeast1"
spanner_database_id = "dfo-dev-dc-db"
spanner_instance_id = "dfo-dev-dc-instance"
storage_artifacts_bucket_name = "dfo-dev-dc-artifacts-dev-fin-observatory"

_{Stack dcp · b52ff45 · @jm-rivera · run}

tillywoodfield · 2026-10-26T09:41:24Z

Closes #41

jm-rivera force-pushed the feat/dcp-v1.1.6 branch from a9bf660 to a1fc36a Compare October 23, 2026 14:28

jm-rivera force-pushed the feat/dcp-v116-prep branch from 170b7a3 to 9c31f45 Compare October 23, 2026 14:28

This was referenced Oct 23, 2026

chore(dcp): restore prod stateless deletion protection #62

Closed

feat(ingestion): move to dcp-tools 1.1.0 for Data Commons Platform 1.1.6 DevelopmentFinanceObservatory/pipelines#39

Open

jm-rivera force-pushed the feat/dcp-v116-prep branch from 9c31f45 to d556c94 Compare October 23, 2026 16:57

jm-rivera force-pushed the feat/dcp-v1.1.6 branch from 903b77f to 76d7006 Compare October 23, 2026 16:57

jm-rivera force-pushed the feat/dcp-v116-prep branch from d556c94 to 985a5aa Compare October 23, 2026 18:08

jm-rivera mentioned this pull request Oct 23, 2026

feat(dcp): grant the serving account the Gemini key secret #60

Merged

jm-rivera added 3 commits October 23, 2026 20:08

jm-rivera force-pushed the feat/dcp-v1.1.6 branch from 76d7006 to 7da02cf Compare October 23, 2026 18:11

jm-rivera changed the base branch from feat/dcp-v116-prep to dev October 23, 2026 18:14

jm-rivera marked this pull request as ready for review October 23, 2026 18:14

jm-rivera deployed to dev-plan October 23, 2026 18:14 — with GitHub Actions Active

jm-rivera merged commit b52ff45 into dev Oct 23, 2026
10 checks passed

DevelopmentFinanceObservatory deleted a comment from github-actions Bot Oct 23, 2026

tillywoodfield mentioned this pull request Oct 26, 2026

Upgrade the DCP stack module to v1.1.6 in dev and prod #41

Closed

2 tasks

Navigation Menu

Repository navigation

feat(dcp): move the dcp stack to Data Commons Platform 1.1.6 - #61

feat(dcp): move the dcp stack to Data Commons Platform 1.1.6#61
jm-rivera merged 3 commits into
devfrom
feat/dcp-v1.1.6

jm-rivera commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

tillywoodfield commented Oct 26, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jm-rivera commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026

Terraform Plan — foundation

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026

Terraform Apply — foundation

Uh oh!

github-actions Bot commented Oct 23, 2026

Terraform Apply — dcp

Uh oh!

tillywoodfield commented Oct 26, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jm-rivera commented Oct 23, 2026 •

edited

Loading

Terraform Plan — `foundation`

Terraform Apply — `foundation`

Terraform Apply — `dcp`