Skip to content

feat(dcp): grant the serving account the Gemini key secret - #60

Merged
jm-rivera merged 1 commit into
devfrom
feat/dcp-v116-prep
Oct 23, 2026
Merged

jm-rivera merged 1 commit into
devfrom
feat/dcp-v116-prep

Conversation

@jm-rivera

@jm-rivera jm-rivera commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Data Commons Platform 1.1.6 removes the serving service account's project-wide roles/secretmanager.secretAccessor. null_resource.search_api_gemini_key attaches the ${namespace}-gemini-api-key secret (dfo-dev-gemini-api-key, dfo-prod-gemini-api-key) to the datacommons service as GEMINI_API_KEY, and a Cloud Run revision whose account cannot read a referenced secret fails to start. Under 1.1.6 the script's rollout would fail and take the apply with it. infra/stacks/dcp/dfo.tf adds data.google_secret_manager_secret.search_api_gemini for that secret and google_secret_manager_secret_iam_member.serving_sa_gemini_key on it for module.stack.datacommons_service_service_account_email, an output both module versions expose, and the key script now takes the secret id from the data source and depends on the grant. The grant is additive, so it applies on 1.1.5 ahead of the 1.1.6 port in #61. When #65 moves the dcp stack to a per-caller secret, the data source's secret_id changes to its variable and the grant follows.

The key serves the DFO prototype app's /prototype/api/similarity (which falls back to pseudo-embeddings without it) and /prototype/api/skill-compare. /api/search is proxied to dfo-apis and never reads it. The comments in dfo.tf and scripts/set-search-api-gemini-key.sh, the dev and prod dfo.README.md, and the line in infra/dfo-dev/ai/README.md describe that.

Testing: terraform fmt -check, bash -n scripts/set-search-api-gemini-key.sh, and terraform init -backend=false && terraform validate on dereferenced copies of infra/dfo-dev/dcp and infra/dfo-prod/dcp, all clean. Not applied.

@jm-rivera jm-rivera changed the title feat(dcp): grant the Gemini key per secret and lift prod job protection feat(dcp): grant the serving account the Gemini key secret Oct 23, 2026
@jm-rivera
jm-rivera changed the base branch from dev to feat/gemini-key-per-caller-dev October 23, 2026 16:57
DCP 1.1.6 removes the serving service account's project-wide
secretmanager.secretAccessor role. null_resource.search_api_gemini_key
attaches the ${namespace}-gemini-api-key secret to the datacommons
service as GEMINI_API_KEY, and a revision whose account cannot read the
secret fails to start, so the script's rollout and the apply would
fail. A per-secret grant on data.google_secret_manager_secret.search_api_gemini
keeps the access under both module versions, and the key script waits
for it.

The key serves the DFO prototype app's /prototype/api/similarity and
/prototype/api/skill-compare routes; /api/search goes to dfo-apis.
The comments and READMEs say so.
@jm-rivera
jm-rivera changed the base branch from feat/gemini-key-per-caller-dev to dev October 23, 2026 18:08
@jm-rivera
jm-rivera marked this pull request as ready for review October 23, 2026 18:08
@github-actions

Copy link
Copy Markdown

Terraform Plan — ai

No changes. Infrastructure matches configuration.

Init Validate Plan
pass pass pass
Show plan
ephemeral.random_password.mcp_api_token: Opening...
ephemeral.random_password.studio_api_token: Opening...
ephemeral.random_password.mcp_api_token: Opening complete after 0s
ephemeral.random_password.studio_api_token: Opening complete after 0s
data.google_secret_manager_secret.langfuse_public_key: Reading...
data.google_secret_manager_secret.gemini: Reading...
data.google_secret_manager_secret.langfuse_secret_key: Reading...
google_sql_database.ai_sessions: Refreshing state... [id=projects/dev-fin-observatory/instances/dfo-dev-platform-postgres/databases/ai-sessions]
google_project_iam_custom_role.cloudsql_login_only: Refreshing state... [id=projects/dev-fin-observatory/roles/dfoDevAiCloudsqlLoginOnly]
google_storage_bucket.materialisation_store: Refreshing state... [id=dev-fin-observatory-ai-materialisation-store]
google_secret_manager_secret.studio_api_token: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-studio-api-token]
data.google_service_account.ai_deployer: Reading...
google_secret_manager_secret.mcp_api_token: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-mcp-api-token]
google_storage_bucket.definition_store: Refreshing state... [id=dev-fin-observatory-ai-definition-store]
data.google_secret_manager_secret.langfuse_secret_key: Read complete after 0s [id=projects/dev-fin-observatory/secrets/dfo-dev-langfuse-secret-key]
google_project_service.dns: Refreshing state... [id=dev-fin-observatory/dns.googleapis.com]
data.google_secret_manager_secret.langfuse_public_key: Read complete after 0s [id=projects/dev-fin-observatory/secrets/dfo-dev-langfuse-public-key]
data.google_secret_manager_secret.gemini: Read complete after 0s [id=projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key]
module.ai_service.google_service_account.runtime: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_secret_manager_secret_iam_member.studio_api_token_readers["***"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-studio-api-token/roles/secretmanager.secretAccessor/***]
google_secret_manager_secret_version.studio_api_token: Refreshing state... [id=projects/41292283861/secrets/dfo-dev-studio-api-token/versions/1]
google_secret_manager_secret_version.mcp_api_token: Refreshing state... [id=projects/41292283861/secrets/dfo-dev-mcp-api-token/versions/1]
data.google_service_account.ai_deployer: Read complete after 0s [id=projects/dev-fin-observatory/serviceAccounts/dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-datasets-first-party-principal"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-datasets-first-party-principal/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_service_account_iam_member.deployer_act_as["serviceAccount:dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com"]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-mcp-api-token"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-mcp-api-token/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-gemini-api-key"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-studio-api-token"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-studio-api-token/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-live-refresh-principal"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-live-refresh-principal/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-langfuse-public-key"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-langfuse-public-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
module.ai_service.google_secret_manager_secret_iam_member.runtime_secret_access["dfo-dev-langfuse-secret-key"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-langfuse-secret-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_project_iam_member.ai_cloudsql_client: Refreshing state... [id=dev-fin-observatory/roles/cloudsql.client/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_cloud_run_v2_service_iam_member.ai_apis_invoker: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-apis/roles/run.invoker/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_sql_user.ai_sessions_runtime: Refreshing state... [id=dfo-dev-ai-run@dev-fin-observatory.iam//dfo-dev-platform-postgres]
google_project_iam_member.ai_cloudsql_login: Refreshing state... [id=dev-fin-observatory/projects/dev-fin-observatory/roles/dfoDevAiCloudsqlLoginOnly/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_storage_bucket_iam_member.materialisation_store_creator: Refreshing state... [id=b/dev-fin-observatory-ai-materialisation-store/roles/storage.objectCreator/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_storage_bucket_iam_member.materialisation_store_viewer: Refreshing state... [id=b/dev-fin-observatory-ai-materialisation-store/roles/storage.objectViewer/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_storage_bucket_iam_member.definition_store_creator: Refreshing state... [id=b/dev-fin-observatory-ai-definition-store/roles/storage.objectCreator/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_storage_bucket_iam_member.definition_store_viewer: Refreshing state... [id=b/dev-fin-observatory-ai-definition-store/roles/storage.objectViewer/serviceAccount:dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com]
google_sql_provision_script.ai_sessions_schema_grant: Refreshing state... [id=dev-fin-observatory/dfo-dev-platform-postgres]
google_sql_provision_script.ai_sessions_revoke_public_connect: Refreshing state... [id=dev-fin-observatory/dfo-dev-platform-postgres]
module.psc_network.google_compute_network.this: Refreshing state... [id=projects/dev-fin-observatory/global/networks/dfo-dev-ai-psc]
module.psc_network.google_dns_managed_zone.psc: Refreshing state... [id=projects/dev-fin-observatory/managedZones/dfo-dev-ai-psc-sql]
module.psc_network.google_compute_subnetwork.this: Refreshing state... [id=projects/dev-fin-observatory/regions/northamerica-northeast1/subnetworks/dfo-dev-ai-psc-subnet]
module.psc_network.google_compute_address.psc_endpoint: Refreshing state... [id=projects/dev-fin-observatory/regions/northamerica-northeast1/addresses/dfo-dev-ai-psc-endpoint]
module.ai_service.google_cloud_run_v2_service.this: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-ai]
module.psc_network.google_dns_record_set.psc: Refreshing state... [id=projects/dev-fin-observatory/managedZones/dfo-dev-ai-psc-sql/rrsets/bbd723248b19.3niso0mm9bket.northamerica-northeast1.sql.goog./A]
module.psc_network.google_compute_forwarding_rule.psc_endpoint: Refreshing state... [id=projects/dev-fin-observatory/regions/northamerica-northeast1/forwardingRules/dfo-dev-ai-psc-fwd]
module.ai_service.google_cloud_run_v2_service_iam_member.public[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-ai/roles/run.invoker/allUsers]
module.ai_service.google_cloud_run_v2_service_iam_member.deployer_run_developer["serviceAccount:dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com"]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-ai/roles/run.developer/serviceAccount:dfo-ai-deployer@dev-fin-observatory.iam.gserviceaccount.com]
ephemeral.random_password.studio_api_token: Closing...
ephemeral.random_password.mcp_api_token: Closing...
ephemeral.random_password.mcp_api_token: Closing complete after 0s
ephemeral.random_password.studio_api_token: Closing complete after 0s

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Stack ai · 985a5aa · @jm-rivera · run

@github-actions

Copy link
Copy Markdown

Terraform Plan — dcp

Plan: 2 to add, 1 to change, 1 to destroy.

Init Validate Plan
pass pass pass
Show plan
ephemeral.random_password.presentation_migrator: Opening...
ephemeral.random_password.datasets_migrator: Opening...
ephemeral.random_password.presentation_migrator: Opening complete after 0s
ephemeral.random_password.datasets_migrator: Opening complete after 0s
data.google_secret_manager_secret_version.dc_key: Reading...
google_sql_database.datasets: Refreshing state... [id=projects/dev-fin-observatory/instances/dfo-dev-platform-postgres/databases/datasets]
google_secret_manager_secret.datasets_migrator_password: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-datasets-migrator-password]
google_sql_database.presentation: Refreshing state... [id=projects/dev-fin-observatory/instances/dfo-dev-platform-postgres/databases/presentation]
google_sql_user.datasets_migrator: Refreshing state... [id=dfo-dev-datasets-migrator//dfo-dev-platform-postgres]
google_storage_bucket.datasets_pinned: Refreshing state... [id=dev-fin-observatory-datasets-pinned]
google_service_account.datasets_gc: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam.gserviceaccount.com]
google_secret_manager_secret.presentation_migrator_password: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-presentation-migrator-password]
google_storage_bucket.dc_artifacts: Refreshing state... [id=dfo-dev-dc-artifacts-dev-fin-observatory]
google_storage_bucket.datasets_working: Refreshing state... [id=dev-fin-observatory-datasets-working]
data.google_secret_manager_secret.search_api_gemini: Reading...
google_sql_user.presentation_migrator: Refreshing state... [id=dfo-dev-presentation-migrator//dfo-dev-platform-postgres]
data.google_secret_manager_secret.search_api_gemini: Read complete after 0s [id=projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key]
google_project_service.apis["artifactregistry.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/artifactregistry.googleapis.com]
data.google_secret_manager_secret_version.dc_key: Read complete after 0s [id=projects/41292283861/secrets/dc-api-key/versions/2]
google_project_service.apis["iam.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/iam.googleapis.com]
google_project_service.apis["run.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/run.googleapis.com]
google_project_service.apis["sqladmin.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/sqladmin.googleapis.com]
google_project_service.apis["places-backend.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/places-backend.googleapis.com]
google_project_service.apis["bigqueryconnection.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/bigqueryconnection.googleapis.com]
google_project_service.apis["dataflow.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/dataflow.googleapis.com]
google_project_service.apis["workflows.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/workflows.googleapis.com]
google_project_service.apis["bigqueryreservation.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/bigqueryreservation.googleapis.com]
google_project_service.apis["vpcaccess.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/vpcaccess.googleapis.com]
google_project_service.apis["apikeys.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/apikeys.googleapis.com]
google_project_service.apis["spanner.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/spanner.googleapis.com]
google_project_service.apis["cloudresourcemanager.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/cloudresourcemanager.googleapis.com]
google_project_service.apis["secretmanager.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/secretmanager.googleapis.com]
google_project_service.apis["aiplatform.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/aiplatform.googleapis.com]
google_project_service.apis["bigquery.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/bigquery.googleapis.com]
google_project_service.apis["redis.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/redis.googleapis.com]
google_project_service.apis["workflowexecutions.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/workflowexecutions.googleapis.com]
google_project_service.apis["compute.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/compute.googleapis.com]
google_project_service.apis["maps-backend.googleapis.com"]: Refreshing state... [id=dev-fin-observatory/maps-backend.googleapis.com]
google_secret_manager_secret_version.presentation_migrator_password: Refreshing state... [id=projects/41292283861/secrets/dfo-dev-presentation-migrator-password/versions/1]
null_resource.dc_key_guard: Refreshing state... [id=6055240469338298543]
google_storage_bucket_iam_member.datasets_pinned_gc_admin: Refreshing state... [id=b/dev-fin-observatory-datasets-pinned/roles/storage.objectAdmin/serviceAccount:dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam.gserviceaccount.com]
google_secret_manager_secret_version.datasets_migrator_password: Refreshing state... [id=projects/41292283861/secrets/dfo-dev-datasets-migrator-password/versions/2]
google_sql_provision_script.presentation_revoke_public_connect: Refreshing state... [id=dev-fin-observatory/dfo-dev-platform-postgres]
google_sql_user.datasets_gc: Refreshing state... [id=dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam//dfo-dev-platform-postgres]
module.stack.data.google_compute_network.default: Reading...
module.stack.module.auth.random_id.api_key_suffix: Refreshing state... [id=8cdVBA]
module.stack.module.spanner[0].data.google_bigquery_default_service_account.bq_sa: Reading...
module.stack.module.ingestion_postprocessing_job[0].google_service_account.postprocessing_sa: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-pst-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.data.google_project.project: Reading...
module.stack.module.spanner[0].google_bigquery_reservation.default[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/reservations/default]
module.stack.module.ingestion_dataflow.google_service_account.dataflow_sa[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.auth.google_secret_manager_secret.dc_api_key: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-dc-api-key]
module.stack.module.ingestion_helper_service.google_service_account.helper_sa[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-hlp-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_spanner_instance.main[0]: Refreshing state... [id=dev-fin-observatory/dfo-dev-dc-instance]
module.stack.module.auth.google_secret_manager_secret.maps_api_key[0]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-dc-google-maps-api-key-f1c75504]
module.stack.module.auth.google_apikeys_key.maps_api_key[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/global/keys/dfo-dev-dc-google-maps-key-f1c75504]
ephemeral.random_password.presentation_migrator: Closing...
ephemeral.random_password.presentation_migrator: Closing complete after 0s
module.stack.module.spanner[0].google_bigquery_reservation_assignment.project_assignment[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/reservations/default/assignments/10971338433911481598]
module.stack.module.ingestion_helper_service.google_project_iam_member.helper_dataflow_viewer[0]: Refreshing state... [id=dev-fin-observatory/roles/dataflow.viewer/serviceAccount:dfo-dev-dc-ing-hlp-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_helper_service.google_project_iam_member.helper_spanner_user[0]: Refreshing state... [id=dev-fin-observatory/roles/spanner.databaseUser/serviceAccount:dfo-dev-dc-ing-hlp-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_postprocessing_job[0].google_project_iam_member.postprocessing_bq_job_user[0]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.jobUser/serviceAccount:dfo-dev-dc-ing-pst-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.spanner[0].data.google_bigquery_default_service_account.bq_sa: Read complete after 0s [id=bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.ingestion_postprocessing_job[0].google_project_iam_member.postprocessing_spanner[0]: Refreshing state... [id=dev-fin-observatory/roles/spanner.databaseUser/serviceAccount:dfo-dev-dc-ing-pst-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.data.google_compute_network.default: Read complete after 0s [id=projects/dev-fin-observatory/global/networks/default]
module.stack.module.ingestion_postprocessing_job[0].google_project_iam_member.postprocessing_bq_connection_user[0]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.connectionUser/serviceAccount:dfo-dev-dc-ing-pst-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_postprocessing_job[0].google_project_iam_member.postprocessing_bq_data_editor[0]: Refreshing state... [id=dev-fin-observatory/roles/bigquery.dataEditor/serviceAccount:dfo-dev-dc-ing-pst-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.google_project_iam_member.dataflow_worker[0]: Refreshing state... [id=dev-fin-observatory/roles/dataflow.worker/serviceAccount:dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.google_project_iam_member.ingestion_spanner_user[0]: Refreshing state... [id=dev-fin-observatory/roles/spanner.databaseUser/serviceAccount:dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.data.google_project.project: Read complete after 1s [id=projects/dev-fin-observatory]
module.stack.module.ingestion_helper_service.google_storage_bucket_iam_member.helper_bucket_access[0]: Refreshing state... [id=b/dfo-dev-dc-artifacts-dev-fin-observatory/roles/storage.objectAdmin/serviceAccount:dfo-dev-dc-ing-hlp-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.auth.google_secret_manager_secret_version.dc_api_key_version: Refreshing state... [id=projects/41292283861/secrets/dfo-dev-dc-api-key/versions/1]
module.stack.module.spanner[0].google_project_iam_member.bq_sa_spanner_viewer[0]: Refreshing state... [id=dev-fin-observatory/roles/spanner.viewer/serviceAccount:bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
google_sql_provision_script.datasets_revoke_public_connect: Refreshing state... [id=dev-fin-observatory/dfo-dev-platform-postgres]
module.stack.google_storage_bucket_iam_member.dataflow_bucket_access[0]: Refreshing state... [id=b/dfo-dev-dc-artifacts-dev-fin-observatory/roles/storage.objectAdmin/serviceAccount:dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_dataflow.google_service_account_iam_member.workflows_token_creator[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountTokenCreator/serviceAccount:service-41292283861@gcp-sa-workflows.iam.gserviceaccount.com]
module.stack.module.spanner[0].google_spanner_database.database[0]: Refreshing state... [id=dfo-dev-dc-instance/dfo-dev-dc-db]
ephemeral.random_password.datasets_migrator: Closing...
ephemeral.random_password.datasets_migrator: Closing complete after 0s
module.stack.module.auth.google_secret_manager_secret_version.maps_api_key_version[0]: Refreshing state... [id=projects/41292283861/secrets/dfo-dev-dc-google-maps-api-key-f1c75504/versions/1]
module.stack.module.spanner[0].google_bigquery_connection.spanner_connection[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/connections/dfo_dev_dc_spanner_connection]
module.stack.module.spanner[0].google_spanner_database_iam_member.spanner_reader[0]: Refreshing state... [id=dev-fin-observatory/dfo-dev-dc-instance/dfo-dev-dc-db/roles/spanner.databaseUser/serviceAccount:bq-41292283861@bigquery-encryption.iam.gserviceaccount.com]
module.stack.module.ingestion_helper_service.google_cloud_run_v2_service.ingestion_helper[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-ingestion-helper]
module.stack.module.ingestion_postprocessing_job[0].google_cloud_run_v2_job.dc_postprocessing_job: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-postprocessing-job]
module.stack.module.ingestion_preprocessing_job[0].google_service_account.preprocessing_sa: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_preprocessing_job[0].google_secret_manager_secret_iam_member.preprocessing_secret_accessor["MAPS_API_KEY"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-dc-google-maps-api-key-f1c75504/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_preprocessing_job[0].google_secret_manager_secret_iam_member.preprocessing_secret_accessor["DC_API_KEY"]: Refreshing state... [id=projects/dev-fin-observatory/secrets/dfo-dev-dc-api-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_storage_bucket_iam_member.preprocessing_bucket_access[0]: Refreshing state... [id=b/dfo-dev-dc-artifacts-dev-fin-observatory/roles/storage.objectAdmin/serviceAccount:dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_preprocessing_job[0].google_cloud_run_v2_job.dc_data_job: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job]
module.stack.module.datacommons_services[0].google_service_account.serving_sa: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/aiplatform.user"]: Refreshing state... [id=dev-fin-observatory/roles/aiplatform.user/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/redis.editor"]: Refreshing state... [id=dev-fin-observatory/roles/redis.editor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/workflows.invoker"]: Refreshing state... [id=dev-fin-observatory/roles/workflows.invoker/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/compute.networkViewer"]: Refreshing state... [id=dev-fin-observatory/roles/compute.networkViewer/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/spanner.databaseUser"]: Refreshing state... [id=dev-fin-observatory/roles/spanner.databaseUser/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/storage.objectViewer"]: Refreshing state... [id=dev-fin-observatory/roles/storage.objectViewer/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/vpcaccess.user"]: Refreshing state... [id=dev-fin-observatory/roles/vpcaccess.user/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/iam.serviceAccountUser"]: Refreshing state... [id=dev-fin-observatory/roles/iam.serviceAccountUser/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_project_iam_member.serving_sa_roles["roles/secretmanager.secretAccessor"]: Refreshing state... [id=dev-fin-observatory/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_workflow.google_service_account.workflow_sa[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_workflow.google_cloud_run_v2_service_iam_member.helper_invoker[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-ingestion-helper/roles/run.invoker/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.ingestion_workflow.google_workflows_workflow.ingestion_orchestrator[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/workflows/dfo-dev-dc-ingestion-workflow]
module.stack.module.ingestion_workflow.google_service_account_iam_member.workflow_act_as_dataflow_sa[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-df-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_project_iam_member.workflow_dataflow_developer[0]: Refreshing state... [id=dev-fin-observatory/roles/dataflow.developer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_invoker[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job/roles/run.invoker/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_viewer[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job/roles/run.viewer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_post_viewer[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-postprocessing-job/roles/run.viewer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_service_account_iam_member.workflow_pre_sa_user[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-pre-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_service_account_iam_member.workflow_post_sa_user[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-ing-pst-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_post_developer[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-postprocessing-job/roles/run.developer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_pre_developer[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-preprocessing-job/roles/run.developer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_storage_bucket_iam_member.workflow_bucket_access[0]: Refreshing state... [id=b/dfo-dev-dc-artifacts-dev-fin-observatory/roles/storage.objectAdmin/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_project_iam_member.workflow_invoker[0]: Refreshing state... [id=dev-fin-observatory/roles/workflows.invoker/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_cloud_run_v2_job_iam_member.workflow_post_invoker[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/jobs/dfo-dev-dc-ingestion-postprocessing-job/roles/run.invoker/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_service_account_iam_member.ingestion_workflow_act_as_serving_sa[0]: Refreshing state... [id=projects/dev-fin-observatory/serviceAccounts/dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.google_project_iam_member.workflow_run_viewer[0]: Refreshing state... [id=dev-fin-observatory/roles/run.viewer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service_iam_member.public_access[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service/roles/run.invoker/allUsers]
module.stack.google_cloud_run_v2_service_iam_member.workflow_serving_developer[0]: Refreshing state... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service/roles/run.developer/serviceAccount:dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com]
null_resource.search_api_gemini_key: Refreshing state... [id=7561527881215010758]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # google_secret_manager_secret_iam_member.serving_sa_gemini_key will be created
  + resource "google_secret_manager_secret_iam_member" "serving_sa_gemini_key" {
      + etag      = (known after apply)
      + id        = (known after apply)
      + member    = "serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com"
      + project   = "dev-fin-observatory"
      + role      = "roles/secretmanager.secretAccessor"
      + secret_id = "dfo-dev-gemini-api-key"
    }

  # null_resource.search_api_gemini_key must be replaced
-/+ resource "null_resource" "search_api_gemini_key" {
      ~ id       = "7561527881215010758" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "always_run" = "2026-10-23T18:03:34Z" -> (known after apply)
        }
    }

  # module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service will be updated in-place
  ~ resource "google_cloud_run_v2_service" "dc_web_service" {
        id                      = "projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service"
        name                    = "dfo-dev-dc-datacommons-service"
        # (36 unchanged attributes hidden)

      ~ template {
            # (11 unchanged attributes hidden)

          ~ containers {
                name             = null
                # (8 unchanged attributes hidden)

              - env {
                  - name  = "GEMINI_API_KEY" -> null
                    # (1 unchanged attribute hidden)

                  - value_source {
                      - secret_key_ref {
                          - secret  = "projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key" -> null
                          - version = "latest" -> null
                        }
                    }
                }

                # (27 unchanged blocks hidden)
            }

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

Plan: 2 to add, 1 to change, 1 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

Stack dcp · 985a5aa · @jm-rivera · run

@jm-rivera
jm-rivera merged commit 5a180ba into dev Oct 23, 2026
15 checks passed
@github-actions

Copy link
Copy Markdown

Terraform Apply — ai

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Init Validate Apply
pass pass pass
Show apply output
ephemeral.random_password.studio_api_token: Opening...
ephemeral.random_password.mcp_api_token: Opening...
ephemeral.random_password.studio_api_token: Opening complete after 0s
ephemeral.random_password.mcp_api_token: Opening complete after 0s
ephemeral.random_password.studio_api_token: Closing...
ephemeral.random_password.mcp_api_token: Closing...
ephemeral.random_password.mcp_api_token: Closing complete after 0s
ephemeral.random_password.studio_api_token: Closing complete after 0s

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

runtime_service_account_email = "dfo-dev-ai-run@dev-fin-observatory.iam.gserviceaccount.com"
service_name = "dfo-dev-ai"
service_url = "https://dfo-dev-ai-zms5lj4zhq-nn.a.run.app"
studio_api_token_secret = "dfo-dev-studio-api-token"

Stack ai · 5a180ba · @jm-rivera · run

@github-actions

Copy link
Copy Markdown

Terraform Apply — dcp

Apply complete! Resources: 2 added, 1 changed, 1 destroyed.

Init Validate Apply
pass pass pass
Show apply output
ephemeral.random_password.presentation_migrator: Opening...
ephemeral.random_password.datasets_migrator: Opening...
null_resource.search_api_gemini_key: Destroying... [id=7561527881215010758]
null_resource.search_api_gemini_key: Destruction complete after 0s
ephemeral.random_password.presentation_migrator: Opening complete after 0s
ephemeral.random_password.datasets_migrator: Opening complete after 0s
ephemeral.random_password.datasets_migrator: Closing...
ephemeral.random_password.presentation_migrator: Closing...
ephemeral.random_password.datasets_migrator: Closing complete after 0s
ephemeral.random_password.presentation_migrator: Closing complete after 0s
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Modifying... [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m10s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m20s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Still modifying... [id=projects/dev-fin-observatory/locations/...ervices/dfo-dev-dc-datacommons-service, 00m30s elapsed]
module.stack.module.datacommons_services[0].google_cloud_run_v2_service.dc_web_service: Modifications complete after 31s [id=projects/dev-fin-observatory/locations/northamerica-northeast1/services/dfo-dev-dc-datacommons-service]
google_secret_manager_secret_iam_member.serving_sa_gemini_key: Creating...
google_secret_manager_secret_iam_member.serving_sa_gemini_key: Creation complete after 4s [id=projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key/roles/secretmanager.secretAccessor/serviceAccount:dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com]
null_resource.search_api_gemini_key: Creating...
null_resource.search_api_gemini_key: Provisioning with 'local-exec'...
null_resource.search_api_gemini_key (local-exec): Executing: ["/bin/sh" "-c" "\"./../../../scripts/set-search-api-gemini-key.sh\" \\\n  dev-fin-observatory \\\n  northamerica-northeast1 \\\n  dfo-dev-dc-datacommons-service \\\n  dfo-dev-gemini-api-key\n"]
null_resource.search_api_gemini_key: Still creating... [00m10s elapsed]
null_resource.search_api_gemini_key: Still creating... [00m20s elapsed]
null_resource.search_api_gemini_key: Still creating... [00m30s elapsed]
null_resource.search_api_gemini_key (local-exec): GEMINI_API_KEY set on dfo-dev-dc-datacommons-service from projects/dev-fin-observatory/secrets/dfo-dev-gemini-api-key
null_resource.search_api_gemini_key: Creation complete after 30s [id=7603058370291887627]

Apply complete! Resources: 2 added, 1 changed, 1 destroyed.

Outputs:

datacommons_service_name = "dfo-dev-dc-datacommons-service"
datacommons_service_service_account_email = "dfo-dev-dc-srvs-sa@dev-fin-observatory.iam.gserviceaccount.com"
datacommons_service_url = "https://dfo-dev-dc-datacommons-service-zms5lj4zhq-nn.a.run.app"
datasets_database_name = "datasets"
datasets_gc_db_user = "dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam"
datasets_gc_service_account_email = "dfo-dev-dc-datasets-gc-sa@dev-fin-observatory.iam.gserviceaccount.com"
datasets_instance_connection_name = "dev-fin-observatory:northamerica-northeast1:dfo-dev-platform-postgres"
datasets_migrator_secret_id = "projects/dev-fin-observatory/secrets/dfo-dev-datasets-migrator-password"
datasets_migrator_secret_name = "dfo-dev-datasets-migrator-password"
datasets_pinned_bucket_name = "dev-fin-observatory-datasets-pinned"
datasets_working_bucket_name = "dev-fin-observatory-datasets-working"
ingestion_bucket_url = "gs://dfo-dev-dc-artifacts-dev-fin-observatory"
ingestion_input_path = "ingestion/input"
ingestion_prep_job_name = "dfo-dev-dc-ingestion-preprocessing-job"
ingestion_service_url = "https://dfo-dev-dc-ingestion-helper-zms5lj4zhq-nn.a.run.app"
ingestion_workflow_id = "projects/dev-fin-observatory/locations/northamerica-northeast1/workflows/dfo-dev-dc-ingestion-workflow"
ingestion_workflow_name = "dfo-dev-dc-ingestion-workflow"
ingestion_workflow_service_account_email = "dfo-dev-dc-ing-wf-sa@dev-fin-observatory.iam.gserviceaccount.com"
presentation_database_name = "presentation"
presentation_migrator_secret_id = "projects/dev-fin-observatory/secrets/dfo-dev-presentation-migrator-password"
presentation_migrator_secret_name = "dfo-dev-presentation-migrator-password"
project_id = "dev-fin-observatory"
region = "northamerica-northeast1"
spanner_database_id = "dfo-dev-dc-db"
spanner_instance_id = "dfo-dev-dc-instance"
storage_artifacts_bucket_name = "dfo-dev-dc-artifacts-dev-fin-observatory"

Stack dcp · 5a180ba · @jm-rivera · run

This branch was successfully deployed

1 active deployment
dev-plan — 985a5aa1 Deployed Oct 23, 2026 by jm-rivera via plan (ai) #119
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant