chore(dcp): restore prod stateless deletion protection

docs/development.md

@@ -88,7 +88,7 @@ cd infra/dfo-dev/dcp
 terraform init && terraform apply
 ```
 
-After `apply` completes, grant the operator `roles/iam.serviceAccountTokenCreator` on the ingestion workflow service account (printed in the Terraform outputs) so the CLI can impersonate it for `init-db` and `ingest`:
+After `apply` completes, grant the operator `roles/spanner.databaseAdmin` and `roles/spanner.databaseUser` on the dcp database for `init-db` and `migrate-db`, and `roles/iam.serviceAccountTokenCreator` on the ingestion workflow service account (printed in the Terraform outputs) so the CLI can impersonate it for `ingest`:
 
 ```bash
 gcloud iam service-accounts add-iam-policy-binding \
@@ -116,7 +116,7 @@ For a break-glass deploy, `just deploy <image>` applies an image directly agains
 ### Upgrading the DCP module
 
 ```bash
-just dcp-scaffold 1.1.5
+just dcp-scaffold 1.1.6
 ```
 
 The recipe regenerates `main.tf`, `variables.tf`, `outputs.tf`, and `README.md` from the upstream CLI at the given version. It copies the three `.tf` files over the tracked files in `infra/stacks/dcp`, which every environment's dcp root links to, and `README.md` into `infra/<ENV>/dcp`. It leaves `backend.tf` and `terraform.tfvars` untouched, because the CLI's own `backend.tf` assumes one stack per instance and would point this stack at empty state. [infra/dfo-dev/dcp/dfo.README.md](../infra/dfo-dev/dcp/dfo.README.md) explains why. It fails on uncommitted edits to `backend.tf` or any `dfo.*` file, so those do not blend into the diff it prints. It also checks the file set the CLI emitted against the expected six, which catches upstream drift the copy loop would otherwise drop silently.

infra/dfo-dev/dcp/dfo.README.md

@@ -36,37 +36,32 @@ Two resources appear in every plan. `null_resource.search_api_gemini_key` carrie
 
 ## The database commands
 
-Four `just` recipes touch the dcp database. Terraform grants none of the IAM they need, so grant it by hand before the first run.
+Three `just` recipes touch the dcp database. Terraform grants none of the IAM they need, so grant it by hand before the first run.
 
-| Recipe                  | What it does                                                                | Spanner role on your principal | Token creator on the workflow account |
-| ----------------------- | --------------------------------------------------------------------------- | ------------------------------ | ------------------------------------- |
-| `just init-db`          | Creates the schema on a brand-new database                                  | `roles/spanner.databaseReader` | yes                                   |
-| `just migrate-db`       | Applies pending Spanner schema migrations                                   | `roles/spanner.databaseAdmin`  | yes                                   |
-| `just seed-db`          | Re-applies Base DC's own geographic and schema seeds                        | none                           | yes                                   |
-| `just load data/<name>` | Stages one of this repo's datasets into the ingestion bucket and ingests it | none                           | yes                                   |
+| Recipe                  | What it does                                                                | Spanner roles on your principal                             | Token creator on the workflow account |
+| ----------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------- |
+| `just init-db`          | Creates the schema on a brand-new database                                  | `roles/spanner.databaseAdmin`, `roles/spanner.databaseUser` | no                                    |
+| `just migrate-db`       | Applies pending Spanner schema migrations                                   | `roles/spanner.databaseAdmin`, `roles/spanner.databaseUser` | no                                    |
+| `just load data/<name>` | Stages one of this repo's datasets into the ingestion bucket and ingests it | none                                                        | yes                                   |
 
 `just datasets` lists the values `load` accepts. `load` reaches the bucket named by the `storage_artifacts_bucket_name` output under your own credentials rather than the workflow account's, and it needs object write and object delete there.
 
 A permission error from any of these names the resource it denied rather than the principal it denied, which sends you to the wrong service account. Read the principal off the table first.
 
-### init-db is safe only on a fresh database
+### init-db on an existing database
 
-`init-db`'s already-initialised check fails open. Any exception from the Spanner client or the table lookup returns "not initialised", and a permissions error is the default case, since Terraform grants `roles/spanner.databaseReader` to no principal. On that path `database/initialize` still fires against the live database. It runs under the impersonated workflow service account, so a 403 on your own principal does nothing to stop it. Nothing outside the ingestion-helper image documents what that call does to a populated database.
-
-The recipe passes `--init-only` and puts a `[confirm]` prompt in front of it. Answer that prompt only for a database you know is empty.
+`init-db` skips a database whose `Node` table exists. If that check errors, the CLI treats the database as empty and applies the baseline schema under your own credentials. The schema's first statement, `CREATE TABLE Node`, then fails against an existing database before any DDL applies.
 
 ### Before running migrate-db
 
 Take a Spanner backup. Migrations are forward-only, the framework has no `downgrade()`, and Spanner has no transactional DDL rollback. The CLI asks whether you have taken a backup and does not take one for you.
 
-Then wait for a `terraform apply` on this stack to finish. `apply.yml` runs asynchronously, so a merged PR leaves the apply still in flight. The ingestion-helper has to reach the matching image before the lock endpoints exist, and these commands read their targets from `terraform output -json`, which stays empty until the apply populates state.
+Then wait for a `terraform apply` on this stack to finish. `apply.yml` runs asynchronously, so a merged PR leaves the apply still in flight. These commands read their targets from `terraform output -json`, which stays empty until the apply populates state. `migrate-db` takes the ingestion lock in Spanner and fails while an ingestion holds it.
 
 ### load replaces the whole input prefix
 
 Ingestion reads every file under `gs://<bucket>/ingestion/input/` against a single `config.json`, so `load` clears that prefix before uploading. The clear runs under your own credentials and discards its own errors. Without object delete on the bucket it does nothing, the new dataset lands on top of the old files, and ingestion reads both.
 
-`seed-db` and `load` both add data. `seed-db` re-applies Data Commons' own base seeds. `load` imports the datasets under `data/`. `seed-db` has no already-seeded check and no confirmation prompt, so it reseeds the moment you run it.
-
 ## Migrator password rotation and recovery
 
 `datasets_migrator_password_version` in `dfo_datasets.tf` and `presentation_migrator_password_version` in `dfo_presentation.tf` each control the write-only password for a Secret Manager version and Cloud SQL migrator user. For normal rotation, increment the relevant counter in a PR. After the PR merges, CI applies the whole dcp stack.

infra/dfo-dev/dcp/dfo.auto.tfvars

@@ -37,8 +37,9 @@ stateless_deletion_protection = false
 # dfo_storage.tf declares the artifacts bucket so it can carry lifecycle
 # rules, which the upstream module has no input for. The module receives
 # the name and passes it through to its consumers.
-storage_create_artifacts_bucket = false
-storage_artifacts_bucket_name   = "dfo-dev-dc-artifacts-dev-fin-observatory"
+storage_create_artifacts_bucket            = false
+storage_artifacts_bucket_name              = "dfo-dev-dc-artifacts-dev-fin-observatory"
+storage_artifacts_bucket_enable_versioning = false
 
 # =============================================================================
 # Redis Module

infra/dfo-prod/dcp/dfo.auto.tfvars

@@ -19,8 +19,9 @@ stateless_deletion_protection = true
 # =============================================================================
 # Storage Module
 # =============================================================================
-storage_create_artifacts_bucket = false
-storage_artifacts_bucket_name   = "dfo-prod-dc-artifacts-prod-fin-observatory"
+storage_create_artifacts_bucket            = false
+storage_artifacts_bucket_name              = "dfo-prod-dc-artifacts-prod-fin-observatory"
+storage_artifacts_bucket_enable_versioning = false
 
 # =============================================================================
 # Redis Module

infra/stacks/dcp/dfo.tf

@@ -37,13 +37,24 @@ resource "null_resource" "dc_key_guard" {
   }
 }
 
+# The serving SA reads the Gemini key from Secret Manager as
+# "${namespace}-gemini-api-key" through this per-secret grant, so the access
+# holds whether or not the stack module grants secretmanager.secretAccessor
+# project-wide.
+resource "google_secret_manager_secret_iam_member" "serving_sa_gemini_key" {
+  project   = var.project_id
+  secret_id = "${var.namespace}-gemini-api-key"
+  role      = "roles/secretmanager.secretAccessor"
+  member    = "serviceAccount:${module.stack.datacommons_service_service_account_email}"
+}
+
 # Wire GEMINI_API_KEY onto the datacommons service so the ONE/DFO Search API
 # boots. Without it, custom_dc/api_server/start.sh deliberately skips launching
 # the FastAPI engine (it warm-loads Gemini at startup and would restart-loop),
 # so nginx returns 502 on every /api/search/* call while the DC site is fine.
 #
-# The key lives in Secret Manager as "${namespace}-gemini-api-key", which the
-# serving SA can already read through its project-level secretmanager.secretAccessor.
+# The key lives in Secret Manager as "${namespace}-gemini-api-key", and the
+# serving SA reads it through the per-secret secretAccessor grant above.
 #
 # The pinned stack module takes no input for adding secret env vars, so the key
 # goes on after the module deploys the service. The script header explains why it
@@ -70,5 +81,8 @@ resource "null_resource" "search_api_gemini_key" {
     EOT
   }
 
-  depends_on = [module.stack]
+  depends_on = [
+    module.stack,
+    google_secret_manager_secret_iam_member.serving_sa_gemini_key,
+  ]
 }

infra/stacks/dcp/dfo_override.tf

@@ -9,8 +9,9 @@
 #
 # What it does: rewrites ONLY module.stack's auth_config, swapping the empty
 # var-sourced DC API key for the value read from Secret Manager in dfo.tf. The
-# other two auth_config fields (google_maps_api_key, create_google_maps_key)
-# are preserved from local.auth_config via merge().
+# other auth_config fields (google_maps_api_key, create_google_maps_key,
+# google_maps_allowed_referrers) are preserved from local.auth_config via
+# merge().
 #
 # If you regenerate main.tf, this file and dfo.tf survive untouched and the
 # redirection keeps working. See dfo.README.md for the full story.

infra/stacks/dcp/dfo_storage.tf

@@ -4,6 +4,13 @@
 # optional list, so rules added out of band on a module-owned bucket are
 # stripped on the module's next apply. The moved block below re-parents
 # the module's existing bucket onto this resource in place.
+#
+# Object versioning is off for this bucket. The module's
+# storage_artifacts_bucket_enable_versioning flag only reaches the bucket
+# the module creates, which storage_create_artifacts_bucket = false
+# suppresses, so it is set to false in each environment's dfo.auto.tfvars
+# and has no effect here. Versioning for this bucket would need a
+# versioning block on the resource below.
 resource "google_storage_bucket" "dc_artifacts" {
   name                        = var.storage_artifacts_bucket_name
   project                     = var.project_id

infra/stacks/dcp/main.tf

@@ -40,14 +40,14 @@ resource "google_project_service" "apis" {
     "sqladmin.googleapis.com",
     "redis.googleapis.com",
     "secretmanager.googleapis.com",
-    "vpcaccess.googleapis.com",
     "artifactregistry.googleapis.com",
     "compute.googleapis.com"
     ], var.enable_spanner ? ["spanner.googleapis.com"] : [],
     var.enable_ingestion ? [
       "workflows.googleapis.com",
       "workflowexecutions.googleapis.com",
-      "dataflow.googleapis.com"
+      "dataflow.googleapis.com",
+      "batch.googleapis.com"
     ] : [],
     var.spanner_enable_bigquery_connection ? [
       "bigqueryconnection.googleapis.com",
@@ -107,10 +107,26 @@ locals {
     website_search_scope            = var.datacommons_services_website_search_scope
   }
 
+  network_config = {
+    enable                    = var.enable_network
+    enable_workload_vpc       = var.enable_workload_vpc
+    prune_cloud_run_revisions = var.network_prune_cloud_run_revisions
+    create_vpc                = var.network_create_vpc
+    network_name              = var.network_name != "" ? var.network_name : "dc-vpc"
+    subnet_cidr               = var.network_subnet_cidr
+    enable_cloud_nat          = var.network_enable_cloud_nat
+    enable_flow_logs          = var.network_enable_flow_logs
+    flow_sampling             = var.network_flow_sampling
+    existing_network_id       = var.network_existing_network_id
+    existing_subnet_id        = var.network_existing_subnet_id
+    vpc_egress_mode           = var.network_vpc_egress_mode
+  }
+
   auth_config = {
-    google_datacommons_api_key = var.auth_google_datacommons_api_key
-    google_maps_api_key        = var.auth_google_maps_api_key
-    create_google_maps_key     = var.auth_create_google_maps_api_key
+    google_datacommons_api_key    = var.auth_google_datacommons_api_key
+    google_maps_api_key           = var.auth_google_maps_api_key
+    create_google_maps_key        = var.auth_create_google_maps_api_key
+    google_maps_allowed_referrers = var.auth_google_maps_allowed_referrers
   }
 
   redis_config = {
@@ -121,8 +137,8 @@ locals {
     location_id             = var.redis_location_id
     alternative_location_id = var.redis_alternative_location_id
     replica_count           = var.redis_replica_count
-    vpc_network_name        = var.redis_vpc_network_name
-    vpc_connector_cidr      = var.redis_vpc_connector_cidr
+    enable_auth             = var.redis_enable_auth
+    enable_tls              = var.redis_enable_tls
   }
 
   ingestion_config = {
@@ -150,29 +166,34 @@ locals {
     workflow_lock_acquisition_timeout = var.ingestion_workflow_lock_acquisition_timeout
     helper_service_image              = coalesce(var.ingestion_helper_service_image, "gcr.io/datcom-ci/datacommons-ingestion-helper:${var.dcp_version}")
 
-    # Dataflow Network & Scaling Configuration
-    dataflow_ip_configuration    = var.ingestion_dataflow_ip_configuration
-    dataflow_subnetwork          = var.ingestion_dataflow_subnetwork
-    dataflow_template_gcs_path   = coalesce(var.ingestion_dataflow_template_gcs_path, "gs://datcom-templates/templates/flex/ingestion-${local.df_template_version}.json")
-    dataflow_max_workers         = var.ingestion_dataflow_max_workers
-    dataflow_num_workers         = var.ingestion_dataflow_num_workers
-    dataflow_worker_machine_type = var.ingestion_dataflow_worker_machine_type
+    # Worker Network Configuration (Dataflow & Cloud Batch)
+    worker_ip_configuration = var.ingestion_worker_ip_configuration
+    worker_subnetwork       = var.ingestion_worker_subnetwork
+
+    # Ingestion Dataflow Template & Worker Configuration
+    ingestion_dataflow_template_gcs_path = coalesce(var.ingestion_dataflow_template_gcs_path, "gs://datcom-templates/templates/flex/ingestion-${local.df_template_version}.json")
+    rollback_dataflow_template_gcs_path  = coalesce(var.ingestion_rollback_dataflow_template_gcs_path, "gs://datcom-templates/templates/flex/rollback/rollback-${local.df_template_version}.json")
+    dataflow_max_workers                 = var.ingestion_dataflow_max_workers
+    dataflow_num_workers                 = var.ingestion_dataflow_num_workers
+    dataflow_worker_machine_type         = var.ingestion_dataflow_worker_machine_type
   }
 }
 
 module "stack" {
   # DO NOT CHANGE the format of the source line below.
-  # The Data Commons CLI relies on matching 'source = "git::https://github.com/datacommonsorg/datacommons.git//infra/dcp/modules/stack?ref=v1.1.5"' to generate user scaffolding.
-  source = "git::https://github.com/datacommonsorg/datacommons.git//infra/dcp/modules/stack?ref=v1.1.5"
-
-  global                          = local.global_config
-  spanner_config                  = local.spanner_config
-  storage_create_artifacts_bucket = var.storage_create_artifacts_bucket
-  storage_artifacts_bucket_name   = var.storage_artifacts_bucket_name
-  datacommons_services_config     = local.datacommons_services_config
-  auth_config                     = local.auth_config
-  redis_config                    = local.redis_config
-  ingestion_config                = local.ingestion_config
+  # The Data Commons CLI relies on matching 'source = "git::https://github.com/datacommonsorg/datacommons.git//infra/dcp/modules/stack?ref=v1.1.6"' to generate user scaffolding.
+  source = "git::https://github.com/datacommonsorg/datacommons.git//infra/dcp/modules/stack?ref=v1.1.6"
+
+  global                                     = local.global_config
+  network_config                             = local.network_config
+  spanner_config                             = local.spanner_config
+  storage_create_artifacts_bucket            = var.storage_create_artifacts_bucket
+  storage_artifacts_bucket_name              = var.storage_artifacts_bucket_name
+  storage_artifacts_bucket_enable_versioning = var.storage_artifacts_bucket_enable_versioning
+  datacommons_services_config                = local.datacommons_services_config
+  auth_config                                = local.auth_config
+  redis_config                               = local.redis_config
+  ingestion_config                           = local.ingestion_config
 
   depends_on = [google_project_service.apis]
 }