Skip to content

cvd: Reconcile stale instance states and prune deleted group directories - #3609

Open
rmuthiah wants to merge 4 commits into
google:mainfrom
rmuthiah:fix-cvd-state-reconciliation
Open

rmuthiah wants to merge 4 commits into
google:mainfrom
rmuthiah:fix-cvd-state-reconciliation

Conversation

@rmuthiah

@rmuthiah rmuthiah commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #3606, #3607, and #3608 (part 4 of 5; review just this patch's commit).

stop_cvd terminates run_cvd directly over launcher_monitor.sock without updating instance_database.binpb, leaving stopped groups stuck in INSTANCE_STATE_RUNNING and leaving deleted slot directories behind as dangling symlinks holding instance ID locks.

Reconcile database state against live host state and harden group lifecycle transitions:

  1. In-flight transition lock (<BaseDir>/in_progress.lock): Protect PREPARING (cvd load) and STARTING (cvd start) group transitions with AcquireGroupInProgressLock (LOCK_EX | LOCK_NB) and IsGroupTransitionInProgress (LOCK_SH | LOCK_NB), transferring the open lock fd into CvdStartCommandHandler so concurrent read/prune probes never clobber a preparing or booting group.
  2. Live state reconciliation: Add RefreshInstanceGroupState and batched RefreshInstanceGroups (ProbeGroupStateUpdates + BatchCompareAndUpdateInstanceStates), checking CanConnectToLauncherMonitor and active run_cvd processes (caching /proc scans across groups and invalidating on slow socket probes) to reconcile stale active/inactive states under conditional database CAS.
  3. Broken-group pruning: Add PruneBrokenInstanceGroups to remove inactive groups whose backing HomeDir, HostArtifactsPath start binary, or ProductOutPath directories were deleted on disk, releasing their instance ID locks.
  4. Lock ordering & atomic numeric BaseDir allocation: Acquire requested instance locks in ascending ID order in AllocateAndLockInstanceIds, atomically claim unique numeric <base_root_dir>/<timestamp> group directories (mkdir/symlink retrying on EEXIST while keeping pure-digit basenames compatible with podcvd path translation), roll back newly allocated locks on group-creation failure, reject duplicate canonical RealPath(HOME) groups in InstanceDatabase::AddInstanceGroup, and refresh group state in cvd start, cvd stop, cvd remove, and cvd reset.

Direct dependencies added to BUILD.bazel targets:

  • //cuttlefish/host/commands/cvd/instances:instances (instances/BUILD.bazel):
    • //cuttlefish/posix:realpath: instance_database.cpp:37 includes "cuttlefish/posix/realpath.h" (for RealPath()); previously pulled in transitively via :config_path.
  • //cuttlefish/host/commands/cvd/instances:instance_manager (instances/BUILD.bazel):
    • //cuttlefish/common/libs/fs:fd: instance_manager.cpp:45 includes "cuttlefish/common/libs/fs/fd.h" (for Fd::Open()); previously pulled in transitively via :instances, :data_viewer, and //cuttlefish/host/commands/cvd/instances/lock.
    • //cuttlefish/files:file_exists: instance_manager.cpp:49 includes "cuttlefish/files/file_exists.h" (for FileExists()); previously pulled in transitively via :instances, :config_path, :run_cvd_proc_collector, and :stop.
    • //cuttlefish/posix:realpath: instance_manager.cpp:62 includes "cuttlefish/posix/realpath.h" (for RealPath()); previously pulled in transitively via :config_path.
  • //cuttlefish/host/commands/cvd/cli:nesting_commands (cli/BUILD.bazel):
    • //cuttlefish/common/libs/fs: load_configs.h:26 and load_configs.cpp:29 include "cuttlefish/common/libs/fs/shared_fd.h" (for SharedFD); previously pulled in transitively via //cuttlefish/host/commands/cvd/instances and //cuttlefish/host/commands/cvd/cli/commands:start.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...

`DataViewer::WithExclusiveLock` passed `LOCK_SH` instead of `LOCK_EX` to
`LockBackingFile`, allowing concurrent `cvd` writers to hold shared locks
simultaneously while truncating (`ftruncate(0)`) and rewriting
`instance_database.binpb`, which could interleave writes and corrupt the
serialized `cvd::PersistentData` protobuf.

Acquire `LOCK_EX` in `WithExclusiveLock`, pass `O_CLOEXEC` when opening
the backing file so child daemons do not inherit an open database file
descriptor across `exec`, centralize the signal-masked overwrite helper
in `DataViewer::StoreData`, and add `WithConditionalExclusiveLock` so
read-modify-write callers that leave `data` unchanged can skip
truncating and rewriting the backing file.

Add `data_viewer_test` (`instances/BUILD.bazel`) to verify `LOCK_EX`
mutual exclusion, concurrent writer serialization, and no-op conditional
locks.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/...
Fix two `SIGABRT` crash paths triggered during instance probing and
slot-directory reuse:

1. `LocalInstance::GetLauncherMonitor` called `SharedFD::SocketLocalClient`
   unconditionally on `uds_dir + "/launcher_monitor.sock"`. When
   `.cuttlefish_config.json` was missing or unreadable and the fallback
   `InstanceDirectory()` path exceeded 107 bytes, `SocketLocalClient`
   aborted at `CHECK_LE(namelen, 108)`. Tolerate missing JSON configs,
   verify `FileExists(monitor_path)` and
   `monitor_path.size() < sizeof(sockaddr_un::sun_path)` before
   connecting, and expose `LocalInstance::CanConnectToLauncherMonitor`
   for liveness checks.

2. `GatherVmStartMetrics` in `device_metrics_orchestration.cc` invoked
   `RunMetrics` -> `CreateLogger(metrics_directory)` without ensuring
   `<HomeDir>/metrics` existed first, and `RunMetrics` opened
   `metrics_v2.log` before checking `FileExists(metrics_directory)`.
   When a caller wiped `$HOME/*` between runs and restarted the group,
   `SeverityTarget::FromFile` hit `LOG(FATAL)` (`SIGABRT`). Run
   `SetUpMetrics(instance_group.MetricsDir())` unconditionally in both
   `GatherVmInstantiationMetrics` and `GatherVmStartMetrics`, verify
   `FileExists` before `CreateLogger` in `RunMetrics`, and skip
   `GetDeviceMetricsInput` / `GatherDeviceMetrics` when
   `!AreMetricsEnabled()` so `<HomeDir>/metrics` is always initialized
   while avoiding unnecessary host/GCE probes when metrics transmission
   is disabled.

Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/metrics:device_metrics_orchestration` (`metrics/BUILD.bazel`):
  - `//cuttlefish/metrics/io:enabled`: `device_metrics_orchestration.cc:34`
    directly `#include`s `"cuttlefish/metrics/io/enabled.h"` (for
    `AreMetricsEnabled()`); previously pulled in transitively via
    `//cuttlefish/metrics:metrics_orchestration`.
- `//cuttlefish/host/commands/cvd/instances:local_instance_test` (`instances/BUILD.bazel`):
  - `:cvd_persistent_data`, `:data_viewer`, `//cuttlefish/common/libs/fs:fd`,
    `//cuttlefish/common/libs/utils:files`,
    `//cuttlefish/files:directory_exists`,
    `//cuttlefish/files:recursively_remove_directory`,
    `//cuttlefish/files:write_new_file`,
    `//cuttlefish/metrics:device_metrics_orchestration`,
    `//cuttlefish/result`, `//cuttlefish/result:result_matchers`, and
    `@abseil-cpp//absl/strings` for the new monitor-socket and metrics-dir
    regression tests in `local_instance_test.cpp:24-39`.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/... //cuttlefish/metrics/...
Follow up on 2545596 (`cvd: Use 0777 permissions on shared instance
locks directory`) to fix three multi-user and concurrency hazards on
shared `/var/tmp/cvd/lock/local-instance-<N>.lock` files:

1. `LockFileManager::OpenLockFile` called `open(O_CREAT | O_RDWR, 0666)`
   even when the lock file already existed. On kernels with
   `fs.protected_regular = 2` (default on Debian 13), opening an
   existing regular file owned by another UID inside a world-writable
   (`0777`) directory with `O_CREAT` without `O_EXCL` fails with
   `EACCES`. Open existing lock files with `O_RDWR | O_CLOEXEC` first,
   and only fall back to `O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC` (using
   `fchmod(0666)` on the newly created fd and skipping `chmod(parent,
   0777)` when `parent_dir` is owned by another UID).

2. `AcquireLock` and `TryAcquireLock` did not re-check that the locked
   fd still matched the path's inode (`fstat` vs `stat`), allowing a
   concurrent `RemoveLockFile` / `DeleteLockFile` to unlink a locked
   inode and let a second caller create and lock a distinct inode for
   the same instance ID simultaneously. Retry if `IsSameInode` fails
   after `flock(LOCK_EX)`.

3. Add `AcquireCurrentUserLock`, `AcquireOwnedLockIfPresent`,
   `ReleaseHeldLock`, and `ReleaseInstanceLock` on
   `InstanceLockFileManager`, and update `AcquireUnusedLock` to reject
   active (`kInUse`) locks held by other UIDs while replacing stale
   (`kNotInUse`) foreign-UID lock files with fresh inodes owned by
   `geteuid()` so subsequent `fchmod` status updates cannot fail with
   `EPERM`.

Add `instance_lock_test` (`instances/lock/BUILD.bazel`) covering
`O_CLOEXEC`, concurrent `AcquireLock` + `RemoveLockFile` inode races,
and `AcquireCurrentUserLock` / `ReleaseInstanceLock` semantics.

Bug: b/554100008
Bug: b/584825999
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/lock/...
`stop_cvd` terminates `run_cvd` directly over `launcher_monitor.sock`
without updating `instance_database.binpb`, leaving stopped groups stuck
in `INSTANCE_STATE_RUNNING` and leaving deleted slot directories behind
as dangling symlinks holding instance ID locks.

Reconcile database state against live host state and harden group
lifecycle transitions:

1. Protect in-flight `PREPARING` (`cvd load`) and `STARTING` (`cvd start`)
   group transitions with an exclusive per-group `<BaseDir>/in_progress.lock`
   (`AcquireGroupInProgressLock` / `IsGroupTransitionInProgress`),
   transferring the open lock fd into `CvdStartCommandHandler` so
   concurrent read/prune probes (`LOCK_SH | LOCK_NB`) never clobber a
   preparing or booting group.

2. Add `RefreshInstanceGroupState` and batched `RefreshInstanceGroups`
   (`ProbeGroupStateUpdates` + `BatchCompareAndUpdateInstanceStates`),
   checking `CanConnectToLauncherMonitor` and active `run_cvd` processes
   (caching `/proc` scans across groups and invalidating on slow socket
   probes) to reconcile stale active/inactive states under conditional
   database CAS.

3. Add `PruneBrokenInstanceGroups` to remove inactive groups whose
   backing `HomeDir`, `HostArtifactsPath` start binary, or
   `ProductOutPath` directories were deleted on disk, releasing their
   instance ID locks.

4. Acquire requested instance locks in ascending ID order in
   `AllocateAndLockInstanceIds`, atomically claim unique numeric
   `<base_root_dir>/<timestamp>` group directories (`mkdir`/`symlink`
   retrying on `EEXIST` while keeping pure-digit basenames compatible
   with `podcvd` path translation), roll back newly allocated locks on
   group-creation failure, reject duplicate canonical `RealPath(HOME)`
   groups in `InstanceDatabase::AddInstanceGroup`, and refresh group
   state in `cvd start`, `cvd stop`, `cvd remove`, and `cvd reset`.

Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/host/commands/cvd/instances:instances` (`instances/BUILD.bazel`):
  - `//cuttlefish/posix:realpath`: `instance_database.cpp:37` includes
    `"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
    in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/instances:instance_manager` (`instances/BUILD.bazel`):
  - `//cuttlefish/common/libs/fs:fd`: `instance_manager.cpp:45` includes
    `"cuttlefish/common/libs/fs/fd.h"` (for `Fd::Open()`); previously
    pulled in transitively via `:instances`, `:data_viewer`, and
    `//cuttlefish/host/commands/cvd/instances/lock`.
  - `//cuttlefish/files:file_exists`: `instance_manager.cpp:49` includes
    `"cuttlefish/files/file_exists.h"` (for `FileExists()`); previously
    pulled in transitively via `:instances`, `:config_path`,
    `:run_cvd_proc_collector`, and `:stop`.
  - `//cuttlefish/posix:realpath`: `instance_manager.cpp:62` includes
    `"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
    in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/cli:nesting_commands` (`cli/BUILD.bazel`):
  - `//cuttlefish/common/libs/fs`: `load_configs.h:26` and
    `load_configs.cpp:29` include
    `"cuttlefish/common/libs/fs/shared_fd.h"` (for `SharedFD`);
    previously pulled in transitively via
    `//cuttlefish/host/commands/cvd/instances` and
    `//cuttlefish/host/commands/cvd/cli/commands:start`.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant