Skip to content

cvd: Scope create --reuse by slot and preserve default-HOME userdata - #3602

Open
rmuthiah wants to merge 5 commits into
google:mainfrom
rmuthiah:fix-cvd-reuse-groups
Open

rmuthiah wants to merge 5 commits into
google:mainfrom
rmuthiah:fix-cvd-reuse-groups

Conversation

@rmuthiah

@rmuthiah rmuthiah commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #3609 (part 5 of 5; review just this patch's commit).

When multiple Cuttlefish instances run concurrently under the same host UID (such as multi-slot runners passing distinct HOME directories, --base_instance_num, or --instance_name), cvd create --reuse=true queried instance_manager_.FindGroups({}) across all groups owned by the UID and either failed with "Trying to change the host paths on an existing group" (when 1 other slot existed) or "Unclear which group to reuse" (when 2+ slots existed).

Scope --reuse=true candidate lookup to the target slot:

  1. Slot-scoped FindReusableGroup: Prune deleted/broken inactive groups before group creation or reuse, and isolate FindReusableGroup by explicit --group_name selector first, custom HOME directory second (GroupCustomHome, tolerating wiped/dangling custom-HOME symlinks via ReadLink), and requested instance IDs or per-instance names third, falling back to default-HOME groups only when running in the system-wide default HOME.
  2. Custom-HOME replacement vs default-HOME userdata preservation: If a stopped group occupying the same custom-HOME slot has a host path, guest product path, or instance topology mismatch, remove and replace that slot group cleanly while preserving default-HOME stopped groups (returning a mismatch error rather than deleting user userdata.img).
  3. Symlink repair & lock handoff: Hold the group's in_progress.lock across reuse validation, symlink repair (EnsureSymlink when $HOME/* was wiped), EnsureInstanceLocksInUse, and StartGroup handoff, and evaluate IsDefaultGroup from request.Env() rather than process getenv.
  4. Hermetic regression suite (create_test): Add create_test (cli/BUILD.bazel) covering multi-slot isolation, stale RUNNING recovery, live launcher_monitor.sock and in_progress.lock active-group protection, custom-HOME slot replacement, default-HOME userdata preservation, deleted-directory pruning, and multi-process concurrency.

Dependency updates in //cuttlefish/host/commands/cvd/cli:nesting_commands (cli/BUILD.bazel):

  • Added direct dependency:
    • //cuttlefish/posix:readlink: create.cpp:64 includes "cuttlefish/posix/readlink.h" (for ReadLink()).
  • Removed unused direct dependencies (unreferenced across request_context.{h,cpp}, create.{h,cpp}, help.{h,cpp}, and load_configs.{h,cpp}):
    • //cuttlefish/common/libs/utils:environment: IsDefaultGroup in create.cpp now inspects request.Env() instead of StringFromEnv from environment.h.
    • //cuttlefish/host/commands/cvd/cli:utils: create.cpp previously included cli/utils.h without referencing any symbol from it.
    • //cuttlefish/host/commands/cvd/cli/selector: create.cpp matches reusable groups via InstanceManager::FindGroups directly instead of selector::SelectGroup from selector.h.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...

`DataViewer::WithExclusiveLock` passed `LOCK_SH` instead of `LOCK_EX` to
`LockBackingFile`, allowing concurrent `cvd` writers to hold shared locks
simultaneously while truncating (`ftruncate(0)`) and rewriting
`instance_database.binpb`, which could interleave writes and corrupt the
serialized `cvd::PersistentData` protobuf.

Acquire `LOCK_EX` in `WithExclusiveLock`, pass `O_CLOEXEC` when opening
the backing file so child daemons do not inherit an open database file
descriptor across `exec`, centralize the signal-masked overwrite helper
in `DataViewer::StoreData`, and add `WithConditionalExclusiveLock` so
read-modify-write callers that leave `data` unchanged can skip
truncating and rewriting the backing file.

Add `data_viewer_test` (`instances/BUILD.bazel`) to verify `LOCK_EX`
mutual exclusion, concurrent writer serialization, and no-op conditional
locks.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/...
Fix two `SIGABRT` crash paths triggered during instance probing and
slot-directory reuse:

1. `LocalInstance::GetLauncherMonitor` called `SharedFD::SocketLocalClient`
   unconditionally on `uds_dir + "/launcher_monitor.sock"`. When
   `.cuttlefish_config.json` was missing or unreadable and the fallback
   `InstanceDirectory()` path exceeded 107 bytes, `SocketLocalClient`
   aborted at `CHECK_LE(namelen, 108)`. Tolerate missing JSON configs,
   verify `FileExists(monitor_path)` and
   `monitor_path.size() < sizeof(sockaddr_un::sun_path)` before
   connecting, and expose `LocalInstance::CanConnectToLauncherMonitor`
   for liveness checks.

2. `GatherVmStartMetrics` in `device_metrics_orchestration.cc` invoked
   `RunMetrics` -> `CreateLogger(metrics_directory)` without ensuring
   `<HomeDir>/metrics` existed first, and `RunMetrics` opened
   `metrics_v2.log` before checking `FileExists(metrics_directory)`.
   When a caller wiped `$HOME/*` between runs and restarted the group,
   `SeverityTarget::FromFile` hit `LOG(FATAL)` (`SIGABRT`). Run
   `SetUpMetrics(instance_group.MetricsDir())` unconditionally in both
   `GatherVmInstantiationMetrics` and `GatherVmStartMetrics`, verify
   `FileExists` before `CreateLogger` in `RunMetrics`, and skip
   `GetDeviceMetricsInput` / `GatherDeviceMetrics` when
   `!AreMetricsEnabled()` so `<HomeDir>/metrics` is always initialized
   while avoiding unnecessary host/GCE probes when metrics transmission
   is disabled.

Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/metrics:device_metrics_orchestration` (`metrics/BUILD.bazel`):
  - `//cuttlefish/metrics/io:enabled`: `device_metrics_orchestration.cc:34`
    directly `#include`s `"cuttlefish/metrics/io/enabled.h"` (for
    `AreMetricsEnabled()`); previously pulled in transitively via
    `//cuttlefish/metrics:metrics_orchestration`.
- `//cuttlefish/host/commands/cvd/instances:local_instance_test` (`instances/BUILD.bazel`):
  - `:cvd_persistent_data`, `:data_viewer`, `//cuttlefish/common/libs/fs:fd`,
    `//cuttlefish/common/libs/utils:files`,
    `//cuttlefish/files:directory_exists`,
    `//cuttlefish/files:recursively_remove_directory`,
    `//cuttlefish/files:write_new_file`,
    `//cuttlefish/metrics:device_metrics_orchestration`,
    `//cuttlefish/result`, `//cuttlefish/result:result_matchers`, and
    `@abseil-cpp//absl/strings` for the new monitor-socket and metrics-dir
    regression tests in `local_instance_test.cpp:24-39`.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/... //cuttlefish/metrics/...
Follow up on 2545596 (`cvd: Use 0777 permissions on shared instance
locks directory`) to fix three multi-user and concurrency hazards on
shared `/var/tmp/cvd/lock/local-instance-<N>.lock` files:

1. `LockFileManager::OpenLockFile` called `open(O_CREAT | O_RDWR, 0666)`
   even when the lock file already existed. On kernels with
   `fs.protected_regular = 2` (default on Debian 13), opening an
   existing regular file owned by another UID inside a world-writable
   (`0777`) directory with `O_CREAT` without `O_EXCL` fails with
   `EACCES`. Open existing lock files with `O_RDWR | O_CLOEXEC` first,
   and only fall back to `O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC` (using
   `fchmod(0666)` on the newly created fd and skipping `chmod(parent,
   0777)` when `parent_dir` is owned by another UID).

2. `AcquireLock` and `TryAcquireLock` did not re-check that the locked
   fd still matched the path's inode (`fstat` vs `stat`), allowing a
   concurrent `RemoveLockFile` / `DeleteLockFile` to unlink a locked
   inode and let a second caller create and lock a distinct inode for
   the same instance ID simultaneously. Retry if `IsSameInode` fails
   after `flock(LOCK_EX)`.

3. Add `AcquireCurrentUserLock`, `AcquireOwnedLockIfPresent`,
   `ReleaseHeldLock`, and `ReleaseInstanceLock` on
   `InstanceLockFileManager`, and update `AcquireUnusedLock` to reject
   active (`kInUse`) locks held by other UIDs while replacing stale
   (`kNotInUse`) foreign-UID lock files with fresh inodes owned by
   `geteuid()` so subsequent `fchmod` status updates cannot fail with
   `EPERM`.

Add `instance_lock_test` (`instances/lock/BUILD.bazel`) covering
`O_CLOEXEC`, concurrent `AcquireLock` + `RemoveLockFile` inode races,
and `AcquireCurrentUserLock` / `ReleaseInstanceLock` semantics.

Bug: b/554100008
Bug: b/584825999
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/lock/...
`stop_cvd` terminates `run_cvd` directly over `launcher_monitor.sock`
without updating `instance_database.binpb`, leaving stopped groups stuck
in `INSTANCE_STATE_RUNNING` and leaving deleted slot directories behind
as dangling symlinks holding instance ID locks.

Reconcile database state against live host state and harden group
lifecycle transitions:

1. Protect in-flight `PREPARING` (`cvd load`) and `STARTING` (`cvd start`)
   group transitions with an exclusive per-group `<BaseDir>/in_progress.lock`
   (`AcquireGroupInProgressLock` / `IsGroupTransitionInProgress`),
   transferring the open lock fd into `CvdStartCommandHandler` so
   concurrent read/prune probes (`LOCK_SH | LOCK_NB`) never clobber a
   preparing or booting group.

2. Add `RefreshInstanceGroupState` and batched `RefreshInstanceGroups`
   (`ProbeGroupStateUpdates` + `BatchCompareAndUpdateInstanceStates`),
   checking `CanConnectToLauncherMonitor` and active `run_cvd` processes
   (caching `/proc` scans across groups and invalidating on slow socket
   probes) to reconcile stale active/inactive states under conditional
   database CAS.

3. Add `PruneBrokenInstanceGroups` to remove inactive groups whose
   backing `HomeDir`, `HostArtifactsPath` start binary, or
   `ProductOutPath` directories were deleted on disk, releasing their
   instance ID locks.

4. Acquire requested instance locks in ascending ID order in
   `AllocateAndLockInstanceIds`, atomically claim unique numeric
   `<base_root_dir>/<timestamp>` group directories (`mkdir`/`symlink`
   retrying on `EEXIST` while keeping pure-digit basenames compatible
   with `podcvd` path translation), roll back newly allocated locks on
   group-creation failure, reject duplicate canonical `RealPath(HOME)`
   groups in `InstanceDatabase::AddInstanceGroup`, and refresh group
   state in `cvd start`, `cvd stop`, `cvd remove`, and `cvd reset`.

Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/host/commands/cvd/instances:instances` (`instances/BUILD.bazel`):
  - `//cuttlefish/posix:realpath`: `instance_database.cpp:37` includes
    `"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
    in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/instances:instance_manager` (`instances/BUILD.bazel`):
  - `//cuttlefish/common/libs/fs:fd`: `instance_manager.cpp:45` includes
    `"cuttlefish/common/libs/fs/fd.h"` (for `Fd::Open()`); previously
    pulled in transitively via `:instances`, `:data_viewer`, and
    `//cuttlefish/host/commands/cvd/instances/lock`.
  - `//cuttlefish/files:file_exists`: `instance_manager.cpp:49` includes
    `"cuttlefish/files/file_exists.h"` (for `FileExists()`); previously
    pulled in transitively via `:instances`, `:config_path`,
    `:run_cvd_proc_collector`, and `:stop`.
  - `//cuttlefish/posix:realpath`: `instance_manager.cpp:62` includes
    `"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
    in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/cli:nesting_commands` (`cli/BUILD.bazel`):
  - `//cuttlefish/common/libs/fs`: `load_configs.h:26` and
    `load_configs.cpp:29` include
    `"cuttlefish/common/libs/fs/shared_fd.h"` (for `SharedFD`);
    previously pulled in transitively via
    `//cuttlefish/host/commands/cvd/instances` and
    `//cuttlefish/host/commands/cvd/cli/commands:start`.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...
When multiple Cuttlefish instances run concurrently under the same host
UID (such as multi-slot runners passing distinct `HOME` directories,
`--base_instance_num`, or `--instance_name`), `cvd create --reuse=true`
queried `instance_manager_.FindGroups({})` across all groups owned by
the UID and either failed with `"Trying to change the host paths on an
existing group"` (when 1 other slot existed) or `"Unclear which group to
reuse"` (when 2+ slots existed).

Scope `--reuse=true` candidate lookup to the target slot:

1. Prune deleted/broken inactive groups before group creation or reuse,
   and isolate `FindReusableGroup` by explicit `--group_name` selector
   first, custom `HOME` directory second (`GroupCustomHome`, tolerating
   wiped/dangling custom-`HOME` symlinks via `ReadLink`), and requested
   instance IDs or per-instance names third, falling back to default-`HOME`
   groups only when running in the system-wide default `HOME`.

2. If a stopped group occupying the same custom-`HOME` slot has a host
   path, guest product path, or instance topology mismatch, remove and
   replace that slot group cleanly while preserving default-`HOME`
   stopped groups (returning a mismatch error rather than deleting user
   `userdata.img`).

3. Hold the group's `in_progress.lock` across reuse validation, symlink
   repair (`EnsureSymlink` when `$HOME/*` was wiped),
   `EnsureInstanceLocksInUse`, and `StartGroup` handoff, and evaluate
   `IsDefaultGroup` from `request.Env()` rather than process `getenv`.

4. Add hermetic `create_test` (`cli/BUILD.bazel`) covering multi-slot
   isolation, stale `RUNNING` recovery, live `launcher_monitor.sock` and
   `in_progress.lock` active-group protection, custom-`HOME` slot
   replacement, default-`HOME` userdata preservation, deleted-directory
   pruning, and multi-process concurrency.

Dependency updates in `//cuttlefish/host/commands/cvd/cli:nesting_commands`
(`cli/BUILD.bazel`):
- Added direct dependency:
  - `//cuttlefish/posix:readlink`: `create.cpp:64` includes
    `"cuttlefish/posix/readlink.h"` (for `ReadLink()`).
- Removed unused direct dependencies (unreferenced across
  `request_context.{h,cpp}`, `create.{h,cpp}`, `help.{h,cpp}`, and
  `load_configs.{h,cpp}`):
  - `//cuttlefish/common/libs/utils:environment`: `IsDefaultGroup` in
    `create.cpp` now inspects `request.Env()` instead of `StringFromEnv`
    from `environment.h`.
  - `//cuttlefish/host/commands/cvd/cli:utils`: `create.cpp` previously
    included `cli/utils.h` without referencing any symbol from it.
  - `//cuttlefish/host/commands/cvd/cli/selector`: `create.cpp` matches
    reusable groups via `InstanceManager::FindGroups` directly instead of
    `selector::SelectGroup` from `selector.h`.

Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...
@rmuthiah
rmuthiah force-pushed the fix-cvd-reuse-groups branch from e353e65 to 1a36d76 Compare November 17, 2026 18:26
@rmuthiah rmuthiah changed the title cvd: Fix multi-slot group reuse, state reconciliation, and lock concurrency cvd: Scope create --reuse by slot and preserve default-HOME userdata Nov 17, 2026
@rmuthiah
rmuthiah requested a review from Databean November 18, 2026 15:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant