Repository navigation
Conversation
`DataViewer::WithExclusiveLock` passed `LOCK_SH` instead of `LOCK_EX` to `LockBackingFile`, allowing concurrent `cvd` writers to hold shared locks simultaneously while truncating (`ftruncate(0)`) and rewriting `instance_database.binpb`, which could interleave writes and corrupt the serialized `cvd::PersistentData` protobuf. Acquire `LOCK_EX` in `WithExclusiveLock`, pass `O_CLOEXEC` when opening the backing file so child daemons do not inherit an open database file descriptor across `exec`, centralize the signal-masked overwrite helper in `DataViewer::StoreData`, and add `WithConditionalExclusiveLock` so read-modify-write callers that leave `data` unchanged can skip truncating and rewriting the backing file. Add `data_viewer_test` (`instances/BUILD.bazel`) to verify `LOCK_EX` mutual exclusion, concurrent writer serialization, and no-op conditional locks. Bug: b/554100008 Bug: b/586523782 Test: bazel test //cuttlefish/host/commands/cvd/instances/...
Fix two `SIGABRT` crash paths triggered during instance probing and
slot-directory reuse:
1. `LocalInstance::GetLauncherMonitor` called `SharedFD::SocketLocalClient`
unconditionally on `uds_dir + "/launcher_monitor.sock"`. When
`.cuttlefish_config.json` was missing or unreadable and the fallback
`InstanceDirectory()` path exceeded 107 bytes, `SocketLocalClient`
aborted at `CHECK_LE(namelen, 108)`. Tolerate missing JSON configs,
verify `FileExists(monitor_path)` and
`monitor_path.size() < sizeof(sockaddr_un::sun_path)` before
connecting, and expose `LocalInstance::CanConnectToLauncherMonitor`
for liveness checks.
2. `GatherVmStartMetrics` in `device_metrics_orchestration.cc` invoked
`RunMetrics` -> `CreateLogger(metrics_directory)` without ensuring
`<HomeDir>/metrics` existed first, and `RunMetrics` opened
`metrics_v2.log` before checking `FileExists(metrics_directory)`.
When a caller wiped `$HOME/*` between runs and restarted the group,
`SeverityTarget::FromFile` hit `LOG(FATAL)` (`SIGABRT`). Run
`SetUpMetrics(instance_group.MetricsDir())` unconditionally in both
`GatherVmInstantiationMetrics` and `GatherVmStartMetrics`, verify
`FileExists` before `CreateLogger` in `RunMetrics`, and skip
`GetDeviceMetricsInput` / `GatherDeviceMetrics` when
`!AreMetricsEnabled()` so `<HomeDir>/metrics` is always initialized
while avoiding unnecessary host/GCE probes when metrics transmission
is disabled.
Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/metrics:device_metrics_orchestration` (`metrics/BUILD.bazel`):
- `//cuttlefish/metrics/io:enabled`: `device_metrics_orchestration.cc:34`
directly `#include`s `"cuttlefish/metrics/io/enabled.h"` (for
`AreMetricsEnabled()`); previously pulled in transitively via
`//cuttlefish/metrics:metrics_orchestration`.
- `//cuttlefish/host/commands/cvd/instances:local_instance_test` (`instances/BUILD.bazel`):
- `:cvd_persistent_data`, `:data_viewer`, `//cuttlefish/common/libs/fs:fd`,
`//cuttlefish/common/libs/utils:files`,
`//cuttlefish/files:directory_exists`,
`//cuttlefish/files:recursively_remove_directory`,
`//cuttlefish/files:write_new_file`,
`//cuttlefish/metrics:device_metrics_orchestration`,
`//cuttlefish/result`, `//cuttlefish/result:result_matchers`, and
`@abseil-cpp//absl/strings` for the new monitor-socket and metrics-dir
regression tests in `local_instance_test.cpp:24-39`.
Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/... //cuttlefish/metrics/...
Follow up on 2545596 (`cvd: Use 0777 permissions on shared instance locks directory`) to fix three multi-user and concurrency hazards on shared `/var/tmp/cvd/lock/local-instance-<N>.lock` files: 1. `LockFileManager::OpenLockFile` called `open(O_CREAT | O_RDWR, 0666)` even when the lock file already existed. On kernels with `fs.protected_regular = 2` (default on Debian 13), opening an existing regular file owned by another UID inside a world-writable (`0777`) directory with `O_CREAT` without `O_EXCL` fails with `EACCES`. Open existing lock files with `O_RDWR | O_CLOEXEC` first, and only fall back to `O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC` (using `fchmod(0666)` on the newly created fd and skipping `chmod(parent, 0777)` when `parent_dir` is owned by another UID). 2. `AcquireLock` and `TryAcquireLock` did not re-check that the locked fd still matched the path's inode (`fstat` vs `stat`), allowing a concurrent `RemoveLockFile` / `DeleteLockFile` to unlink a locked inode and let a second caller create and lock a distinct inode for the same instance ID simultaneously. Retry if `IsSameInode` fails after `flock(LOCK_EX)`. 3. Add `AcquireCurrentUserLock`, `AcquireOwnedLockIfPresent`, `ReleaseHeldLock`, and `ReleaseInstanceLock` on `InstanceLockFileManager`, and update `AcquireUnusedLock` to reject active (`kInUse`) locks held by other UIDs while replacing stale (`kNotInUse`) foreign-UID lock files with fresh inodes owned by `geteuid()` so subsequent `fchmod` status updates cannot fail with `EPERM`. Add `instance_lock_test` (`instances/lock/BUILD.bazel`) covering `O_CLOEXEC`, concurrent `AcquireLock` + `RemoveLockFile` inode races, and `AcquireCurrentUserLock` / `ReleaseInstanceLock` semantics. Bug: b/554100008 Bug: b/584825999 Bug: b/586523782 Test: bazel test //cuttlefish/host/commands/cvd/instances/lock/...
`stop_cvd` terminates `run_cvd` directly over `launcher_monitor.sock`
without updating `instance_database.binpb`, leaving stopped groups stuck
in `INSTANCE_STATE_RUNNING` and leaving deleted slot directories behind
as dangling symlinks holding instance ID locks.
Reconcile database state against live host state and harden group
lifecycle transitions:
1. Protect in-flight `PREPARING` (`cvd load`) and `STARTING` (`cvd start`)
group transitions with an exclusive per-group `<BaseDir>/in_progress.lock`
(`AcquireGroupInProgressLock` / `IsGroupTransitionInProgress`),
transferring the open lock fd into `CvdStartCommandHandler` so
concurrent read/prune probes (`LOCK_SH | LOCK_NB`) never clobber a
preparing or booting group.
2. Add `RefreshInstanceGroupState` and batched `RefreshInstanceGroups`
(`ProbeGroupStateUpdates` + `BatchCompareAndUpdateInstanceStates`),
checking `CanConnectToLauncherMonitor` and active `run_cvd` processes
(caching `/proc` scans across groups and invalidating on slow socket
probes) to reconcile stale active/inactive states under conditional
database CAS.
3. Add `PruneBrokenInstanceGroups` to remove inactive groups whose
backing `HomeDir`, `HostArtifactsPath` start binary, or
`ProductOutPath` directories were deleted on disk, releasing their
instance ID locks.
4. Acquire requested instance locks in ascending ID order in
`AllocateAndLockInstanceIds`, atomically claim unique numeric
`<base_root_dir>/<timestamp>` group directories (`mkdir`/`symlink`
retrying on `EEXIST` while keeping pure-digit basenames compatible
with `podcvd` path translation), roll back newly allocated locks on
group-creation failure, reject duplicate canonical `RealPath(HOME)`
groups in `InstanceDatabase::AddInstanceGroup`, and refresh group
state in `cvd start`, `cvd stop`, `cvd remove`, and `cvd reset`.
Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/host/commands/cvd/instances:instances` (`instances/BUILD.bazel`):
- `//cuttlefish/posix:realpath`: `instance_database.cpp:37` includes
`"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/instances:instance_manager` (`instances/BUILD.bazel`):
- `//cuttlefish/common/libs/fs:fd`: `instance_manager.cpp:45` includes
`"cuttlefish/common/libs/fs/fd.h"` (for `Fd::Open()`); previously
pulled in transitively via `:instances`, `:data_viewer`, and
`//cuttlefish/host/commands/cvd/instances/lock`.
- `//cuttlefish/files:file_exists`: `instance_manager.cpp:49` includes
`"cuttlefish/files/file_exists.h"` (for `FileExists()`); previously
pulled in transitively via `:instances`, `:config_path`,
`:run_cvd_proc_collector`, and `:stop`.
- `//cuttlefish/posix:realpath`: `instance_manager.cpp:62` includes
`"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/cli:nesting_commands` (`cli/BUILD.bazel`):
- `//cuttlefish/common/libs/fs`: `load_configs.h:26` and
`load_configs.cpp:29` include
`"cuttlefish/common/libs/fs/shared_fd.h"` (for `SharedFD`);
previously pulled in transitively via
`//cuttlefish/host/commands/cvd/instances` and
`//cuttlefish/host/commands/cvd/cli/commands:start`.
Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...
When multiple Cuttlefish instances run concurrently under the same host
UID (such as multi-slot runners passing distinct `HOME` directories,
`--base_instance_num`, or `--instance_name`), `cvd create --reuse=true`
queried `instance_manager_.FindGroups({})` across all groups owned by
the UID and either failed with `"Trying to change the host paths on an
existing group"` (when 1 other slot existed) or `"Unclear which group to
reuse"` (when 2+ slots existed).
Scope `--reuse=true` candidate lookup to the target slot:
1. Prune deleted/broken inactive groups before group creation or reuse,
and isolate `FindReusableGroup` by explicit `--group_name` selector
first, custom `HOME` directory second (`GroupCustomHome`, tolerating
wiped/dangling custom-`HOME` symlinks via `ReadLink`), and requested
instance IDs or per-instance names third, falling back to default-`HOME`
groups only when running in the system-wide default `HOME`.
2. If a stopped group occupying the same custom-`HOME` slot has a host
path, guest product path, or instance topology mismatch, remove and
replace that slot group cleanly while preserving default-`HOME`
stopped groups (returning a mismatch error rather than deleting user
`userdata.img`).
3. Hold the group's `in_progress.lock` across reuse validation, symlink
repair (`EnsureSymlink` when `$HOME/*` was wiped),
`EnsureInstanceLocksInUse`, and `StartGroup` handoff, and evaluate
`IsDefaultGroup` from `request.Env()` rather than process `getenv`.
4. Add hermetic `create_test` (`cli/BUILD.bazel`) covering multi-slot
isolation, stale `RUNNING` recovery, live `launcher_monitor.sock` and
`in_progress.lock` active-group protection, custom-`HOME` slot
replacement, default-`HOME` userdata preservation, deleted-directory
pruning, and multi-process concurrency.
Dependency updates in `//cuttlefish/host/commands/cvd/cli:nesting_commands`
(`cli/BUILD.bazel`):
- Added direct dependency:
- `//cuttlefish/posix:readlink`: `create.cpp:64` includes
`"cuttlefish/posix/readlink.h"` (for `ReadLink()`).
- Removed unused direct dependencies (unreferenced across
`request_context.{h,cpp}`, `create.{h,cpp}`, `help.{h,cpp}`, and
`load_configs.{h,cpp}`):
- `//cuttlefish/common/libs/utils:environment`: `IsDefaultGroup` in
`create.cpp` now inspects `request.Env()` instead of `StringFromEnv`
from `environment.h`.
- `//cuttlefish/host/commands/cvd/cli:utils`: `create.cpp` previously
included `cli/utils.h` without referencing any symbol from it.
- `//cuttlefish/host/commands/cvd/cli/selector`: `create.cpp` matches
reusable groups via `InstanceManager::FindGroups` directly instead of
`selector::SelectGroup` from `selector.h`.
Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...
rmuthiah
force-pushed
the
fix-cvd-reuse-groups
branch
from
November 17, 2026 18:26
e353e65 to
1a36d76
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3609 (part 5 of 5; review just this patch's commit).
When multiple Cuttlefish instances run concurrently under the same host UID (such as multi-slot runners passing distinct
HOMEdirectories,--base_instance_num, or--instance_name),cvd create --reuse=truequeriedinstance_manager_.FindGroups({})across all groups owned by the UID and either failed with"Trying to change the host paths on an existing group"(when 1 other slot existed) or"Unclear which group to reuse"(when 2+ slots existed).Scope
--reuse=truecandidate lookup to the target slot:FindReusableGroup: Prune deleted/broken inactive groups before group creation or reuse, and isolateFindReusableGroupby explicit--group_nameselector first, customHOMEdirectory second (GroupCustomHome, tolerating wiped/dangling custom-HOMEsymlinks viaReadLink), and requested instance IDs or per-instance names third, falling back to default-HOMEgroups only when running in the system-wide defaultHOME.HOMEreplacement vs default-HOMEuserdata preservation: If a stopped group occupying the same custom-HOMEslot has a host path, guest product path, or instance topology mismatch, remove and replace that slot group cleanly while preserving default-HOMEstopped groups (returning a mismatch error rather than deleting useruserdata.img).in_progress.lockacross reuse validation, symlink repair (EnsureSymlinkwhen$HOME/*was wiped),EnsureInstanceLocksInUse, andStartGrouphandoff, and evaluateIsDefaultGroupfromrequest.Env()rather than processgetenv.create_test): Addcreate_test(cli/BUILD.bazel) covering multi-slot isolation, staleRUNNINGrecovery, livelauncher_monitor.sockandin_progress.lockactive-group protection, custom-HOMEslot replacement, default-HOMEuserdata preservation, deleted-directory pruning, and multi-process concurrency.Dependency updates in
//cuttlefish/host/commands/cvd/cli:nesting_commands(cli/BUILD.bazel)://cuttlefish/posix:readlink:create.cpp:64includes"cuttlefish/posix/readlink.h"(forReadLink()).request_context.{h,cpp},create.{h,cpp},help.{h,cpp}, andload_configs.{h,cpp})://cuttlefish/common/libs/utils:environment:IsDefaultGroupincreate.cppnow inspectsrequest.Env()instead ofStringFromEnvfromenvironment.h.//cuttlefish/host/commands/cvd/cli:utils:create.cpppreviously includedcli/utils.hwithout referencing any symbol from it.//cuttlefish/host/commands/cvd/cli/selector:create.cppmatches reusable groups viaInstanceManager::FindGroupsdirectly instead ofselector::SelectGroupfromselector.h.Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...