Repository navigation
Conversation
`DataViewer::WithExclusiveLock` passed `LOCK_SH` instead of `LOCK_EX` to `LockBackingFile`, allowing concurrent `cvd` writers to hold shared locks simultaneously while truncating (`ftruncate(0)`) and rewriting `instance_database.binpb`, which could interleave writes and corrupt the serialized `cvd::PersistentData` protobuf. Acquire `LOCK_EX` in `WithExclusiveLock`, pass `O_CLOEXEC` when opening the backing file so child daemons do not inherit an open database file descriptor across `exec`, centralize the signal-masked overwrite helper in `DataViewer::StoreData`, and add `WithConditionalExclusiveLock` so read-modify-write callers that leave `data` unchanged can skip truncating and rewriting the backing file. Add `data_viewer_test` (`instances/BUILD.bazel`) to verify `LOCK_EX` mutual exclusion, concurrent writer serialization, and no-op conditional locks. Bug: b/554100008 Bug: b/586523782 Test: bazel test //cuttlefish/host/commands/cvd/instances/...
Fix two `SIGABRT` crash paths triggered during instance probing and
slot-directory reuse:
1. `LocalInstance::GetLauncherMonitor` called `SharedFD::SocketLocalClient`
unconditionally on `uds_dir + "/launcher_monitor.sock"`. When
`.cuttlefish_config.json` was missing or unreadable and the fallback
`InstanceDirectory()` path exceeded 107 bytes, `SocketLocalClient`
aborted at `CHECK_LE(namelen, 108)`. Tolerate missing JSON configs,
verify `FileExists(monitor_path)` and
`monitor_path.size() < sizeof(sockaddr_un::sun_path)` before
connecting, and expose `LocalInstance::CanConnectToLauncherMonitor`
for liveness checks.
2. `GatherVmStartMetrics` in `device_metrics_orchestration.cc` invoked
`RunMetrics` -> `CreateLogger(metrics_directory)` without ensuring
`<HomeDir>/metrics` existed first, and `RunMetrics` opened
`metrics_v2.log` before checking `FileExists(metrics_directory)`.
When a caller wiped `$HOME/*` between runs and restarted the group,
`SeverityTarget::FromFile` hit `LOG(FATAL)` (`SIGABRT`). Run
`SetUpMetrics(instance_group.MetricsDir())` unconditionally in both
`GatherVmInstantiationMetrics` and `GatherVmStartMetrics`, verify
`FileExists` before `CreateLogger` in `RunMetrics`, and skip
`GetDeviceMetricsInput` / `GatherDeviceMetrics` when
`!AreMetricsEnabled()` so `<HomeDir>/metrics` is always initialized
while avoiding unnecessary host/GCE probes when metrics transmission
is disabled.
Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/metrics:device_metrics_orchestration` (`metrics/BUILD.bazel`):
- `//cuttlefish/metrics/io:enabled`: `device_metrics_orchestration.cc:34`
directly `#include`s `"cuttlefish/metrics/io/enabled.h"` (for
`AreMetricsEnabled()`); previously pulled in transitively via
`//cuttlefish/metrics:metrics_orchestration`.
- `//cuttlefish/host/commands/cvd/instances:local_instance_test` (`instances/BUILD.bazel`):
- `:cvd_persistent_data`, `:data_viewer`, `//cuttlefish/common/libs/fs:fd`,
`//cuttlefish/common/libs/utils:files`,
`//cuttlefish/files:directory_exists`,
`//cuttlefish/files:recursively_remove_directory`,
`//cuttlefish/files:write_new_file`,
`//cuttlefish/metrics:device_metrics_orchestration`,
`//cuttlefish/result`, `//cuttlefish/result:result_matchers`, and
`@abseil-cpp//absl/strings` for the new monitor-socket and metrics-dir
regression tests in `local_instance_test.cpp:24-39`.
Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/... //cuttlefish/metrics/...
Follow up on 2545596 (`cvd: Use 0777 permissions on shared instance locks directory`) to fix three multi-user and concurrency hazards on shared `/var/tmp/cvd/lock/local-instance-<N>.lock` files: 1. `LockFileManager::OpenLockFile` called `open(O_CREAT | O_RDWR, 0666)` even when the lock file already existed. On kernels with `fs.protected_regular = 2` (default on Debian 13), opening an existing regular file owned by another UID inside a world-writable (`0777`) directory with `O_CREAT` without `O_EXCL` fails with `EACCES`. Open existing lock files with `O_RDWR | O_CLOEXEC` first, and only fall back to `O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC` (using `fchmod(0666)` on the newly created fd and skipping `chmod(parent, 0777)` when `parent_dir` is owned by another UID). 2. `AcquireLock` and `TryAcquireLock` did not re-check that the locked fd still matched the path's inode (`fstat` vs `stat`), allowing a concurrent `RemoveLockFile` / `DeleteLockFile` to unlink a locked inode and let a second caller create and lock a distinct inode for the same instance ID simultaneously. Retry if `IsSameInode` fails after `flock(LOCK_EX)`. 3. Add `AcquireCurrentUserLock`, `AcquireOwnedLockIfPresent`, `ReleaseHeldLock`, and `ReleaseInstanceLock` on `InstanceLockFileManager`, and update `AcquireUnusedLock` to reject active (`kInUse`) locks held by other UIDs while replacing stale (`kNotInUse`) foreign-UID lock files with fresh inodes owned by `geteuid()` so subsequent `fchmod` status updates cannot fail with `EPERM`. Add `instance_lock_test` (`instances/lock/BUILD.bazel`) covering `O_CLOEXEC`, concurrent `AcquireLock` + `RemoveLockFile` inode races, and `AcquireCurrentUserLock` / `ReleaseInstanceLock` semantics. Bug: b/554100008 Bug: b/584825999 Bug: b/586523782 Test: bazel test //cuttlefish/host/commands/cvd/instances/lock/...
`stop_cvd` terminates `run_cvd` directly over `launcher_monitor.sock`
without updating `instance_database.binpb`, leaving stopped groups stuck
in `INSTANCE_STATE_RUNNING` and leaving deleted slot directories behind
as dangling symlinks holding instance ID locks.
Reconcile database state against live host state and harden group
lifecycle transitions:
1. Protect in-flight `PREPARING` (`cvd load`) and `STARTING` (`cvd start`)
group transitions with an exclusive per-group `<BaseDir>/in_progress.lock`
(`AcquireGroupInProgressLock` / `IsGroupTransitionInProgress`),
transferring the open lock fd into `CvdStartCommandHandler` so
concurrent read/prune probes (`LOCK_SH | LOCK_NB`) never clobber a
preparing or booting group.
2. Add `RefreshInstanceGroupState` and batched `RefreshInstanceGroups`
(`ProbeGroupStateUpdates` + `BatchCompareAndUpdateInstanceStates`),
checking `CanConnectToLauncherMonitor` and active `run_cvd` processes
(caching `/proc` scans across groups and invalidating on slow socket
probes) to reconcile stale active/inactive states under conditional
database CAS.
3. Add `PruneBrokenInstanceGroups` to remove inactive groups whose
backing `HomeDir`, `HostArtifactsPath` start binary, or
`ProductOutPath` directories were deleted on disk, releasing their
instance ID locks.
4. Acquire requested instance locks in ascending ID order in
`AllocateAndLockInstanceIds`, atomically claim unique numeric
`<base_root_dir>/<timestamp>` group directories (`mkdir`/`symlink`
retrying on `EEXIST` while keeping pure-digit basenames compatible
with `podcvd` path translation), roll back newly allocated locks on
group-creation failure, reject duplicate canonical `RealPath(HOME)`
groups in `InstanceDatabase::AddInstanceGroup`, and refresh group
state in `cvd start`, `cvd stop`, `cvd remove`, and `cvd reset`.
Direct dependencies added to `BUILD.bazel` targets:
- `//cuttlefish/host/commands/cvd/instances:instances` (`instances/BUILD.bazel`):
- `//cuttlefish/posix:realpath`: `instance_database.cpp:37` includes
`"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/instances:instance_manager` (`instances/BUILD.bazel`):
- `//cuttlefish/common/libs/fs:fd`: `instance_manager.cpp:45` includes
`"cuttlefish/common/libs/fs/fd.h"` (for `Fd::Open()`); previously
pulled in transitively via `:instances`, `:data_viewer`, and
`//cuttlefish/host/commands/cvd/instances/lock`.
- `//cuttlefish/files:file_exists`: `instance_manager.cpp:49` includes
`"cuttlefish/files/file_exists.h"` (for `FileExists()`); previously
pulled in transitively via `:instances`, `:config_path`,
`:run_cvd_proc_collector`, and `:stop`.
- `//cuttlefish/posix:realpath`: `instance_manager.cpp:62` includes
`"cuttlefish/posix/realpath.h"` (for `RealPath()`); previously pulled
in transitively via `:config_path`.
- `//cuttlefish/host/commands/cvd/cli:nesting_commands` (`cli/BUILD.bazel`):
- `//cuttlefish/common/libs/fs`: `load_configs.h:26` and
`load_configs.cpp:29` include
`"cuttlefish/common/libs/fs/shared_fd.h"` (for `SharedFD`);
previously pulled in transitively via
`//cuttlefish/host/commands/cvd/instances` and
`//cuttlefish/host/commands/cvd/cli/commands:start`.
Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3606, #3607, and #3608 (part 4 of 5; review just this patch's commit).
stop_cvdterminatesrun_cvddirectly overlauncher_monitor.sockwithout updatinginstance_database.binpb, leaving stopped groups stuck inINSTANCE_STATE_RUNNINGand leaving deleted slot directories behind as dangling symlinks holding instance ID locks.Reconcile database state against live host state and harden group lifecycle transitions:
<BaseDir>/in_progress.lock): ProtectPREPARING(cvd load) andSTARTING(cvd start) group transitions withAcquireGroupInProgressLock(LOCK_EX | LOCK_NB) andIsGroupTransitionInProgress(LOCK_SH | LOCK_NB), transferring the open lock fd intoCvdStartCommandHandlerso concurrent read/prune probes never clobber a preparing or booting group.RefreshInstanceGroupStateand batchedRefreshInstanceGroups(ProbeGroupStateUpdates+BatchCompareAndUpdateInstanceStates), checkingCanConnectToLauncherMonitorand activerun_cvdprocesses (caching/procscans across groups and invalidating on slow socket probes) to reconcile stale active/inactive states under conditional database CAS.PruneBrokenInstanceGroupsto remove inactive groups whose backingHomeDir,HostArtifactsPathstart binary, orProductOutPathdirectories were deleted on disk, releasing their instance ID locks.BaseDirallocation: Acquire requested instance locks in ascending ID order inAllocateAndLockInstanceIds, atomically claim unique numeric<base_root_dir>/<timestamp>group directories (mkdir/symlinkretrying onEEXISTwhile keeping pure-digit basenames compatible withpodcvdpath translation), roll back newly allocated locks on group-creation failure, reject duplicate canonicalRealPath(HOME)groups inInstanceDatabase::AddInstanceGroup, and refresh group state incvd start,cvd stop,cvd remove, andcvd reset.Direct dependencies added to
BUILD.bazeltargets://cuttlefish/host/commands/cvd/instances:instances(instances/BUILD.bazel)://cuttlefish/posix:realpath:instance_database.cpp:37includes"cuttlefish/posix/realpath.h"(forRealPath()); previously pulled in transitively via:config_path.//cuttlefish/host/commands/cvd/instances:instance_manager(instances/BUILD.bazel)://cuttlefish/common/libs/fs:fd:instance_manager.cpp:45includes"cuttlefish/common/libs/fs/fd.h"(forFd::Open()); previously pulled in transitively via:instances,:data_viewer, and//cuttlefish/host/commands/cvd/instances/lock.//cuttlefish/files:file_exists:instance_manager.cpp:49includes"cuttlefish/files/file_exists.h"(forFileExists()); previously pulled in transitively via:instances,:config_path,:run_cvd_proc_collector, and:stop.//cuttlefish/posix:realpath:instance_manager.cpp:62includes"cuttlefish/posix/realpath.h"(forRealPath()); previously pulled in transitively via:config_path.//cuttlefish/host/commands/cvd/cli:nesting_commands(cli/BUILD.bazel)://cuttlefish/common/libs/fs:load_configs.h:26andload_configs.cpp:29include"cuttlefish/common/libs/fs/shared_fd.h"(forSharedFD); previously pulled in transitively via//cuttlefish/host/commands/cvd/instancesand//cuttlefish/host/commands/cvd/cli/commands:start.Bug: b/554100008
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/...