Skip to content

cvd: Harden shared instance locks for multi-user hosts and concurrency - #3608

Open
rmuthiah wants to merge 1 commit into
google:mainfrom
rmuthiah:fix-cvd-shared-instance-locks
Open

rmuthiah wants to merge 1 commit into
google:mainfrom
rmuthiah:fix-cvd-shared-instance-locks

Conversation

@rmuthiah

Copy link
Copy Markdown
Collaborator

Follow up on 2545596 (cvd: Use 0777 permissions on shared instance locks directory) to fix three multi-user and concurrency hazards on shared /var/tmp/cvd/lock/local-instance-<N>.lock files:

  1. Linux fs.protected_regular = 2 EACCES: LockFileManager::OpenLockFile called open(O_CREAT | O_RDWR, 0666) even when the lock file already existed. On kernels with fs.protected_regular = 2 (default on Debian 13), opening an existing regular file owned by another UID inside a world-writable (0777) directory with O_CREAT without O_EXCL fails with EACCES. Open existing lock files with O_RDWR | O_CLOEXEC first, and only fall back to O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC (using fchmod(0666) on the newly created fd and skipping chmod(parent, 0777) when parent_dir is owned by another UID).
  2. flock + unlink split-brain race: AcquireLock and TryAcquireLock did not re-check that the locked fd still matched the path's inode (fstat vs stat), allowing a concurrent RemoveLockFile / DeleteLockFile to unlink a locked inode and let a second caller create and lock a distinct inode for the same instance ID simultaneously. Retry if IsSameInode fails after flock(LOCK_EX).
  3. Cross-user lock ownership safety: Add AcquireCurrentUserLock, AcquireOwnedLockIfPresent, ReleaseHeldLock, and ReleaseInstanceLock on InstanceLockFileManager, and update AcquireUnusedLock to reject active (kInUse) locks held by other UIDs while replacing stale (kNotInUse) foreign-UID lock files with fresh inodes owned by geteuid() so subsequent fchmod status updates cannot fail with EPERM.

Add instance_lock_test (instances/lock/BUILD.bazel) covering O_CLOEXEC, concurrent AcquireLock + ReleaseHeldLock inode races, and AcquireCurrentUserLock / ReleaseInstanceLock semantics.

Bug: b/554100008
Bug: b/584825999
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/lock/...

Follow up on 2545596 (`cvd: Use 0777 permissions on shared instance
locks directory`) to fix three multi-user and concurrency hazards on
shared `/var/tmp/cvd/lock/local-instance-<N>.lock` files:

1. `LockFileManager::OpenLockFile` called `open(O_CREAT | O_RDWR, 0666)`
   even when the lock file already existed. On kernels with
   `fs.protected_regular = 2` (default on Debian 13), opening an
   existing regular file owned by another UID inside a world-writable
   (`0777`) directory with `O_CREAT` without `O_EXCL` fails with
   `EACCES`. Open existing lock files with `O_RDWR | O_CLOEXEC` first,
   and only fall back to `O_CREAT | O_EXCL | O_RDWR | O_CLOEXEC` (using
   `fchmod(0666)` on the newly created fd and skipping `chmod(parent,
   0777)` when `parent_dir` is owned by another UID).

2. `AcquireLock` and `TryAcquireLock` did not re-check that the locked
   fd still matched the path's inode (`fstat` vs `stat`), allowing a
   concurrent `RemoveLockFile` / `DeleteLockFile` to unlink a locked
   inode and let a second caller create and lock a distinct inode for
   the same instance ID simultaneously. Retry if `IsSameInode` fails
   after `flock(LOCK_EX)`.

3. Add `AcquireCurrentUserLock`, `AcquireOwnedLockIfPresent`,
   `ReleaseHeldLock`, and `ReleaseInstanceLock` on
   `InstanceLockFileManager`, and update `AcquireUnusedLock` to reject
   active (`kInUse`) locks held by other UIDs while replacing stale
   (`kNotInUse`) foreign-UID lock files with fresh inodes owned by
   `geteuid()` so subsequent `fchmod` status updates cannot fail with
   `EPERM`.

Add `instance_lock_test` (`instances/lock/BUILD.bazel`) covering
`O_CLOEXEC`, concurrent `AcquireLock` + `RemoveLockFile` inode races,
and `AcquireCurrentUserLock` / `ReleaseInstanceLock` semantics.

Bug: b/554100008
Bug: b/584825999
Bug: b/586523782
Test: bazel test //cuttlefish/host/commands/cvd/instances/lock/...

@Databean Databean left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand the description correctly, the problem is that the atomic create-or-open behavior with open(..., O_CREAT) doesn't work anymore on a directory owned by another user, which happens now with the locks directory. Most of the PR tries to work around this deficiency in C++ logic, now having to deal with races between flock and unlink.

Another new behavior introduced by #3593 is to set up /var/tmp/cvd/lock in the debian postinst file. Rather than introduce all this multi-process synchronization, would it work to have the systemd service defined by the debian package create all the lock files in advance on startup? Then the C++ code doesn't have to worry about files existing / not existing and can open without O_CREAT.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants