Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/new-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,8 @@ Containerbase then removes the shell wrappers, the current-version marker and th
That default is only correct if `install()` writes **everything** below `createVersionedToolPath()`.
If your tool writes anywhere else - `$HOME`, a shared cache, or system packages - either override `uninstall()` to clean that up, or raise it on the Issue so we can agree what should happen.

If the tool keeps a download cache in the containerbase cache, eg. a folder its prepare service links from `$HOME`, declare it in `cacheDirs`, so `containerbase-cli cleanup cache` removes it, see the [tool installer best practices](./tool-installer-best-practices.md#folders-ownership-and-links).

Two more things worth knowing:

- If you attempt to uninstall a tool that other installed tools depend on (by declaring it as a `parent`), Containerbase will refuse to uninstall until `--recursive` is passed
Expand Down
10 changes: 10 additions & 0 deletions docs/tool-installer-best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,16 @@ override async prepare(): Promise<void> {
}
```

Declare the folders in the cache which only hold data the tool downloads again, like a package cache, in `cacheDirs` on the install service, relative to the cache path.
[`containerbase-cli cleanup cache`](./tools.md#cleanup-cache-command) removes them, so never list config files or folders holding them, and list a sub folder where needed:

```ts
override readonly cacheDirs = ['.cargo/git', '.cargo/registry'];
```

Caches in the containerbase user's `~/.cache`, the `.cache` folder in the cache path, need no entry, its contents are always removed.
Root's `/root/.cache` isn't touched.

See [`src/cli/services/path.service.ts`](../src/cli/services/path.service.ts) for the full set of path helpers before reaching for `node:fs` directly.

## Shared helpers
Expand Down
35 changes: 35 additions & 0 deletions docs/tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
- [List](#list)
- [list-tools](#list-tools-command)
- [Cleanup](#cleanup)
- [cleanup-cache](#cleanup-cache-command)
- [cleanup-downloads](#cleanup-downloads-command)

---
Expand Down Expand Up @@ -525,6 +526,40 @@ When writing to a file, JSON output is minified.

## Cleanup

### `containerbase-cli cleanup cache` <a name="cleanup-cache-command"></a>

#### Description <a name="Description-cleanup-cache"></a>

Removes the known tool caches from `/tmp/containerbase/cache`.

These are the cache folders of every supported tool, whether it is installed or not, eg. the npm cache in `.npm/_cacache`, the pnpm store in `.local/share/pnpm/store`, the maven repository in `.m2/repository` or the go module cache in `go/pkg/mod`.
The contents of the generic `.cache` folder and the virtualenv cache in `.local/share/virtualenv` are removed too.
Config files, like `.npmrc`, `.m2/settings.xml` or the conan profiles, are kept.
When a tool's cache folder or `.cache` itself is a link, eg. to a volume, its target is emptied and the link kept, a link to a missing target is skipped.
Entries inside `.cache` are removed as they are, so a link there is removed, not its target.

The download cache in `CONTAINERBASE_CACHE_DIR` is left alone, use [`cleanup downloads`](#cleanup-downloads-command) for it.
A cache folder or `.cache` entry holding it, eg. `.cache/containerbase`, is skipped.

With `--dry-run` nothing is removed and every path which would be removed is logged.

#### Usage <a name="Usage-cleanup-cache"></a>

`$ containerbase-cli cleanup cache [-d,--dry-run]`

#### Examples <a name="Examples-cleanup-cache"></a>

- Removes all known tool caches
```bash
containerbase-cli cleanup cache
```
- Lists the caches which would be removed
```bash
containerbase-cli cleanup cache --dry-run
```

<br>

### `containerbase-cli cleanup downloads` <a name="cleanup-downloads-command"></a>

#### Description <a name="Description-cleanup-downloads"></a>
Expand Down
316 changes: 316 additions & 0 deletions src/cli/cleanup/cleanup-cache.service.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,316 @@
import fs from 'node:fs/promises';
import { join } from 'node:path';
import { injectFromHierarchy, injectable } from 'inversify';
import { beforeEach, describe, expect, test, vi } from 'vitest';
import { BaseInstallService } from '../install-tool/base-install.service.ts';
import { INSTALL_TOOL_TOKEN } from '../install-tool/install-tool.service.ts';
import { PathService, createContainer } from '../services/index.ts';
import { logger, pathExists } from '../utils/index.ts';
import { CleanupCacheService } from './cleanup-cache.service.ts';
import { cachePath, rootPath } from '~test/path.ts';

@injectable()
@injectFromHierarchy()
abstract class TestInstallService extends BaseInstallService {
/** Does nothing, for tests. */
override install(_version: string): Promise<void> {
return Promise.resolve();
}

/** Does nothing, for tests. */
override link(_version: string): Promise<void> {
return Promise.resolve();
}
}

/** a tool with a cache folder */
@injectable()
@injectFromHierarchy()
class CachingInstallService extends TestInstallService {
override readonly name = 'caching';

override readonly cacheDirs = ['.caching/cache', '.shared/cache'];
}

/** another tool sharing a cache folder */
@injectable()
@injectFromHierarchy()
class SharingInstallService extends TestInstallService {
override readonly name = 'sharing';

override readonly cacheDirs = ['.shared/cache'];
}

/** a tool without cache folders */
@injectable()
@injectFromHierarchy()
class PlainInstallService extends TestInstallService {
override readonly name = 'plain';
}

/** Creates a file and its missing parent folders. */
async function touch(path: string): Promise<void> {
await fs.mkdir(join(path, '..'), { recursive: true });
await fs.writeFile(path, '');
}

describe('cli/cleanup/cleanup-cache.service', () => {
let tmpCache: string;
let svc: CleanupCacheService;

beforeEach(async () => {
tmpCache = rootPath('tmp/containerbase/cache');
await fs.rm(rootPath('tmp'), { recursive: true, force: true });
await fs.rm(rootPath('var/cache'), { recursive: true, force: true });
await fs.rm(rootPath('volume'), { recursive: true, force: true });
await fs.rm(cachePath(''), { recursive: true, force: true });
await fs.mkdir(cachePath(''), { recursive: true });

const container = createContainer();
container.bind(CleanupCacheService).toSelf();
container.bind(INSTALL_TOOL_TOKEN).to(CachingInstallService);
container.bind(INSTALL_TOOL_TOKEN).to(SharingInstallService);
container.bind(INSTALL_TOOL_TOKEN).to(PlainInstallService);
svc = await container.getAsync(CleanupCacheService);
});

test('uses the containerbase cache', async () => {
const pathSvc = await createContainer().getAsync(PathService);
expect(pathSvc.cachePath).toBe(tmpCache);
});

test('removes the known caches and keeps the rest', async () => {
await touch(join(tmpCache, '.caching/cache/file'));
await touch(join(tmpCache, '.caching/config'));
await touch(join(tmpCache, '.shared/cache/file'));
await touch(join(tmpCache, '.cache/pip/file'));
await touch(join(tmpCache, '.cache/.hidden'));
await touch(join(tmpCache, '.local/share/virtualenv/file'));
await touch(join(tmpCache, '.local/share/other'));
await touch(cachePath('download/file'));
// a dangling link counts as well
await fs.symlink('missing', join(tmpCache, '.cache/link'));

expect(await svc.cleanupCache()).toEqual([
join(tmpCache, '.cache/.hidden'),
join(tmpCache, '.cache/link'),
join(tmpCache, '.cache/pip'),
join(tmpCache, '.caching/cache'),
join(tmpCache, '.local/share/virtualenv'),
join(tmpCache, '.shared/cache'),
]);

expect(await fs.readdir(join(tmpCache, '.cache'))).toEqual([]);
// the download cache is left alone
expect(await pathExists(cachePath('download/file'))).toBe(true);
expect(await fs.readdir(join(tmpCache, '.caching'))).toEqual(['config']);
expect(await fs.readdir(join(tmpCache, '.local/share'))).toEqual(['other']);
expect(await pathExists(join(tmpCache, '.shared/cache'))).toBe(false);
expect(logger.debug).toHaveBeenCalledWith(
{ path: join(tmpCache, '.caching/cache') },
'remove cache',
);
});

test('removes the caches when the download cache is missing', async () => {
await touch(join(tmpCache, '.cache/pip/file'));
vi.stubEnv('CONTAINERBASE_CACHE_DIR', rootPath('missing-downloads'));

expect(await svc.cleanupCache()).toEqual([join(tmpCache, '.cache/pip')]);
});

test('skips paths holding the download cache', async () => {
await touch(join(tmpCache, '.cache/containerbase/download/file'));
await touch(join(tmpCache, '.cache/pip/file'));
vi.stubEnv(
'CONTAINERBASE_CACHE_DIR',
join(tmpCache, '.cache/containerbase/download'),
);

expect(await svc.cleanupCache()).toEqual([join(tmpCache, '.cache/pip')]);

expect(
await pathExists(join(tmpCache, '.cache/containerbase/download/file')),
).toBe(true);
expect(logger.debug).toHaveBeenCalledWith(
{ path: join(tmpCache, '.cache/containerbase') },
'cache holds the download cache',
);

// through a link, the download cache in the link target
await touch(rootPath('volume/downloads/file'));
await fs.mkdir(join(tmpCache, '.shared'), { recursive: true });
await fs.symlink(rootPath('volume'), join(tmpCache, '.shared/cache'));
vi.stubEnv('CONTAINERBASE_CACHE_DIR', rootPath('volume/downloads'));
expect(await svc.cleanupCache()).toEqual([
join(tmpCache, '.cache/containerbase'),
]);
expect(await pathExists(rootPath('volume/downloads/file'))).toBe(true);
});

test('dry run', async () => {
await touch(join(tmpCache, '.caching/cache/file'));

expect(await svc.cleanupCache(true)).toEqual([
join(tmpCache, '.caching/cache'),
]);

expect(await pathExists(join(tmpCache, '.caching/cache/file'))).toBe(true);
expect(logger.info).toHaveBeenCalledWith(
`Dry run: remove cache ${join(tmpCache, '.caching/cache')} ...`,
);
});

test('works without any cache', async () => {
expect(await svc.cleanupCache()).toEqual([]);
});

test('empties a linked cache folder and keeps the link', async () => {
const volume = rootPath('volume/caching');
await touch(join(volume, 'file'));
await touch(join(volume, 'dir/file'));
await touch(join(rootPath('volume/cache'), 'pip/file'));
await fs.mkdir(join(tmpCache, '.caching'), { recursive: true });
await fs.symlink(volume, join(tmpCache, '.caching/cache'));
// the generic `.cache` folder is emptied through its link too
await fs.symlink(rootPath('volume/cache'), join(tmpCache, '.cache'));

expect(await svc.cleanupCache()).toEqual([
join(tmpCache, '.cache/pip'),
join(tmpCache, '.caching/cache'),
]);

expect(await fs.readlink(join(tmpCache, '.caching/cache'))).toBe(volume);
expect(await fs.readdir(volume)).toEqual([]);
expect(await fs.readdir(rootPath('volume/cache'))).toEqual([]);
});

test('finds the caches in a configured tmp dir without the link', async () => {
vi.stubEnv('CONTAINERBASE_TMP_DIR', rootPath('var/cache/containerbase'));
await fs.mkdir(rootPath('tmp'), { recursive: true });
await touch(rootPath('var/cache/containerbase/cache/.caching/cache/file'));

expect(await svc.cleanupCache()).toEqual([
join(tmpCache, '.caching/cache'),
]);

expect(await fs.readlink(rootPath('tmp/containerbase'))).toBe(
rootPath('var/cache/containerbase'),
);
expect(
await pathExists(
rootPath('var/cache/containerbase/cache/.caching/cache'),
),
).toBe(false);
});

test('dry run reads a configured tmp dir without the link and changes nothing', async () => {
vi.stubEnv('CONTAINERBASE_TMP_DIR', rootPath('var/cache/containerbase'));
await fs.mkdir(rootPath('tmp'), { recursive: true });
const cache = rootPath('var/cache/containerbase/cache');
await touch(join(cache, '.caching/cache/file'));
await touch(join(cache, '.cache/pip/file'));

expect(await svc.cleanupCache(true)).toEqual([
join(cache, '.cache/pip'),
join(cache, '.caching/cache'),
]);

expect(await fs.readdir(rootPath('tmp'))).toEqual([]);
expect(await pathExists(join(cache, '.caching/cache/file'))).toBe(true);
expect(await pathExists(join(cache, '.cache/pip/file'))).toBe(true);
});

test('dry run reads the default cache when the link exists', async () => {
vi.stubEnv('CONTAINERBASE_TMP_DIR', rootPath('var/cache/containerbase'));
await fs.mkdir(rootPath('tmp'), { recursive: true });
await touch(rootPath('var/cache/containerbase/cache/.caching/cache/file'));
await fs.symlink(
rootPath('var/cache/containerbase'),
rootPath('tmp/containerbase'),
);

expect(await svc.cleanupCache(true)).toEqual([
join(tmpCache, '.caching/cache'),
]);
});

test('skips a linked cache folder with a missing target', async () => {
await fs.mkdir(join(tmpCache, '.caching'), { recursive: true });
await fs.symlink(
rootPath('volume/missing'),
join(tmpCache, '.caching/cache'),
);

expect(await svc.cleanupCache()).toEqual([]);

expect(await fs.readlink(join(tmpCache, '.caching/cache'))).toBe(
rootPath('volume/missing'),
);
expect(logger.debug).toHaveBeenCalledWith(
{ path: join(tmpCache, '.caching/cache') },
'cache link target missing',
);
});

test('does not make a link target writable', async () => {
const target = rootPath('volume/readonly');
await fs.mkdir(target, { recursive: true });
await fs.chmod(target, 0o555);
await fs.mkdir(join(tmpCache, '.cache'), { recursive: true });
await fs.symlink(target, join(tmpCache, '.cache/link'));
vi.spyOn(fs, 'rm').mockRejectedValueOnce(
Object.assign(new Error('EACCES'), { code: 'EACCES' }),
);

await expect(svc.cleanupCache()).rejects.toThrow(
'Failed to remove 1 cache paths',
);

expect((await fs.stat(target)).mode & 0o777).toBe(0o555);
await fs.chmod(target, 0o755);
});

test('makes a folder writable when it cannot be removed', async () => {
await touch(join(tmpCache, '.caching/cache/file'));
vi.spyOn(fs, 'rm').mockRejectedValueOnce(
Object.assign(new Error('EACCES'), { code: 'EACCES' }),
);

expect(await svc.cleanupCache()).toEqual([
join(tmpCache, '.caching/cache'),
]);

expect(await pathExists(join(tmpCache, '.caching/cache'))).toBe(false);
});

test('removes read-only folders', async () => {
const dir = join(tmpCache, '.caching/cache/mod');
await touch(join(dir, 'file'));
await fs.chmod(dir, 0o555);

expect(await svc.cleanupCache()).toEqual([
join(tmpCache, '.caching/cache'),
]);

expect(await pathExists(join(tmpCache, '.caching/cache'))).toBe(false);
});

test('removes the other paths when one fails and throws at the end', async () => {
await touch(join(tmpCache, '.caching/cache/file'));
await touch(join(tmpCache, '.shared/cache/file'));
const err = Object.assign(new Error('EBUSY'), { code: 'EBUSY' });
vi.spyOn(fs, 'rm').mockRejectedValueOnce(err);

await expect(svc.cleanupCache()).rejects.toThrow(
'Failed to remove 1 cache paths',
);

expect(logger.warn).toHaveBeenCalledExactlyOnceWith(
{ path: join(tmpCache, '.caching/cache'), err },
'cannot remove cache',
);
expect(await pathExists(join(tmpCache, '.caching/cache/file'))).toBe(true);
expect(await pathExists(join(tmpCache, '.shared/cache'))).toBe(false);
});
});
Loading
Loading