Skip to content

perf(nodejs): derive a monorepo function's dependency closure from the lockfile - #938

Open
bnusunny wants to merge 1 commit into
aws:developfrom
bnusunny:fix/nodejs-closure-from-lockfile
Open

bnusunny wants to merge 1 commit into
aws:developfrom
bnusunny:fix/nodejs-closure-from-lockfile

Conversation

@bnusunny

@bnusunny bnusunny commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available: N/A — a performance change on the path #933 added. No linked issue.

Description of changes

Goal: stop spawning one npm ls per function when building a workspaces monorepo in source, by reading the closure out of the lockfile npm already wrote.

#935 narrows a function's artifacts to its own dependencies by asking npm ls --all --parseable --omit=dev which packages that function resolves. That answer is correct, and it costs an npm process per function — on a 20-function monorepo, twenty sequential node startups whose only job is to re-derive what package-lock.json already records.

lockfile_closure.production_closure derives the same set from the lockfile. The lockfile's packages map is keyed by install path, so resolving a dependency is the same walk node performs: look for <prefix>/node_modules/<name>, strip a path segment, repeat. A workspace link resolves to its real source directory, which is what npm's --parseable output also prints.

npm invocations median wall clock
8 functions, npm ls 26 6.11 s
8 functions, lockfile 19 3.68 s
20 functions, npm ls 62 14.77 s
20 functions, lockfile 43 8.31 s

Measured on two generated fixture sets, npm 11.19.0, Linux; medians of repeated runs.

npm ls stays as the fallback and nothing about the artifacts changes. lockfileVersion 1 has no per-path map, so the resolver returns None for it, as it does for an unreadable lockfile or an install directory the lockfile does not describe — and the caller then asks npm exactly as before. The experimental flag gating is unchanged: this runs only where #935's link step already runs.

Description of how you validated changes

The lockfile's answer is checked against npm's, not against a fixture. test_the_lockfile_closure_agrees_with_npm_ls_on_a_real_install runs a real npm install on the workspaces-monorepo fixture and asserts both endpoints' closures are set-equal between production_closure and npm ls --all --parseable --omit=dev. A unit test cannot establish this — it would compare this code against a fixture written from this code.

Mutation-checked: removing the walk-up from _resolve_key makes that test fail, naming the two directories it then loses (packages/shared and the hoisted minimal-request-promise).

Unit tests cover the resolver directly — a member's own dependencies plus its shared workspace package, a conflicting version resolving to the nested copy rather than the hoisted one, a member that wants the hoisted version getting both copies, dev dependencies excluded, an entry not on disk excluded — and each way it declines to answer: version 1, an undescribed install directory, an unreadable file.

Two tests pin the wiring, which is where a silent regression would hide: test_a_usable_lockfile_answers_the_closure_without_running_npm asserts resolve_dependency_closure is never called, and test_a_lockfile_that_cannot_answer_falls_back_to_npm asserts it is called once. Without the first, the lockfile path could stop being used and every test would still pass.

  • 874 unit tests pass; ruff check and black --check clean.
  • nodejs + esbuild integration: 301 passed, npm 11.19.0, Linux.
  • The existing #933 monorepo integration test now exercises the lockfile path end to end, since the workflow passes the lockfile in — so the artifact assertions it already makes cover this change too.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@bnusunny
bnusunny requested a review from a team as a code owner October 9, 2026 21:44

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: dfe97f9..8abc789
Files: 6
Comments: 1

Comment thread aws_lambda_builders/workflows/nodejs_npm/lockfile_closure.py Outdated

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: dfe97f9..1b5b92b
Files: 8
Comments: 1

Comment thread aws_lambda_builders/workflows/nodejs_npm/actions.py Outdated
…pm ls

The artifacts link asks npm which packages a workspace function resolves, and
`npm ls --all --parseable --omit=dev` costs 0.33s per function - the largest
remaining per-function cost once the install is no longer the bottleneck. The
lockfile the build already locates describes the same resolved tree, so read it
instead: walk the lockfileVersion 2/3 `packages` map from the member's own entry,
resolve each name the way node does (`<key>/node_modules/<name>`, walking up),
follow `link: true` entries to their source directory, and skip anything marked
`dev`. A version 1 lockfile has no path map, so that falls back to `npm ls`, as
does a lockfile that cannot be read.

An entry can be in the lockfile without being on disk: `fsevents` is
darwin-only and optional, so webpack's and jest's closures list it while Linux
never installs it. `npm ls` reports the installed tree, so the resolver keeps
only directories that exist; a required dependency missing there means the
install itself failed, which the install step already surfaces.

Behaviour is unchanged, which is the point - the existing integration tests that
assert each function's artifacts hold its own dependencies and not its sibling's
are the regression proof, and they pass unchanged. The new unit tests cover the
resolver itself: nested copies, scoped names, workspace links, dev entries,
version 1 fallback, an unreadable lockfile, and an entry that is not on disk.
@bnusunny
bnusunny force-pushed the fix/nodejs-closure-from-lockfile branch from 1b5b92b to fb8fcca Compare October 9, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant