Skip to content

feat(pypi): generate requirements.bzl in the unified @pypi hub - #4223

Merged
rickeylev merged 5 commits into
bazel-contrib:mainfrom
thirtyseven:unified-hub-requirements-bzl
Oct 7, 2026
Merged

rickeylev merged 5 commits into
bazel-contrib:mainfrom
thirtyseven:unified-hub-requirements-bzl

Conversation

@thirtyseven

@thirtyseven thirtyseven commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

The unified @pypi hub (#3837) does not generate a requirements.bzl. That makes renaming a hub away from the now-reserved pypi name an all-at-once change: the moment the concrete hub stops being @pypi, every load("@pypi//:requirements.bzl", "requirement") in the repo fails to load. In our monorepo that is ~940 BUILD files, and any BUILD file that lands on main mid-migration re-breaks it. The same thing would happen to every such repo if RULES_PYTHON_PYPI_HUB_RESERVED is flipped on by default, as discussed in #3837.

This adds an opt-in, pip.default(unified_hub_requirements_bzl = True) (root module only, off by default), that generates a requirements.bzl in the unified hub with the per-package macros of a concrete hub's. It's opt-in because the docs already steer users to @pypi//<pkg> labels over requirement(); this is a migration aid for repos whose hub used to be named pypi, not a new default surface. With it on:

  • requirement(), whl_requirement(), data_requirement() and dist_info_requirement() return labels in the unified hub (@@<unified>//<pkg>:pkg etc., the same canonical-name form the concrete hubs use), so they route through --@rules_python//python/config_settings:venv exactly like a plain @pypi//<pkg> label. All of those targets already exist in every unified package (_STANDARD_ALIASES).
  • The all_* lists (all_requirements, all_whl_requirements(_by_package), all_data_requirements) are deliberately not generated. They are fixed at loading time, before the venv flag is known, so they could only list one hub's packages and would silently not follow the flag like the rest of the unified hub. Code that wants a whole lock loads them from that concrete hub, and a stale load("@pypi//:requirements.bzl", "all_requirements") fails loudly instead.

Before: after renaming hub_name = "pypi" to e.g. "pypi_main" with pip.default(default_hub = "pypi_main"), load("@pypi//:requirements.bzl", ...) fails with "no such file". After, with the opt-in: it keeps working and resolves to the same wheels, and targets can migrate to "@pypi//<pkg>" labels incrementally (we're moving our own repo to labels and gating new requirement() calls in CI). Without the opt-in nothing changes.

Tests:

  • tests/pypi/extension: the setting is off by default, on when the root module sets it, and ignored from a non-root module.
  • tests/integration/unified_pypi (which now opts in): requirement() with a non-normalized name on the default hub, requirement() under a venv transition, and loading @pypi//:requirements.bzl failing once the opt-in is removed. Passes on bazel_self and bazel_7.7.0.

Context: we're carrying this as a local patch in a large monorepo, where renaming our main hub and loading requirement from the unified hub leaves the resolved wheel set unchanged. #4172 also touches tests/integration/unified_pypi; the two should merge independently, but one of them may need a trivial rebase.

(Done with help from an agent.)

🤖 Generated with Claude Code

The unified `@pypi` hub generates no `requirements.bzl`. A repo whose hub
was named `pypi` has to rename it now that the name is reserved, and once
it does, every `load("@pypi//:requirements.bzl", "requirement")` stops
resolving at the same moment, so the rename can't be split into smaller
changes. Flipping `RULES_PYTHON_PYPI_HUB_RESERVED` on by default would
cause the same breakage for every such repo.

Generate a `requirements.bzl` in the unified hub from the same template
as a concrete hub's. `requirement()`, `whl_requirement()`,
`data_requirement()` and `dist_info_requirement()` return labels in the
unified hub, so they route through
`--@rules_python//python/config_settings:venv` like `@pypi//<pkg>`.
The `all_*` lists are fixed at loading time, before the venv flag is
known, so they list the default hub's packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thirtyseven and others added 4 commits October 5, 2026 23:26
They are fixed at loading time, before the venv flag is known, so they
could only ever list one hub's packages and would not follow the flag
like the rest of the unified hub. Load them from a concrete hub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generate the unified hub's requirements.bzl only when the root module
sets `pip.default(unified_hub_requirements_bzl = True)`. The docs steer
users to `@pypi//<pkg>` labels over the requirement() helper, so keep the
helper on the unified hub a deliberate choice for repos migrating a hub
that used to be named `pypi`, not a default surface.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_pypi MODULE.bazel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice PR and fix. Thanks!

@rickeylev
rickeylev added this pull request to the merge queue Oct 7, 2026
Merged via the queue into bazel-contrib:main with commit 966f002 Oct 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants