feat: add authz schema loading by rodmgwgu · Pull Request #475 · openedx/openedx-authz

rodmgwgu · 2026-09-23T16:19:22Z

Problem

Discovered resources are still opaque bytes. They need to be parsed into typed schema documents, with the source identity and content digest that later phases and the provenance tables depend on (ADR 0018 §1).

Approach

The load phase, plus the shared vocabulary the remaining phases are written against.

openedx_authz/engine/schema/loading.py — SchemaLoader
openedx_authz/engine/schema/types.py — the schema dataclasses
openedx_authz/engine/schema/exceptions.py — the pipeline exception hierarchy
openedx_authz/constants/schema.py — SchemaOriginKind
openedx_authz/tests/schema/{factories,test_loading,test_types}.py

types.py and exceptions.py land whole rather than sliced across the stack: they are the shared vocabulary, and splitting them would make every later PR conflict on the same two files. Some names in them (CompiledSchema, CompiledDefinition, RelationshipSource, and the compile/validate/apply errors) are unused until later PRs.

Manual testing instructions

pytest openedx_authz/tests/schema/

Rollback plan

Revert this PR. Nothing consumes the loader yet, so the revert is inert.

Retro compatibility

No authorization behavior changes. No models, no migration, no automatic code path.

AI Usage

Kiro was used to assist on feature planning and implementation. Implementation was done step by step with human guidance and validation, based on the ADRs.

Stack (2/8) — #446 split into reviewable pieces. Bases chain bottom-up; merge in order.

feat: add authz schema discovery #474 — schema discovery
This PR — schema loading (base: rod/authz-schema-discovery)
rod/authz-schema-compiler — schema compilation
rod/authz-schema-validator — schema validation
rod/authz-schema-models — definition models + migration
rod/authz-schema-renderer — policy renderer
rod/authz-schema-applier — schema applier
feat: add authz schema pipeline and load_authz_schema command #446 — pipeline + load_authz_schema command, version bump and changelog

Base is the previous branch; review only the incremental diff.

Merge checklist:

Version bumped — deferred to 8/8 so the stack does not conflict on every restack
Changelog record added — deferred to 8/8
Documentation updated (not only docstrings) — covered by the already-merged ADRs
Fixup commits are squashed away
Unit tests added/updated
Manual testing instructions provided
Noted any: Concerns, dependencies, migration issues, deadlines, tickets — depends on feat: add authz schema discovery #474; no migration

openedx-webhooks · 2026-09-23T16:19:28Z

Thanks for the pull request, @rodmgwgu!

This repository is currently maintained by @openedx/committers-openedx-authz.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

mariajgrimaldi

A few comments for you to review! Thanks a lot for this :))

mariajgrimaldi · 2026-09-24T14:19:31Z

+        except Exception:  # noqa: BLE001 - defensive; metadata quirks across envs
+            mapping = {}
+        candidates = mapping.get(top_level) or []
+        if candidates:


Could there be duplicates?

In theory yes, I'm adding a way to deterministically choosing the candidate:

First search for the distribution that actually has the discovered resource file

If more than one matches exists, sort and return the first one as a deterministic fallback (and log a warning about it.)

I wonder if these decisions should be recorded somehow. When we merge this stack, can we do like a reverse-engineering review to capture the deliberate decisions we're making here?

mariajgrimaldi · 2026-09-24T15:36:35Z

+    ``hidden`` is tri-state: ``None`` leaves the current value untouched.
+    """
+
+    role: str


Should we use some kind of role_id to reference the role we're extending?

This also comes form the yaml definition as specified in the ADRs

The reference mentions role with a description of role ID, which is very straightforward. By itself not so much. We can update the ADRs and references if needed. Let me know what you think!

mariajgrimaldi

LGTM! Thanks so much for addressing my comments!

(The comments left to address are only nits so I'll leave my approval! :))

ADR 0020 asks for the static schema's display_name/description fields to go through OEP-58 translation. Standard extraction tools can't see them because they live in YAML, not Python or JavaScript source. Add a management command that discovers every schema resource (reusing the already-merged SchemaDiscovery step), parses its YAML directly, and writes a generated Python module with one pgettext() call per field, for i18n_tool extract (Django's makemessages) to pick up. pgettext is used instead of plain gettext because ADR 0020 requires a field's display name and description to stay separate messages even when they're identical text; the context string also carries the field's stable id, so the same one-word name on two different roles stays separate too. Wired into the extract_translations Makefile target, ahead of the existing i18n_tool extract step. This only covers the extraction/translation-source side (ADR 0020 points 1-2). Routing the translated strings back out through an API (points 3-4) isn't buildable yet: the schema loader/model layer that would read these fields at runtime, and the role/permission catalog API itself, are both still open PRs (openedx#475-480, openedx#508), not the already-merged discovery step this extraction reuses. Left a comment on openedx-authz#432 with that split and will file a follow-up once those land. This doesn't touch the schema load/validate/compile pipeline at all (intentionally): it only needs raw display_name/description fields, not resolved role extensions or conflict handling, so it stays usable and testable independent of that still-unmerged pipeline.

* feat: extract translatable strings from the authz schema ADR 0020 asks for the static schema's display_name/description fields to go through OEP-58 translation. Standard extraction tools can't see them because they live in YAML, not Python or JavaScript source. Add a management command that discovers every schema resource (reusing the already-merged SchemaDiscovery step), parses its YAML directly, and writes a generated Python module with one pgettext() call per field, for i18n_tool extract (Django's makemessages) to pick up. pgettext is used instead of plain gettext because ADR 0020 requires a field's display name and description to stay separate messages even when they're identical text; the context string also carries the field's stable id, so the same one-word name on two different roles stays separate too. Wired into the extract_translations Makefile target, ahead of the existing i18n_tool extract step. This only covers the extraction/translation-source side (ADR 0020 points 1-2). Routing the translated strings back out through an API (points 3-4) isn't buildable yet: the schema loader/model layer that would read these fields at runtime, and the role/permission catalog API itself, are both still open PRs (#475-480, #508), not the already-merged discovery step this extraction reuses. Left a comment on openedx-authz#432 with that split and will file a follow-up once those land. This doesn't touch the schema load/validate/compile pipeline at all (intentionally): it only needs raw display_name/description fields, not resolved role extensions or conflict handling, so it stays usable and testable independent of that still-unmerged pipeline. * test: cover the extract_schema_translations command and empty-file case The management command had zero test coverage, which dropped overall project coverage and failed the codecov check on this PR. Add tests that invoke it through call_command for both the happy path (writes a valid generated module) and the failure path (a CommandError on extraction failure), and add the one remaining uncovered branch in translation.py: an empty YAML file parsing to None instead of a dict. * fix: validate schema shape and resolve conflicts by priority in extraction Addresses review feedback on the extraction mechanism: - A malformed section (not a list) or entry (not a mapping) now raises SchemaTranslationExtractionError instead of an AttributeError from deep inside a .get() call on a string or int. - An entry missing its identifying field(s) (a role without id, a permission without namespace/name, etc.) now raises, instead of silently producing duplicate or anonymous contexts. - Conflicts between two documents defining the same role/permission/ category now resolve by each document's priority, the same rule engine.schema.compilation uses: higher priority wins, identical content merges, and a disagreement at equal priority raises. This replaces resolving by file path order, which didn't necessarily match the winner the compiled schema would actually serve. - A display_name/description containing gettext's own msgctxt/msgid separator byte (U+0004) now raises instead of generating a module that's valid Python but breaks msguniq once makemessages runs it. - A missing display_name/description is now skipped for every entry kind, not just role_extensions, instead of generating a useless empty pgettext() call. - The four schema sections (permission_categories, permissions, roles, role_extensions) are now driven by one data table instead of repeated per-section code, since they only differ in three things: the document key, the context's field_kind label, and how to read the entry's stable id. - Added an end-to-end test exercising the real extract_schema_translations -> makemessages pipeline, skipped where the system gettext tools it shells out to aren't installed. Also updates the module docstring and PR description: the schema loader/validate/compile pipeline this module was written to stay independent of is merged now, and the role/permission catalog API has its own PR (#510). * fix: add polib as a direct test dependency The new end-to-end test in test_commands.py imports polib to parse the generated .po file. It was only ever present transitively through edx-i18n-tools, which is in the dev dependency group, not test, so CI's django52 and quality jobs failed importing test_commands.py with ModuleNotFoundError.

openedx-webhooks added open-source-contribution PR author is not from Axim or 2U core contributor PR author is a Core Contributor (who may or may not have write access to this repo). labels Sep 23, 2026

openedx-webhooks added this to Contributions Sep 23, 2026

github-project-automation Bot moved this to Needs Triage in Contributions Sep 23, 2026

rodmgwgu added this pull request to stack #481 September 23, 2026 16:41

rodmgwgu requested review from BryanttV, carlos-marquez-wgu and mariajgrimaldi September 23, 2026 17:18

carlos-marquez-wgu approved these changes Sep 23, 2026

View reviewed changes

mariajgrimaldi reviewed Sep 24, 2026

View reviewed changes

rodmgwgu force-pushed the rod/authz-schema-loader-only branch from c9a67dc to bc3bd6b Compare September 24, 2026 21:35

rodmgwgu requested a review from mariajgrimaldi September 28, 2026 15:09

rodmgwgu force-pushed the rod/authz-schema-loader-only branch from 6bbb22e to 442e19d Compare September 29, 2026 15:12

mphilbrick211 added the mao-onboarding Reviewing this will help onboard devs from an Axim mission-aligned organization (MAO). label Sep 29, 2026

Base automatically changed from rod/authz-schema-discovery to main September 30, 2026 21:59

feat: add authz schema loading

bc50391

rodmgwgu force-pushed the rod/authz-schema-loader-only branch from 442e19d to bc50391 Compare September 30, 2026 22:38

This was referenced Oct 1, 2026

Implement extraction and translation of static YAML authz metadata (Internationalization) #432

Closed

feat: extract translatable strings from the authz schema #509

Merged

squash!: Attend PR comments

ccad004

rodmgwgu added 2 commits October 1, 2026 16:19

squash!: Fix lint issues

10bb977

squash!: Refactor types

70302e9

mariajgrimaldi approved these changes Oct 2, 2026

View reviewed changes

rodmgwgu added 2 commits October 2, 2026 10:50

squash!: refactor schema keys to be clearer

9a8a1f0

squash!: Fix CI

0918cbb

rodmgwgu merged commit 4310932 into main Oct 2, 2026
8 checks passed

rodmgwgu deleted the rod/authz-schema-loader-only branch October 2, 2026 17:01

BryanttV linked an issue Oct 2, 2026 that may be closed by this pull request

Core Loader, Compilation, and Definition Validation #410

Closed

Conversation

rodmgwgu commented Sep 23, 2026

Problem

Approach

Manual testing instructions

Rollback plan

Retro compatibility

AI Usage

Uh oh!

openedx-webhooks commented Sep 23, 2026

Uh oh!

mariajgrimaldi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mariajgrimaldi Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

rodmgwgu Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mariajgrimaldi Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mariajgrimaldi Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

rodmgwgu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

mariajgrimaldi Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mariajgrimaldi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rodmgwgu Oct 1, 2026 •

edited

Loading