draft: native types codegen design and generated schemas by stobrien89 · Pull Request #857 · smithy-lang/smithy-python

stobrien89 · 2026-10-20T16:10:44Z

Issue #, if available:
#797

Description

Adds native Python schema generation and the supporting infrastructure for standalone types packages.

This PR includes the approved standalone types design and implements the schema-generation portion. Model generation and CLI integration will follow in separate PRs.

Changes

Add shared generation settings, context, and focused generation plans.
Add native Smithy selector parsing and standalone type selection.
Add dependency tracking shared by symbols, writers, and package generation.
Add staged output management with validation and ownership tracking.
Emit the compact schema representation introduced in Generate schemas as on-demand hydration from tuples #841.
Apply Java-compatible trait filtering and input/output trait synthesis.
Encode trait bit vectors, intern strings, and use prelude integer references.
Order schemas iteratively and support lazy forward and recursive references.
Add collision-safe schema constant naming.
Add a versioned compact encoding contract with smithy-core.
Add schema generation, selection, dependency, and output tests.

Follow-up work

Apply the lazy model representation introduced in Lazy load model data classes #851 to standalone types.
Generate typed stubs and package metadata.
Add measured schema and model partitioning for large generated packages.
Integrate the smithy-python generate types artifact pipeline with the CLI.
Add complete Java/native parity testing and large-model performance measurements.

Compatibility

The Java generator remains authoritative until the native implementation passes complete parity validation. Existing generation and generated packages are unaffected.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

jonathan343

Thanks @stobrien89!

The design itself looks great. Most of my feedback is about keeping the doc focused on the goal state. Anything that describes current state, specific PRs, or the Java migration will go stale as we land the implementation, so I'd move that to the PR description or #797.

Copilot

🟡 Changes recommended

Output-path safety, stale-file cleanup, selector evaluation, and generated-name collisions remain unresolved.

5 open issues

What changed in this PR

This PR lays the foundation for native Python standalone-types generation under #797. It adds compact schema emission and shared generation infrastructure while leaving the Java generator authoritative.

Changes:

Add selector parsing, standalone-type selection, generation plans, and compact schema emission.
Introduce dependency tracking and staged file publication with ownership manifests.
Expand unit coverage and document the planned standalone-types pipeline.

File	Description
`packages/smithy-python/tests/unit/test_writer.py`	Test dependency collection from type references.
`packages/smithy-python/tests/unit/test_symbols.py`	Cover dependency metadata and streaming schema names.
`packages/smithy-python/tests/unit/test_selectors.py`	Test selector parsing and evaluation.
`packages/smithy-python/tests/unit/test_schemas.py`	Test compact encoding, hydration, and deterministic output.
`packages/smithy-python/tests/unit/test_generation.py`	Test generation settings, context, and plans.
`packages/smithy-python/tests/unit/test_files.py`	Test staging, ownership, cleanup, and path checks.
`packages/smithy-python/tests/unit/test_dependencies.py`	Test dependency validation and merging.
`packages/smithy-python/tests/unit/test_data_selection.py`	Cover standalone-type selection rules.
`packages/smithy-python/src/smithy_python/writer.py`	Record dependencies from nested references.
`packages/smithy-python/src/smithy_python/symbols.py`	Plan schema and union names; attach dependencies.
`packages/smithy-python/src/smithy_python/selection.py`	Add standalone-type selection and dependency closure.
`packages/smithy-python/src/smithy_python/schemas.py`	Plan and emit compact, lazy schemas.
`packages/smithy-python/src/smithy_python/model/_shapes.py`	Define shared declaration shape types.
`packages/smithy-python/src/smithy_python/model/__init__.py`	Export declaration shape types.
`packages/smithy-python/src/smithy_python/generation.py`	Introduce shared settings, context, and plans.
`packages/smithy-python/src/smithy_python/files.py`	Implement staged publication and ownership tracking.
`packages/smithy-python/src/smithy_python/exceptions.py`	Add selector syntax errors.
`packages/smithy-python/src/smithy_python/dependencies.py`	Define dependency records and runtime requirements.
`packages/smithy-python/src/smithy_python/_selector.py`	Implement Smithy selector parsing and evaluation.
`packages/smithy-python/.changes/next-release/smithy-python-feature-schema-generation.json`	Announce native compact schema generation.
`packages/smithy-core/src/smithy_core/_schema_compact.py`	Expose the compact encoding version.
`designs/codegen/writer.md`	Document writer dependency tracking.
`designs/codegen/types.md`	Describe the standalone-types architecture and compatibility goals.
`designs/codegen/symbols.md`	Document dependency-bearing type references.
`designs/codegen/selection.md`	Describe standalone selection and selectors.
`designs/codegen/index.md`	Link the standalone-types design.
`designs/codegen/cli.md`	Update documented artifact-specific selection rules.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Unresolved correctness issues affect schema semantics, shape selection, and safe output publication.

6 open issues

5 resolved since last review

_{Review effort: Balanced}

+        visited: set[Shape] = set()
+        pending = [(shape, False)]
+        while pending:
+            current, qualified = pending.pop()
+            if current in visited:
+                continue
+            visited.add(current)


+        parts = PurePosixPath(path).parts
+        current = output
+        for index, part in enumerate(parts):
+            current /= part


+    _shape_id("smithy.api#PrimitiveBoolean"): 2,
+    _shape_id("smithy.api#PrimitiveByte"): 5,
+    _shape_id("smithy.api#PrimitiveShort"): 6,
+    _shape_id("smithy.api#PrimitiveInteger"): 7,


+        self.shapes = tuple(shapes)
+        virtual_traits = _virtual_trait_shapes(model, self.shapes)
+        graph_shapes = (*self.shapes, *virtual_traits.values())


+        self._ws()
+        if self._take("@"):
+            self._expect("{")
+            path = self._parse_path(allow_empty=False)


+
+    shapes = tuple(shape for shape in model.iter_shapes() if shape.id in closure)
+    _check_names(
+        tuple(shape for shape in shapes if shape.type in DECLARATION_TYPES),


stobrien89 force-pushed the native-types-codegen branch from ab0648a to 5d75777 Compare October 20, 2026 17:33

add types design doc

cf12e65

stobrien89 force-pushed the native-types-codegen branch from 5d75777 to cf12e65 Compare October 20, 2026 20:23

jonathan343 reviewed Oct 20, 2026

View reviewed changes

address feedback, de-slop

bbf1e2a

stobrien89 force-pushed the native-types-codegen branch from 69872c9 to bbf1e2a Compare October 20, 2026 23:49

stobrien89 added 2 commits October 20, 2026 20:20

add generation context, dependency tracking, file manifests

5f9eca6

add selector and generation planning

68f3567

stobrien89 changed the title ~~draft: native types codegen design~~ draft: native types codegen design and generated schemas Oct 21, 2026

stobrien89 requested a balanced review from Copilot October 21, 2026 23:19

Copilot started reviewing on behalf of stobrien89 October 21, 2026 23:20

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

generate schemas

eee35b8

stobrien89 force-pushed the native-types-codegen branch from 743a2bb to eee35b8 Compare October 21, 2026 23:39

stobrien89 requested a balanced review from Copilot October 21, 2026 23:40

Copilot started reviewing on behalf of stobrien89 October 21, 2026 23:45

View session

stobrien89 marked this pull request as ready for review October 21, 2026 23:51

stobrien89 requested a review from a team as a code owner October 21, 2026 23:51

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Navigation Menu

Repository navigation

draft: native types codegen design and generated schemas - #857

draft: native types codegen design and generated schemas#857
stobrien89 wants to merge 5 commits into
developfrom
native-types-codegen

stobrien89 commented Oct 20, 2026 •

edited

Loading

Uh oh!

jonathan343 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

stobrien89 commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Follow-up work

Compatibility

Uh oh!

jonathan343 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stobrien89 commented Oct 20, 2026 •

edited

Loading