Load Smithy models and select the shapes to generate - #795
Draft
jonathan343 wants to merge 20 commits into
Draft
jonathan343 wants to merge 20 commits into
jonathan343 wants to merge 20 commits into
Conversation
jonathan343
added this pull request to stack #796
September 13, 2026 06:47
Mark generation command examples as schematic and clarify option validation behavior. Exercise python -m smithy_python in a subprocess.
Document that --service is optional when the model contains a single service, that both artifacts generate every data shape in the model rather than the service closure, and that case-insensitive name conflicts are a hard error. Note that run plugin env settings may back command-line options.
Generating every shape in the model diverged from every other Smithy generator and fails on a published AWS model whose leaked, unconnected shapes collide with real ones. Document the service closure as the default selection, with a note when shapes are left out, and keep the whole-model behavior for the types artifact when no service is present.
Port the JSON AST model loader from the Python-native proof of concept and add the selection rules described in the CLI design. The CLI now parses the model before reporting that generation is unimplemented. - Add ordered, immutable Model, Shape, Member, and ShapeID types that resolve prelude shapes on demand and honor apply statements. - Add an optional --service option. A model with one service selects it automatically; multiple services require an explicit choice; the client artifact requires a service and types does not. - Select every data shape in the model rather than the service closure, skipping prelude shapes, trait definitions, and mixins. Private shapes are generated only when reachable from another generated shape or an operation. Case-insensitive name conflicts fail generation. - Distinguish invocation errors (exit 2) from model and generation failures (exit 1).
Smithy serializes only what a shape introduces, so shapes that use mixins arrive without their inherited members, traits, errors, and service operations, and traits added to inherited members arrive as apply statements. Resolve mixins while loading, following the specification's precedence and ordering rules, and apply traits in two passes so that customizations of inherited members are honored. Mixin services are abstract, so they are no longer service candidates and cannot be selected with --service.
Generating every shape in the model diverged from every other Smithy generator and failed on a published AWS model whose leaked, unconnected shapes collide with real ones. Select the service closure when a service is resolved, matching the existing SDK surface, and report how many unconnected shapes were left out. Without a service, the types artifact still generates every data shape and fails on case-insensitive name conflicts, which Smithy only guarantees within a closure. Synthesized prelude shapes now carry their real traits: defaults on the Primitive* shapes and unitType on Unit.
Applying traits to inherited members after all mixins were resolved meant a shape using an intermediate mixin copied that mixin's members before its apply statements were merged. Merge each shape's applies as part of resolving it, before anything inherits from it, and drop the two-pass bookkeeping.
The model's containers are read-only views, but the JSON values inside them are not copied. Say so rather than claiming full immutability.
Trait and attribute values inherited through mixins are shared between the mixin and every shape that uses it, so a mutable nested value let one shape's consumer corrupt its siblings. Freeze JSON objects as read-only mappings and arrays as tuples when parsing, and narrow JSONValue to the read-only abstract types so pyright rejects mutation statically as well.
Share one empty mapping across the whole model instead of allocating a proxy per shape, layer the prelude under the shape index so a lookup is a single step rather than a fallback chain, and replace the branch ladder in Shape.references with a table naming the attributes that hold references. Move the trait accessors that Member and Shape each defined to a shared base, and the member scan that Shape and Model each open-coded to Shape.get_member. Parsing a 2.5 MB model is about 35% faster and retains 23% less memory as a result. Collapse the CLI's two-layer invocation handoff into one request built by one function, and route its output through helpers that own the program prefix so it is spelled once. Tighten the behaviors the loader was lenient about along the way: an apply statement must target a member, since a shape's own traits are serialized with its definition; a mixin must have the same shape type as the shape using it; a list or map that inherits its members no longer needs to restate them; and a member ID resolves only when the shape actually defines that member. Require --output for direct invocation rather than defaulting it, and reject it alongside SMITHY_PLUGIN_DIR instead of silently letting the plugin's directory win.
jonathan343
force-pushed
the
python-codegen-model
branch
from
September 13, 2026 06:58
8858fd4 to
560ac4d
Compare
Model.get and Model.expect returned the containing shape when given a member ID, so a caller resolving `ns#Shape$member` received a Shape whose type and traits belonged to the container rather than the member. Reject member IDs in the shape lookups and add Model.get_member and Model.expect_member, which return the Member itself and reject shape IDs symmetrically.
Any string was accepted as the `smithy` version, so a Smithy 1.0 AST failed later with an unrelated error such as "Unsupported shape type 'set'". Check the major version while loading and report the version along with how to fix it. Document the 2.x requirement in the CLI design.
A missing --model path was pre-checked and reported as an invocation error with exit 2, while a file that existed but could not be read raised OSError and exited 1. The design classifies I/O failures as 1, and tools that distinguish usage errors from runtime failures, including the Smithy CLI, treat a missing input file as the latter. Drop the pre-check so every unreadable model exits 1 with a message that names the path and the cause. Also state in the design that model failures return 1, which the implementation already did but the text left implicit.
Bring in the exit-code fix for unreadable model files. The CLI merged cleanly; the two affected tests were rewritten in this branch's run_cli style with the new exit code and message.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #744; the base is
add-smithy-python, so this diff is only the model-loading layer.The CLI currently has no way to read a model. It now loads the Smithy JSON AST into an immutable shape index, resolves the service to generate, and selects the shapes to generate from it. Smithy serializes only what a shape introduces, so shapes using mixins arrive without their inherited members, traits, errors, and operations; we resolve mixins on load, merging each shape's apply statements while resolving it so that shapes inheriting through an intermediate mixin see its applied traits. Values inherited from a mixin are shared with every shape using it, so nested objects are frozen as read-only mappings and arrays as tuples, with
JSONValuenarrowed to the read-only abstract types so pyright rejects mutation statically.We currently generate every shape in the model, which diverges from every other Smithy generator and fails on a published AWS model whose leaked, unconnected shapes collide with real ones. We now generate the resolved service's closure — named with
--servicewhen a model declares several — and report how many shapes were excluded. Without a service, thetypesartifact still generates every data shape and reports case-insensitive name conflicts with aremoveUnusedShapessuggestion, since Smithy only guarantees unique names within a closure.The loader also rejects what it previously accepted: an apply statement must target a member, a mixin must share the shape type of its user, and a member ID resolves only when the shape defines that member.
--outputis now required for direct invocation instead of defaulting. Parsing a 2.5 MB model is 35% faster and retains 23% less memory.Test Plan
1586 unit tests with 100% coverage of the new modules;
ruffandpyright --strictclean.Loaded all 431 AWS service models in
api-models-awsand 311 fixtures fromsmithy-model's test resources, checking each parsed model against its source AST. Fault injection confirmed the check caught unmerged mixins, dropped shapes, and unmerged applies. The 1.0setshape and shape-levelapply, which only resolves across documents, are unsupported.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.