Load Smithy models and select the shapes to generate

README.md

@@ -12,6 +12,11 @@ This code generator, and the clients it generates, are unstable and should not
 be used in production systems yet. Several features, such as detailed logging,
 have not been implemented yet.
 
+> [!NOTE]
+> The Java generator in `codegen` remains the authoritative implementation.
+> `packages/smithy-python` contains an experimental Python-native CLI scaffold
+> that does not generate code yet.
+
 ### What is this repository?
 
 This repository contains two major components:
@@ -20,9 +25,9 @@ This repository contains two major components:
 2) Core modules and interfaces for building service clients in Python
 
 These components facilitate generating clients for any [Smithy](https://smithy.io/)
-service. The `codegen` directory contains the source code for generating clients.
-The `python-packages` directory contains the source code for the handwritten python
-components.
+service. The `codegen` directory contains the current Java generator,
+`packages/smithy-python` contains the Python-native generator scaffold, and the
+other directories under `packages` contain the handwritten Python components.
 
 This repository does *not* contain any generated clients, such as for S3 or other
 AWS services. Rather, these are the tools that facilitate the generation of those

designs/codegen/cli.md

@@ -0,0 +1,146 @@
+# Code Generator CLI
+
+The `smithy-python` command is the process interface described in the
+[Python Code Generation](index.md) overview. It supports direct use and
+invocation from Smithy's
+[`run` plugin](https://smithy.io/2.0/guides/smithy-build-json.html#run-plugin).
+
+## Commands
+
+Generation is organized by artifact type:
+
+```console
+smithy-python generate client [OPTIONS]
+smithy-python generate types [OPTIONS]
+```
+
+`client` generates a service client and the data shapes it uses. `types`
+generates a standalone package containing only data shapes. Both commands accept
+the following process options:
+
+* `--model PATH` reads a JSON AST from a file instead of standard input.
+* `--output PATH` selects the output directory. It is required for direct
+  invocation and MUST NOT be used when the Smithy run plugin supplies the output
+  directory (`SMITHY_PLUGIN_DIR`).
+
+Settings specific to each artifact will be added with the functionality that
+consumes them.
+
+### Service Selection
+
+The CLI does not require a service to be named. It resolves the service to
+generate as follows:
+
+* `--service SHAPE_ID` selects a specific service shape. The shape MUST exist in
+  the model and MUST be a service.
+* When `--service` is omitted and the model contains exactly one service shape,
+  that service is used.
+* When `--service` is omitted and the model contains more than one service
+  shape, the command fails with an invocation error that lists the candidates.
+
+The `client` artifact requires a resolved service. The `types` artifact does
+not. The CLI MUST NOT synthesize a placeholder service to satisfy generation.
+
+### Generated Shapes
+
+When a service is resolved, both artifacts generate the data shapes in the
+service closure: every shape reachable from the service through its operations,
+resources, errors, and members. This matches the surface produced by the other
+Smithy code generators. Data shapes in the model that are not connected to the
+service are not generated, and the CLI reports how many were left out.
+
+When no service is resolved, the `types` artifact generates every data shape in
+the model. Smithy guarantees case-insensitively unique shape names only within a
+service closure, so in this mode the command fails when two shapes have
+case-insensitively equal names, identifying the conflicting shape IDs.
+
+Trait definitions, prelude shapes, and shapes marked `@mixin` are never
+generated. Builds that need a different set of shapes, such as types that are
+not bound to any operation, apply smithy-build transforms in the projection.
+An option to generate every shape in the model regardless of the service MAY be
+added when there is a need for it.
+
+The command MUST return zero after successful generation and non-zero when
+arguments, settings, the model, or generation are invalid. Diagnostics are
+written to standard error. Invalid command syntax and invocation inputs, such as
+options that cannot be combined or a service that cannot be selected, return 2.
+Model, I/O, and generation failures, including a model file that cannot be read,
+return 1.
+
+## Smithy `run` Plugin
+
+The Smithy `run` plugin executes an external program during a build. It sends the
+projection's Smithy model as a JSON AST to the process's standard input and runs
+the process in the plugin's output directory.
+
+A plugin ID MUST use `run::` followed by a custom artifact name. The configured
+command identifies the artifact to generate:
+
+```json
+{
+    "version": "1.0",
+    "projections": {
+        "client": {
+            "plugins": {
+                "run::python-client": {
+                    "command": ["smithy-python", "generate", "client"]
+                }
+            }
+        }
+    }
+}
+```
+
+Artifact-specific options will be appended to `command` after they are defined.
+The `run` plugin can also pass settings through its `env` property, so an option
+MAY additionally be read from an environment variable. A command-line option
+takes precedence over its environment variable.
+
+The `smithy-python` executable MUST be installed or otherwise available on the
+Smithy process's `PATH`. Smithy passes no arguments other than those in
+`command`.
+
+### Input and Output
+
+When invoked by Smithy, the CLI reads one JSON AST document from standard input.
+The document represents the model after projection transforms have been applied.
+Only Smithy 2.x JSON ASTs are supported; a document declaring another `smithy`
+version is rejected with an error that names the version.
+
+Smithy serializes only what a shape introduces, so shapes that use mixins arrive
+without their inherited members, traits, and properties, and traits added to
+inherited members arrive as `apply` statements. The CLI resolves mixins while
+loading the model, following the rules of the
+[Smithy mixins specification](https://smithy.io/2.0/spec/mixins.html), so builds
+do not need the `flattenAndRemoveMixins` transform.
+
+The presence of `SMITHY_PLUGIN_DIR` identifies an invocation by the `run` plugin.
+Generated files are written beneath this directory, which Smithy also uses as the
+process's working directory. The CLI MUST NOT write generated files outside it,
+and `--model` and `--output` MUST NOT be used in this mode.
+
+The `run` plugin provides the following environment variables:
+
+| Name | Purpose |
+|------|---------|
+| `SMITHY_ROOT_DIR` | Root directory of the Smithy build. |
+| `SMITHY_PLUGIN_DIR` | Output and working directory for the plugin. |
+| `SMITHY_PROJECTION_NAME` | Name of the active projection. |
+| `SMITHY_ARTIFACT_NAME` | Custom artifact name from the plugin ID. |
+| `SMITHY_INCLUDES_PRELUDE` | Whether the JSON AST includes prelude shapes. |
+
+The CLI uses this context to interpret the model. Protocol and platform
+integrations MAY also use it while generating files.
+
+Smithy omits prelude shapes by default. A build MAY set `sendPrelude` to `true`
+in the `run` plugin configuration when those shapes are needed.
+
+## Direct Invocation
+
+When `SMITHY_PLUGIN_DIR` is absent, the CLI treats the command as a direct
+invocation and requires `--output`. It follows the same
+generation path as Smithy invocation and can read a JSON AST from a file instead
+of standard input by using `--model`. When standard input is an interactive
+terminal, `--model` is required so that an omitted input does not wait indefinitely
+for input. This mode is intended for development, testing, and integration with
+tools other than the Smithy CLI.

designs/codegen/index.md

@@ -0,0 +1,70 @@
+# Python Code Generation
+
+Smithy Python currently generates clients with the Java implementation in
+`codegen`. This document describes the Python code generator that will replace
+that implementation over time.
+
+The Python generator is distributed as `smithy-python`. It is separate from the
+runtime packages used by generated code, and is only needed while generating a
+package.
+
+## Goals
+
+* Generate Python clients and standalone types packages from Smithy models.
+* Integrate with standard Smithy builds without requiring a Java code generator.
+* Provide extension points for protocol and platform-specific behavior.
+* Produce code compatible with the existing Smithy Python runtime packages.
+* Allow the Python and Java generators to coexist during migration.
+
+## Architecture
+
+The generator consumes a Smithy JSON AST and settings for an artifact. It loads
+the model, applies artifact and protocol-specific behavior, and writes a Python
+package.
+
+```text
+Smithy JSON AST + settings
+            |
+            v
+    smithy-python generator
+            |
+            v
+ client or types package
+```
+
+Two artifact types are initially planned:
+
+* `client` will generate a service client and its required types.
+* `types` will generate a standalone package of types selected from a model.
+
+The artifact set may grow over time. A `server` artifact is a natural addition,
+so the generator should not assume that only `client` and `types` exist.
+
+The command-line interface is the generator's first entry point. Smithy's `run`
+plugin invokes it as an external process, so the generator does not need to be
+loaded into the Smithy CLI or implemented in Java.
+
+Generated packages MUST NOT depend on `smithy-python` at runtime. They MAY
+depend on the handwritten runtime packages in this repository.
+
+The generator has no runtime dependencies of its own, including on those
+packages. It therefore defines its own shape IDs, shape types, and prelude
+rather than reusing `smithy-core`'s. Those are shaped for serializing values at
+runtime, whereas the generator needs the JSON AST's own vocabulary: wire-format
+type names, member IDs, and lossless shape attributes. The overlap between the
+two is intentional.
+
+## Migration
+
+The Java generator remains authoritative while the Python generator is under
+development. Features may be implemented and reviewed incrementally without
+changing the Java path. A generated artifact SHOULD move to the Python generator
+only after the required behavior is supported and tested.
+
+The Python generator does not need to reproduce Java implementation details or
+byte-for-byte output. It MUST preserve the supported Smithy semantics and public
+behavior of generated packages.
+
+## Designs
+
+* [Code Generator CLI](cli.md)

packages/smithy-python/.changes/next-release/smithy-python-feature-codegen-cli.json

@@ -0,0 +1,4 @@
+{
+  "type": "feature",
+  "description": "Added the experimental smithy-python package and CLI scaffold."
+}

packages/smithy-python/.changes/next-release/smithy-python-feature-model-loading.json

@@ -0,0 +1,4 @@
+{
+  "type": "feature",
+  "description": "Added JSON AST model loading, service resolution via an optional `--service` option, and generated-shape selection with name-conflict detection."
+}

packages/smithy-python/CHANGELOG.md

@@ -0,0 +1 @@
+# Changelog

packages/smithy-python/NOTICE

@@ -0,0 +1 @@
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

packages/smithy-python/README.md

@@ -0,0 +1,19 @@
+# smithy-python
+
+> [!WARNING]
+> This package is an experimental scaffold. It does not generate code yet. The
+> Java generator in the repository's `codegen` directory remains authoritative.
+
+`smithy-python` will provide Python-native code generation for Smithy models.
+The initial command-line interface exposes the planned client and types generation
+commands so that their top-level shape can be developed independently from the
+generator implementation.
+
+```console
+smithy-python generate client [OPTIONS]
+smithy-python generate types [OPTIONS]
+```
+
+After validating their invocation options, both generation commands currently exit
+with an error explaining that generation has not been implemented. The package is
+included in workspace builds to validate its packaging and entry points.

packages/smithy-python/pyproject.toml

@@ -0,0 +1,51 @@
+[project]
+name = "smithy-python"
+dynamic = ["version"]
+requires-python = ">=3.12"
+authors = [
+  {name = "Amazon Web Services"},
+]
+description = "A Smithy code generator for Python clients and types."
+readme = "README.md"
+license = {text = "Apache License 2.0"}
+keywords = ["smithy", "codegen", "sdk"]
+classifiers = [
+  "Development Status :: 2 - Pre-Alpha",
+  "Intended Audience :: Developers",
+  "Natural Language :: English",
+  "License :: OSI Approved :: Apache Software License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Programming Language :: Python :: Implementation :: CPython",
+  "Programming Language :: Python :: Free Threading :: 2 - Beta",
+  "Topic :: Software Development :: Code Generators",
+]
+dependencies = []
+
+[project.scripts]
+smithy-python = "smithy_python.cli:main"
+
+[project.urls]
+"Changelog" = "https://github.com/smithy-lang/smithy-python/blob/develop/packages/smithy-python/CHANGELOG.md"
+"Code" = "https://github.com/smithy-lang/smithy-python/tree/develop/packages/smithy-python/"
+"Issue tracker" = "https://github.com/smithy-lang/smithy-python/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+path = "src/smithy_python/__init__.py"
+
+[tool.hatch.build]
+exclude = [
+  "tests",
+]
+
+[tool.ruff]
+src = ["src"]

packages/smithy-python/src/smithy_python/__init__.py

@@ -0,0 +1,5 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""A Smithy code generator for Python clients and types."""
+
+__version__ = "0.0.0"

packages/smithy-python/src/smithy_python/__main__.py

@@ -0,0 +1,7 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+from .cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())