Skip to content

[Python] Add codegen - #2507

Merged
kanterov merged 11 commits into
databricks:mainfrom
kanterov:kanterov/python-codegen
Apr 15, 2025
Merged

[Python] Add codegen#2507
kanterov merged 11 commits into
databricks:mainfrom
kanterov:kanterov/python-codegen

Conversation

@kanterov

@kanterovkanterov commented Mar 17, 2025•
edited
Loading

Copy link
Copy Markdown
Collaborator

Changes

Add the code generator from the bundle schema to the Python code. Currently, it's only enabled for the "jobs" package.

There is a new GitHub action to verify that codegen is always up-to-date whenever we update bundle schema.

Depends on #2508

Tests

  • A few unit tests
  • Primarily tested by diffing output

@kanterov
kanterovforce-pushed the kanterov/python-codegen branch 3 times, most recently from 20b87ec to 5891b08CompareMarch 17, 2025 15:26
@kanterov
kanterov marked this pull request as ready for review April 3, 2025 09:34
pietern
pietern reviewed Apr 7, 2025
View reviewed changes
Comment threadexperimental/python/codegen/codegen/openapi.py

@pieternpieternApr 7, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file is called openapi.py, but it looks like this exclusively depends on the JSON schema.

Is that correct?

@kanterovkanterovApr 7, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially, it was implementing OpenAPI until we introduced JSON schema into CLI, let me rename it.

Comment threadexperimental/python/codegen/.python-version Outdated
Comment threadexperimental/python/codegen/codegen/generated_dataclass_patch.py Outdated
Comment threadexperimental/python/codegen/codegen/openapi_patch.py
Comment threadexperimental/python/codegen/pyproject.toml Outdated
@pieternpietern mentioned this pull request Apr 9, 2025
Merged
@kanterov
kanterovforce-pushed the kanterov/python-codegen branch from 5891b08 to 977df62CompareApril 9, 2025 14:55
@kanterov
kanterov requested a review from pieternApril 10, 2025 11:33
@kanterov

kanterov commented Apr 10, 2025

Copy link
Copy Markdown
CollaboratorAuthor

@pietern addressed your comments, please take a look again

pietern
pietern approved these changes Apr 10, 2025
View reviewed changes

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please TAL at the comments.

Comment threadexperimental/python/codegen/codegen/generated_dataclass.py Outdated
Comment threadexperimental/python/codegen/codegen/generated_dataclass.py Outdated
Comment threadexperimental/python/codegen/codegen/packages.py
#
# We can't load all types because of errors while loading some of them.
LOADED_NAMESPACES = [
"compute",

@pieternpieternApr 10, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compute namespace is for job clusters, right?

@kanterovkanterovApr 10, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's for job clusters and pipeline cluster configuration

Comment thread.github/workflows/push.yml

# TODO block PR if this fails once diffs are fixed
# exit 1
fi

@pieternpieternApr 10, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the tests run on push?

@kanterovkanterovApr 10, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@kanterovkanterovApr 10, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a little bit of a problem, if codegen itself crashed CI job will pass now, let me fix it

@github-actions

github-actionsBot commented Apr 10, 2025

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 2507
  • Commit SHA: 1a6449e2ff3b79911554a1cb6479e3d97e61c7c5

Checks will be approved automatically on success.

@pietern

pietern commented Apr 15, 2025

Copy link
Copy Markdown
Contributor

@kanterov Is this ready?

@kanterov
kanterov enabled auto-merge April 15, 2025 09:54
@kanterov
kanterov added this pull request to the merge queueApr 15, 2025
Merged via the queue into databricks:main with commit 77ed564Apr 15, 2025
@kanterov
kanterov deleted the kanterov/python-codegen branch April 15, 2025 19:17
denik pushed a commit that referenced this pull request May 20, 2026
@kanterov
[Python] Add codegen (#2507)
4ccf0d3
## Changes
Add the code generator from the bundle schema to the Python code.
Currently, it's only enabled for the "jobs" package.
There is a new GitHub action to verify that codegen is always up-to-date
whenever we update bundle schema.
Depends on #2508
## Tests
- A few unit tests
- Primarily tested by diffing output
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@andrewnesterandrewnesterAwaiting requested review from andrewnester
@shreyas-goenkashreyas-goenkaAwaiting requested review from shreyas-goenka
@denikdenikAwaiting requested review from denik
@anton-107anton-107Awaiting requested review from anton-107

Assignees

No one assigned

Labels

python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kanterov@pietern