Skip to content

[Python] Add generated code for pipelines - #2639

Merged
kanterov merged 5 commits into
mainfrom
python-pipelines
Apr 9, 2025
Merged

[Python] Add generated code for pipelines#2639
kanterov merged 5 commits into
mainfrom
python-pipelines

Conversation

@kanterov

@kanterovkanterov commented Apr 3, 2025

Copy link
Copy Markdown
Collaborator

Changes

Add generated code for pipelines support introduced in #2574

@kanterovkanterov changed the title [Python] Add pipelines[Python] Add generated code for pipelinesApr 7, 2025
@kanterovkanterov mentioned this pull request Apr 8, 2025
@kanterov
kanterov marked this pull request as ready for review April 8, 2025 14:12

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The target field is not included here, yet it is marked not deprecated in #2646.

How does this work?

@kanterov

Copy link
Copy Markdown
CollaboratorAuthor

@pietern Good catch, the code implementing merging of JSON schema didn't work correctly after rebase. I've fixed it now

All necessary changes are in https://github.com/databricks/cli/tree/codegen-all, but it needs to be split into multiple PRs. It would be great if we could merge the first PR with codegen first and iterate on top of it.

@kanterov
kanterov requested a review from pieternApril 8, 2025 18:19
@pietern

Copy link
Copy Markdown
Contributor

Thanks for clarifying. The first codegen PR has some small comments that should be easy to address + merge. Are you saying the tree you link has a bunch of changes stacked on top of what's in #2507?

@kanterov
kanterov enabled auto-merge April 9, 2025 09:10
@kanterov

kanterov commented Apr 9, 2025

Copy link
Copy Markdown
CollaboratorAuthor

@pietern there are a few changes needed:

  • enum changes that we are merging
  • adding deprecated annotations into the spec (there is draft PR)
  • adding "experimental" annotations into spec so it can be excluded from docs
  • handling newly added annotations, and a few tricks, e.g. excluding dataclass from docs if it's only used as experimental field

There are warnings because the same classes are included into jobs and pipeline packages.
We will fix it when we change code generation to vendor compute package into each resource
using it.
The output for docs already looks correct
@kanterov
kanterov added this pull request to the merge queueApr 9, 2025
Merged via the queue into main with commit 6083062Apr 9, 2025
@kanterov
kanterov deleted the python-pipelines branch April 9, 2025 13:33
denik pushed a commit that referenced this pull request May 20, 2026
## Changes
Add generated code for pipelines support introduced in
#2574
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@kanterov@pietern