Skip to content

Integrate JiT compilation into CLI - #2110

Closed
rafal-hawrylak wants to merge 1 commit into
mainfrom
cli_jit_integration
Closed

Integrate JiT compilation into CLI#2110
rafal-hawrylak wants to merge 1 commit into
mainfrom
cli_jit_integration

Conversation

@rafal-hawrylak

@rafal-hawrylakrafal-hawrylak commented Mar 9, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@rafal-hawrylakrafal-hawrylak self-assigned this Mar 9, 2026
@rafal-hawrylak
rafal-hawrylak marked this pull request as ready for review March 9, 2026 20:28
@rafal-hawrylak
rafal-hawrylak requested a review from a team as a code ownerMarch 9, 2026 20:28
@rafal-hawrylak
rafal-hawrylak enabled auto-merge (squash) March 9, 2026 20:28
kolina
kolina requested changes Mar 11, 2026
View reviewed changes

@kolinakolina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's test compatibility with different Dataform core versions.

Also this change seems worth it to bump a minor version of Dataform core

Comment threadcli/api/commands/compile.ts
Comment threadcli/api/commands/compile.ts
Comment threadcli/api/commands/compile.ts
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/tests/jit/jit_run_test.ts
Comment threadcli/api/commands/compile.ts Outdated
Comment threadpackages/@dataform/cli/worker.ts Outdated
Comment threadprotos/execution.proto
Comment threadtests/api/api.spec.ts Outdated
ikholopov-omni
ikholopov-omni requested changes Mar 11, 2026
View reviewed changes

@ikholopov-omniikholopov-omni left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocked on review of https://github.com/dataform-co/dataform/pull/2109/changes

@rafal-hawrylak
rafal-hawrylakforce-pushed the cli_jit_integration branch 13 times, most recently from 10bb4e6 to 824d7d9CompareApril 3, 2026 14:08
@rafal-hawrylak
rafal-hawrylakforce-pushed the cli_jit_integration branch 3 times, most recently from 2466f28 to 38cd3b9CompareApril 7, 2026 09:24
kolina
kolina reviewed Apr 11, 2026
View reviewed changes
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/api/commands/run.ts Outdated
Comment threadcli/api/dbadapters/bigquery.ts Outdated
Comment threadcli/vm/jit_worker.ts Outdated
Comment threadcli/console.ts Outdated
Comment threadcli/index.ts
Comment threadpackages/@dataform/cli/worker.ts
@apilaskowskiapilaskowski self-assigned this May 8, 2026
@rafal-hawrylak
rafal-hawrylakforce-pushed the cli_jit_integration branch 8 times, most recently from 1555159 to c72977eCompareMay 25, 2026 16:52
@rafal-hawrylak

rafal-hawrylak commented May 25, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Blocked on review of https://github.com/dataform-co/dataform/pull/2109/changes

Unblocked

@rafal-hawrylak

rafal-hawrylak commented May 25, 2026

Copy link
Copy Markdown
CollaboratorAuthor

No outstanding issues from my side except question about access of JiT compiled code to other files

Handled the access of external files to be on pair with GCP approach to provide portability.

kolina
kolina approved these changes May 27, 2026
View reviewed changes
Comment threadprotos/core.proto Outdated
Comment threadprotos/core.proto Outdated
@rafal-hawrylak
feat: integrate JiT compilation and finalize test suite
bf4d057
feat: enable JiT compilation in CLI run, build and compile commands
Connect main CLI execution flow to the JiT worker process
Add comprehensive JiT test suite
Finalize AoT build logic to preserve JiT-specific fields
Update CLI console and entry points for integrated JiT support
Add file_name field (=8) to JitCompilationRequest proto for target source
file resolution in the JiT worker.
ikholopov-omni
ikholopov-omni requested changes May 27, 2026
View reviewed changes

@ikholopov-omniikholopov-omni left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR contains too many changes at once, making it hard to perform an adequate review of changes.

Please split it into smaller PRs, separating the refactoring of the existing infrastructure (like api/commands/compile.ts) and introduction of new functionality.

@rafal-hawrylak

rafal-hawrylak commented May 27, 2026

Copy link
Copy Markdown
CollaboratorAuthor

This PR contains too many changes at once, making it hard to perform an adequate review of changes.

Please split it into smaller PRs, separating the refactoring of the existing infrastructure (like api/commands/compile.ts) and introduction of new functionality.

Ok, I will

@rafal-hawrylak

rafal-hawrylak commented May 27, 2026•
edited
Loading

Copy link
Copy Markdown
CollaboratorAuthor

This was referenced May 28, 2026
Merged
Merged
Merged
Merged
Closed
Merged
auto-merge was automatically disabled August 19, 2026 09:56

Pull request was closed

@rafal-hawrylak

rafal-hawrylak commented Aug 19, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Split into multiple smaller PRs

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ikholopov-omniikholopov-omniikholopov-omni requested changes
@kolinakolinakolina approved these changes
@apilaskowskiapilaskowskiapilaskowski approved these changes
@andrzej-grudzienandrzej-grudzienAwaiting requested review from andrzej-grudzien

Assignees

@apilaskowskiapilaskowski

@rafal-hawrylakrafal-hawrylak

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rafal-hawrylak@kolina@apilaskowski@ikholopov-omni