feat(cli): node selection for dataform compile

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)

[andrebmallmann]

Summary

Adds node selection to dataform compile, mirroring dbt compile --select. compile now accepts the same selection flags as run/build — --actions, --tags, --include-deps, --include-dependents — and filters the printed graph to just the selected action(s):

dataform compile --actions my_model --json
dataform compile --tags daily --include-deps --json

Resolves#2203.

How it works

This is output filtering, not partial compilation. The whole project must
still compile (ref() resolution and the dependency graph require every action
registered), so we compile fully and then prune the result before printing —
reusing the existing prune() that run/build already use, plus the shared
yargs option definitions. No proto changes.

• A clean graph is pruned only when a selector (--actions/--tags) is present.
• On compile errors, the full graph + errors print unchanged (graph-level errors kept as-is).
• A selector matching nothing emits an empty graph and exits 0, matching prune's run-side behavior.
• --include-deps / --include-dependents keep their existing validation (only valid alongside --actions/--tags).

Tests

New compile node selection suite in cli/index_compile_test.ts over an
upstream -> midstream -> downstream project, covering: full graph (no selector),
--actions, --include-deps, --include-dependents, --tags, empty-match
exits 0, and the no-selector validation error. //cli:index_compile_test passes.

Credit

Implementation by @ihistand (Ivan Histand), proposed in SQLAnvil/sqlanvil#27 in
response to this issue. Thank you!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kolina]

/gcbrun

[kolina]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

[andrebmallmann]

I think it makes sense to add some prefix to these options for compile to make it more clear that these options are different from not compiling them. I'd suggest output-{...} format, feel free to suggest something else if you want.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

[kolina]

Sorry for the late reply from my side.

Thanks for the review! Just to check I understood correctly: the concern is that for compile these flags work as filters rather than changing what actually gets compiled, so the unprefixed names could lead to confusion, is that it?

Yes, it may lead to confusion. Also if we decide to implement actual filtering of compiled nodes, it may be hard to pick filters for them in the future.

This version of the solution leans towards keeping parity with commands like run and build, and from a DX perspective I personally like that more. Do you think a clear distinction between filtering vs. compiling outweighs that consistency?

In my opinion, yes.

If we go the prefix way, which do you think is better --output-* or --filter-*? "Filter" reads a bit closer to the actual behavior for me, but I'm happy with either.

output- may be more clear to me, but I don't have a strong opinion here.

And with a prefix, would --include-deps / --include-dependents become --output-include-deps / --output-include-dependents?

Yeah, I think so.

[andrebmallmann]

Hello @kolina

Sorry for late reply on my side, rough weeks.

I've pushed the rename we agreed on the compile selection flags now carry the output- prefix to make it explicit that they filter the printed output rather than change what gets compiled.

run/build keep the unprefixed flags, so there's no change there. Tests updated accordingly, and I verified it end-to-end against examples/stackoverflow_reporter

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

1. Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
2. Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.

I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Compiled example using my version

$CLI compile examples/stackoverflow_reporter --output-actions user_stats --json

{
"tables": [
{
"type": "table",
"target": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"query": "\n\nselect\n stg_users.user_id,\n stg_users.age,\n stg_users.creation_date,\n stg_users.user_tenure,\n count(distinct stg_badges.badge_id) as badge_count,\n count(distinct posts_all.post_id) as questions_and_answer_count,\n count(distinct if(type=\"question\", posts_all.post_id, null)) as question_count,\n count(distinct if(type=\"answer\", posts_all.post_id, null)) as answer_count,\n max(stg_badges.award_timestamp) as last_badge_received_at,\n max(posts_all.created_at) as last_posted_at,\n max(if(type=\"question\", posts_all.created_at, null)) as last_question_posted_at,\n max(if(type=\"answer\", posts_all.created_at, null)) as last_answer_posted_at\nfrom\n `dataform-open-source.staging.stg_users` as stg_users\n left join `dataform-open-source.staging.stg_badges` as stg_badges\n on stg_users.user_id = stg_badges.user_id\n left join `dataform-open-source.reporting.posts_combined` as posts_all\n on stg_users.user_id = posts_all.owner_user_id\ngroup by\n 1,2,3,4",
"disabled": false,
"fileName": "definitions/reporting/user_stats.sqlx",
"tags": [
"daily"
],
"actionDescriptor": {
"description": "Create a summary table for all users including statistics on questions, answeers and badges received."
},
"dependencyTargets": [
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
}
],
"hermeticity": "NON_HERMETIC",
"canonicalTarget": {
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
"enumType": "TABLE"
}
],
"operations": [],
"assertions": [],
"declarations": [
{
"target": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw badges table"
},
"fileName": "definitions/sources/badges.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_answers table"
},
"fileName": "definitions/sources/posts_answers.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw posts_questions table"
},
"fileName": "definitions/sources/posts_questions.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
}
},
{
"target": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
"actionDescriptor": {
"description": "raw users table"
},
"fileName": "definitions/sources/users.sqlx",
"canonicalTarget": {
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
}
}
],
"tests": [],
"notebooks": [],
"dataPreparations": [],
"targets": [
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_posts_combined_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "posts_combined",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_uniqueKey_0",
"database": "dataform-open-source"
},
{
"schema": "dataform_stackoverflow_reporter_example_assertions",
"name": "reporting_user_stats_assertions_rowConditions",
"database": "dataform-open-source"
},
{
"schema": "reporting",
"name": "user_stats",
"database": "dataform-open-source"
},
{
"schema": "stackoverflow",
"name": "badges",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_answers",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "posts_questions",
"database": "bigquery-public-data"
},
{
"schema": "stackoverflow",
"name": "users",
"database": "bigquery-public-data"
},
{
"schema": "staging",
"name": "stg_badges",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_answers",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_posts_questions",
"database": "dataform-open-source"
},
{
"schema": "staging",
"name": "stg_users",
"database": "dataform-open-source"
}
],
"projectConfig": {
"warehouse": "bigquery",
"defaultSchema": "dataform_stackoverflow_reporter_example",
"assertionSchema": "dataform_stackoverflow_reporter_example_assertions",
"defaultDatabase": "dataform-open-source",
"defaultLocation": "us"
},
"graphErrors": {},
"dataformCoreVersion": "3.0.59",
"jitData": {}
}

[kolina]

/gcbrun

[kolina]

One thing I ran into while testing that I'd like your take on. Since this reuses prune(), the filtering only applies to tables/operations/assertions — prune() leaves declarations and the flat targets manifest untouched. So compile --output-actions user_stats --json correctly returns just the user_stats table, but the output still lists all declarations and all targets from the full graph:

Here we might have two options:

Two options:

Leave as-is to keep consistent with run/build, less code; the actual SQL is correctly isolated in tables.
Trim declarations/targets in the compile output path only (not touching shared prune()) for a tighter single-model output.
I think for this PR we could close with (1) and if the option 2 is good for you i can start work in another PR.

Yeah, option 1 is fine with me!

[andrebmallmann]

@kolina

One thing i hit during the review of coerce and i want to make it clear.

The JSON I posted earlier is misleading. With --output-actions user_stats the tables list is
correctly just the one table, but targets still listed all 13 actions. prune() spreads the rest
of the graph through untouched, so targets was never filtered. Harmless for run and build since
nothing reads it, but compile prints it. Fixed in 52c2021, tables and targets now agree.

[kolina]

/gcbrun

[kolina]

/gcbrun(52c2021)