Add Impersonate Service Account argument by wintermi · Pull Request #2015 · dataform-co/dataform

wintermi · 2025-09-11T06:43:03Z

This PR adds an --impersonate-service-account argument to the run and test commands, along with the required changes to allow for the impersonation of service accounts without the need to change ADC or call gcloud

This would resolve issue #2000 and would be an alternative to solution than PR #2001

Impersonation could then be achieved by executing:

dataform run --impersonate-service-account=<sSERVICE_ACCT_EMAIL>

google-cla · 2025-09-11T06:43:08Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

camilleAmaury · 2025-10-30T10:44:42Z

+1, this would enable to use impersonation in CI rather than giving the rights directly to the CI service account.
There is no way to workaround that currently.

kolina · 2025-11-11T11:28:51Z

/gcbrun

kolina · 2025-11-13T23:33:49Z

Sorry for the late review. A couple of things:

Integration tests are failing, can you take a look at fixing them? Now we have a guide of running them locally
Let's resolve conflicts

kolina · 2025-11-13T23:17:49Z

+ clientConfig.authClient = new Impersonated({
+ sourceClient: authClient,
+ targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
+ targetScopes: ['https://www.googleapis.com/auth/cloud-platform']


EXTRA_GOOGLE_SCOPES?

Not sure what you would like done here?

I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

kolina · 2026-01-02T21:13:06Z

@wintermi, in the current version tests are failing due to linter checks: output

You can check errors using this lint script.

wintermi · 2026-01-05T13:31:51Z

Resynced the PR with the latest commit

wintermi · 2026-01-05T14:36:31Z

@wintermi, in the current version tests are failing due to linter checks: output
You can check errors using this lint script.

Fixed the linter issues

wintermi · 2026-01-05T14:37:06Z

@kolina ready for retesting, thanks

kolina · 2026-01-07T12:34:01Z

/gcbrun

kolina · 2026-01-07T12:45:04Z

+ clientConfig.authClient = new Impersonated({
+ sourceClient: authClient,
+ targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
+ targetScopes: ['https://www.googleapis.com/auth/cloud-platform']


I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

wintermi · 2026-04-10T00:59:44Z

I removed the parse-duration dependency to avoid CVE-2025-25283
Moving to the parse-duration@2.1.3 seemed a far bigger change than necessary.

wintermi · 2026-04-10T01:09:29Z

I added a IMPERSONATION_GOOGLE_SCOPES as the EXTRA_GOOGLE_SCOPES only mention drive.

kolina · 2026-04-14T20:57:28Z

 getCredentialsPath(argv[projectDirOption.name], argv[credentialsOption.name])
 );
+ if (argv[impersonateServiceAccountOption.name]) {
+ (readCredentials as any).impersonateServiceAccount =


Can we extend dataform.IBigQuery with your new option to avoid dynamic casts breaking static typing?

kolina · 2026-04-14T21:04:15Z

 return `${value} ${units[i]}`;
 }

+const DURATION_UNITS_IN_MILLIS: { [unit: string]: number } = {


Can you please elaborate about the effort to avoid it and upgrade parse-duration dependency?

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Replace regex-based parseCliDuration with manual character parsing. Add build.sh to package CLI/core artifacts for hephaestus-worker-base. Fix malformed mkdirp integrity entry in yarn.lock.

kolina · 2026-06-25T11:46:45Z

/gcbrun

kolina

One of the tests started to time out for some reason

Step #1: //cli/api:dbadapters/bigquery_test FAILED in 61.1s
Step #1: //testing:index_test PASSED in 0.9s
Step #1: //sqlx:lexer_test PASSED in 0.5s
Step #1: //sqlx:format_test PASSED in 0.8s
Step #1: //examples:examples_test PASSED in 3.2s
Step #1: //core:utils_test PASSED in 0.6s
Step #1: //core:main_test PASSED in 36.5s
Step #1: //core:jit_context_test PASSED in 0.7s
Step #1: //core:jit_compiler_test PASSED in 0.7s
Step #1: //core:compilers_test PASSED in 0.6s
Step #1: //core:actions/view_test PASSED in 8.9s
Step #1: //core:actions/test_test PASSED in 4.6s
Step #1: //core:actions/table_test PASSED in 24.8s
Step #1: //core:actions/operation_test PASSED in 7.9s
Step #1: //core:actions/notebook_test PASSED in 5.0s
Step #1: //core:actions/index_test PASSED in 5.4s
Step #1: //core:actions/incremental_table_test PASSED in 29.1s
Step #1: //core:actions/filename_override_test PASSED in 7.5s
Step #1: //core:actions/declaration_test PASSED in 9.6s
Step #1: //core:actions/data_preparation_test PASSED in 6.5s
Step #1: //core:actions/assertion_test PASSED in 32.5s
Step #1: //common/strings:stringifier.spec PASSED in 1.1s
Step #1: //common/protos:index_test PASSED in 0.7s
Step #1: //common/protos:build_test PASSED in 0.2s
Step #1: //common/promises:index.spec PASSED in 2.9s
Step #1: //common/flags:build_test PASSED in 0.3s
Step #1: //common/errors:errors.spec PASSED in 0.9s
Step #1: //cli/api:utils_test PASSED in 0.8s
Step #1: //cli/api:execution_sql_test PASSED in 0.7s
Step #1: //cli/api:commands/jit/rpc_test PASSED in 0.7s
Step #1: //cli:util_test PASSED in 0.8s
Step #1: //cli:index_run_e2e_test PASSED in 62.8s
Step #1: //cli:index_project_test PASSED in 6.2s
Step #1: //cli:index_init_test PASSED in 4.3s
Step #1: //cli:index_help_test PASSED in 5.2s
Step #1: //cli:index_compile_test PASSED in 11.5s
Step #1: INFO: Step #1: INFO: Build completed, 1 test FAILED, 524 total actions
Step #1: INFO: 524 processes: 368 internal, 103 linux-sandbox, 1 local, 52 worker.
Step #1: INFO: Elapsed time: 123.997s, Critical Path: 99.81s
Step #1: INFO: Found 36 test targets...
Step #1: [523 / 524] 35 / 36 tests, 1 failed; Testing //cli:index_run_e2e_test; 63s linux-sandbox
Step #1: [523 / 524] 35 / 36 tests, 1 failed; Testing //cli:index_run_e2e_test; 61s linux-sandbox
Step #1: [523 / 524] 35 / 36 tests, 1 failed; Testing //cli:index_run_e2e_test; 51s linux-sandbox
Step #1: ================================================================================
Step #1: Tests failed.
Step #1: Step #1: BigQueryDbAdapter > setMetadata correctly maps column descriptions PASSED
Step #1: BigQueryDbAdapter > setMetadata handles action without columns PASSED
Step #1: Step #1: at processTimers (node:internal/timers:541:7)
Step #1: at listOnTimeout (node:internal/timers:605:17)
Step #1: at Timeout.<anonymous> (testing/test.ts:85:22)
Step #1: Error: Timed out (30000ms).
Step #1: Step #1: TIMEOUT
Step #1: BigQueryDbAdapter > tables() without schema lists all datasets and tables
Step #1: Step #1: at processTimers (node:internal/timers:541:7)
Step #1: at listOnTimeout (node:internal/timers:605:17)
Step #1: at Timeout.<anonymous> (testing/test.ts:85:22)
Step #1: Error: Timed out (30000ms).
Step #1: Step #1: BigQueryDbAdapter > tables() with schema filters correctly TIMEOUT

kolina · 2026-06-25T11:58:26Z

 projectId = projectId || credentials.projectId;
 if (!clients.has(projectId)) {
-clients.set(
+const clientConfig: any = {


BigQueryOptions instead of any?

kolina · 2026-06-25T17:01:50Z

+const clientConfig: any = {
 projectId,
- new BigQuery({
+ scopes: EXTRA_GOOGLE_SCOPES,


Are projectId and scopes here used by the auth library if you ovewrite authClient? If they're not used in this case, I'd only set them in else branch below

kolina · 2026-06-25T17:23:32Z

@@ -0,0 +1,12 @@
+#!/bin/bash


Non-intended diff?

wintermi requested a review from a team as a code owner September 11, 2025 06:43

wintermi requested review from Ceridan and removed request for a teamSeptember 11, 2025 06:43

Ceridan requested review from kolina and removed request for Ceridan November 13, 2025 09:15

kolina requested changes Nov 13, 2025
View reviewed changes

ashish10alex mentioned this pull request Dec 23, 2025
[Dataform CLI] Error when using service account impersonation through ADC #2056
Closed

wintermi requested a review from kolina January 7, 2026 05:35

kolina reviewed Jan 7, 2026
View reviewed changes

wintermi force-pushed the main branch from 763681d to 21fb05fCompare April 10, 2026 00:42

wintermi requested a review from kolina April 10, 2026 01:09

kolina requested changes Apr 14, 2026
View reviewed changes

wintermi added 7 commits June 17, 2026 15:23

Add the '--impersonate-service-account' argument to the 'run' and 'te…
cf3881c
…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Make changes as requested
8c2e093

Make changes as requested
35a0e62

Make changes as requested
a96b601

Add the '--impersonate-service-account' argument to the 'run' and 'te…
2086be4
…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Make changes as requested
3b6cd51

Make changes as requested
26b6528

wintermi added 2 commits June 17, 2026 15:27

fix(cli): remove parse-duration dependency
868c5a3

refactor(cli): dedupe bigquery impersonation scopes
25c4480

wintermi force-pushed the main branch from eb2b141 to 25c4480Compare June 17, 2026 06:56

refactor(cli): rewrite duration parser and add packaging build script
3891fdb
Replace regex-based parseCliDuration with manual character parsing. Add build.sh to package CLI/core artifacts for hephaestus-worker-base. Fix malformed mkdirp integrity entry in yarn.lock.

kolina requested changes Jun 25, 2026
View reviewed changes

Uh oh!

Conversation

wintermi commented Sep 11, 2025

Uh oh!

google-claBot commented Sep 11, 2025

Uh oh!

camilleAmaury commented Oct 30, 2025

Uh oh!

kolina commented Nov 11, 2025

Uh oh!

kolina commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolina commented Jan 2, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

kolina commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wintermi commented Apr 10, 2026

Uh oh!

wintermi commented Apr 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kolina commented Jun 25, 2026

Uh oh!

kolina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants