feat: add support for service account impersonation

[ashish10alex]

Fixes: #2000

Solution:

Update @google-cloud/bigquery to a newer version which supports service account impersonation. The version number (~7.1.1) was set from finding the maximum version of @google-cloud/bigquery such that current minimum Node JS version (16.6.0) does not need to be changed.

Tests

1. Able to create .df-credentials.json after authenticating to service account to impersonate by running the following. Earlier to would throw an error: The incoming JSON object does not contain a client_email field

gcloud auth application-default login --impersonate-service-account=<service-account-here>

2. Able to do dataform run with the impersonated service account

3. bazel test //core/... & ./scripts/lint passes

[kolina]

I think it shouldn't be pinned to a specific version similar to other dependencies

[ashish10alex]

Thanks @kolina for the review. Using ^7.1.1 results in the following error when doing bazel run @nodejs//:yarn add @google-cloud/bigquery@^7.1.1

error @google-cloud/promisify@4.1.0: The engine "node" is incompatible with this module. Expected version ">=18". Got "16.16.0"

However, we can use ~ to allow patch updates only (minor and major fixed). This installs without any issues. I have made this change in the lates commit.

Happy to change based on feedback. Trying a lower major version is another option.

[kolina]

@Ceridan, can you please take a look here?

[Ceridan]

Unfortunately, we need to put it on hold for a while. We cannot easily upgrade to @google-cloud/bigquery@7.1.1 because it is not available on our npm mirror at the moment.

Additionally, there is a an ongoing work to update the whole toolchain for Dataform which will make it possible to upgrade NodeJS version and other dependencies to up to date versions.

[wintermi]

Any further updates on when this will be merged?

[Ceridan]

Unfortunately, we cannot merge this right now. It is required some additional work on build toolchain upgrade, which allows us to upgrade all the dependencies. This work will most likely be done in Q4'2025.

[kolina]

@ashish10alex, can you please sync this PR to the latest main? After this I'll re-run tests and if they're successful, I'll merge this PR.

[ashish10alex]

@ashish10alex, can you please sync this PR to the latest main? After this I'll re-run tests and if they're successful, I'll merge this PR.

@kolina done

[kolina]

/gcbrun

[mganeshphani]

@kolina thankyou for getting this in.. when can we expect a new release with this fix?

[kolina]

It's included into this release

[mganeshphani]

Thankyou @kolina .. we will try it.. appreciate getting this one out asap