[Part 2] Calendar to GCS Operator

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!

[rsg17]

related: #8471

This PR adds a Google Calendar to GCS Operator to write Google Calendar Events to GCS.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[rsg17]

I would like some help on how to do system tests. I have created the example dag and system test file; but I am not sure how to test it.

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

[turbaszek]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

[rsg17]

Thank you!

[josh-fell]

Looks pretty good! Just a few suggestions.

[josh-fell]

Is this TODO still needed? The associated doc looks relatively complete.

You can always mark PRs as a draft first if you still have some WIP actions you'd like to do before you'd like a more detailed review.

[rsg17]

Actually I marked it as TODO because I think it needs the doc url.. which I assumed would be created after I merged this PR.

[rsg17]

Removed TODO

[rsg17]

I read through this guide. I think I need make some changes to variables.env file; but I am not sure what these changes should be.

From your example DAG it looks like you need to set up two variables:

• GCP_GCS_BUCKET
• CALENDAR_ID

Apart from that you will need to add a credentials key because you are using it in test:

@pytest.mark.credential_file(GCP_GCS_KEY)

The file should be named gcp_gcs.json and put according to the guide you linked:

If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze.

Regarding this question:

Also, I am not sure about the --forward-credentials option for running system tests. Which credentials will it forward? Do I need to put the credentials somewhere?

You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm).

Thank you for all your help @turbaszek.
I am running into an authentication error through breeze. I don't see this error when I try from my laptop (not through breeze). I have added the key-file, added corresponding variables to variables.env and I am forwarding credentials to breeze.

Could you help take a look?

This gives the error location:
`___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________
/usr/local/lib/python3.7/contextlib.py:73: in inner
with self._recreate_cm():
/usr/local/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context
f"--key-file={key_file_path}",
tests/test_utils/logging_command_executor.py:92: in execute_cmd
env=env,
/usr/local/lib/python3.7/subprocess.py:800: in init
restore_signals, start_new_session)

---

self = <subprocess.Popen object at 0x4065c8b250>
args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False
`

Here is the error:
E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'

I tried running gcloud auth activate-service-account with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5

[rsg17]

@turbaszek
Was able to resolve the issues with system tests and could verify that the events I created on a test calendar were available as JSON file on GCS.

[eladkal]

LGTM
@rsg17 can you please rebase

If no further comments i'll merge after

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[rsg17]

LGTM @rsg17 can you please rebase

If no further comments i'll merge after

Rebased. Thank you!