Add localstack Breeze integration

[jason810496]

Why

As Ash suggested setting up AWS CloudWatch to test system behavior using LocalStack in #49470 (comment), we should provide an easy way to run LocalStack with Breeze.

What

Add localstack to the Breeze --integration flag to enable seamless setup of the Docker network for both Breeze and LocalStack.

Then contributor can setup the following env in files/airflow-breeze-config/environment_variables.env for testing system behavior with localstack:

AWS_ENDPOINT_URL=http://localstack:4566
AWS_ACCESS_KEY_ID="test"
AWS_SECRET_ACCESS_KEY="test"
AWS_DEFAULT_REGION="us-east-1"
AIRFLOW_CONN_AWS_DEFAULT='{"conn_type": "aws", "login": "test", "password": "test", "host": "localstack", "port": 4566, "schema": "", "extra": "{\"region_name\": \"us-east-1\", \"aws_access_key_id\": \"test\", \"aws_secret_access_key\": \"test\"}"}'# For testing CloudWatch Remote Logging Handler# AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=cloudwatch://arn:aws:logs:us-east-1:000000000000:log-group:test# AIRFLOW__LOGGING__REMOTE_LOGGING=True

[potiuk]

Very nice! good addition to Breeze!

Few things:

• breeze tests
• we neeed at least one integration test for localstack - and I think it's generally a good thing to have even if it is quite simple one - because it will also test if the integration is still working
• possibly - if the integration is targetted for testing AWS integrations, it would be great if we had some specific documents on how to test things locally with the localstack - I think it could be a good way to add somewhere in contributing guide - maybe new "manual testing with integration" type of testing

cc: @vincbeck@ferruzzi@o-nikolas -> maybe you have some comments as well for that ?

[gopidesupavan]

Thats cool @jason810496 very good start :)

[ashb]

Nice.

One nit here -- with the AWS_ENDPOINT_URL=http://localstack:4566 env var and the other envs set we shouldn't need to set the connection as you have it. I think we should do one or the other (AWS env vars, or Airflow connection), but not both.

[vincbeck]

Nice! Could be a nice solution to test AWS resources indeed. Potentially, in the future, we could also use that to, say, run AWS system tests in Airflow CI

[jason810496]

Thanks all for the review, I will catch up after #54054.
Since I'm preparing presentation for COSCUP and have limited bandwidth, I will need some additional time to get this PR ready.

[potiuk]

Nice! Could be a nice solution to test AWS resources indeed. Potentially, in the future, we could also use that to, say, run AWS system tests in Airflow CI

Crossed my mind too :)

[kyungjunleeme]

I'm really excited to hear that LocalStack is being added to Breeze!

That said, I've seen many setups in the industry that combine Breeze with MinIO. I was wondering — was there a specific reason for choosing LocalStack over MinIO in this case?

Was LocalStack chosen because other AWS services needed to be tested as well, beyond just S3?

[ashb]

@kyungjunleeme Minio is just S3 compatible storage right? Localstack has lots of other AWS services, including CloudWatch Logs etc.

[kyungjunleeme]

@ashb Yes, I totally agree — makes perfect sense :)

[jason810496]

@kyungjunleeme Minio is just S3 compatible storage right?
Localstack has lots of other AWS services, including CloudWatch Logs etc.

Yes, Minio is just S3 compatible storage and Localstack is more than Minio.
In current case I use Localstack to test the CloudWatchTaskHandler behavior.

Potentially, in the future, we could also use that to, say, run AWS system tests in Airflow CI

LGTM for using Localstack as system test for AWS, as long as the Localstack is fully compatible ( as least will not show inconsistent behavior or introduce flakiness in the system test ).

[potiuk]

LGTM for using Localstack as system test for AWS, as long as the Localstack is fully compatible ( as least will not show inconsistent behavior or introduce flakiness in the system test ).

Yep. I think it won't be a "full" replacement - we'd likely have to skip some tests or add some workarounds, but for me that would be a very cool way to make System Tests for AWS run during the PRs. We were discussing ways how we can do it for Google - and apparently there is a https://cloud.google.com/sdk/gcloud/reference/beta/emulators - @VladaZakharova maybe that something that we could also leverage for Google System Tests ?

[jason810496]

Nice.

One nit here -- with the AWS_ENDPOINT_URL=http://localstack:4566 env var and the other envs set we shouldn't need to set the connection as you have it. I think we should do one or the other (AWS env vars, or Airflow connection), but not both.

If this is the case, do you mean we should the AWS_ENDPOINT_URL=http://localstack:4566 be set in scripts/ci/docker-compose/integration-localstack.yml level? Or would it be better to be set in the test level?

[potiuk]

One nit here -- with the AWS_ENDPOINT_URL=http://localstack:4566 env var and the other envs set we shouldn't need to set the connection as you have it. I think we should do one or the other (AWS env vars, or Airflow connection), but not both.

If this is the case, do you mean we should the AWS_ENDPOINT_URL=http://localstack:4566 be set in scripts/ci/docker-compose/integration-localstack.yml level? Or would it be better to be set in the test level?

I think settin AWS_ENDPOINT_URL is a good approach actually - it does not really replace the connection, it just changes routing of the client libraries to use localstack instead the default "amazon" connections, so you might use regular "conneciton" and set the variable to make it talk to localstack.

I think the best way to make it works in breeze is to change it in entrypoint_ci.sh right after "check_environment.sh". We already do a bunch of checks for integrations and do stuff when they are enabled. Something like that:

if [[ ${INTEGRATION_LOCALSTACK}=="true" ]];thenexport AWS_ENDPOINT_URL=http://localstack:4566
fi

Then everything started with --integration localstack will automatically use localstack regardless what connections are defined.

[potiuk]

Nice. Exactly this !

[potiuk]

I think one thing we need here is some docs about in contributing guidelines - not sure where (somewhere where we mention running tests in breeze.

[jason810496]

I think one thing we need here is some docs about in contributing guidelines - not sure where (somewhere where we mention running tests in breeze.

No problem! I will add the doc in tomorrow afternoon after my talk.
I have to get up early tomorrow 😆

[vincbeck]

Really nice! I can see many different use cases where this integration could be super useful :)

[gopidesupavan]

Fantastic @jason810496

[gopidesupavan]

probably i am going to integrate this with #54072 after our both PR merges.
Basically we can test remote logging any other major scenarios we want to test.

[potiuk]

probably i am going to integrate this with #54072 after our both PR merges. Basically we can test remote logging any other major scenarios we want to test.

Fantastic !

[github-actions]

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 84bd22c v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue