[Playground] Resolve issue with SCIO examples failing on start due to a timeout

[TSultanov]

SCIO examples timeout when it's a first run in the container. The reason is that sbt tool downloads Scala dependencies upon the first run, which happens synchronously and takes a significant amount of time.

Downloading the Scala dependencies during the container build time mitigates this problem as sbt will use already downloaded files instead of fetching them from a remote host.

Fixes issue with excessive memory usage, examples now can be run under 1GB of memory usage.

Also this PR resolves issue with execution environment not being properly removed after the example execution.

resolves#24943

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[olehborysevych]

Run RAT PreCommit

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label build.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damondouglas]

Can you explain again why we need mitmproxy? I read the website briefly and it looks like a networking tool. However, aren't we achieving our networking needs using kubernetes Service and other related kinds? Can our images and the Dockerfiles that describe them serve just to ship the binary?

[TSultanov]

It's being used to filter out which Google Cloud Storage buckets are accessible by the examples.

However, after investigating a little bit it seems that this is not the best place to have it. We discussed this with @MakarkinSAkvelon and think that the best solution would be to have a single proxy node which would act as a reverse proxy for letting external connections to the runners and also as a single point though which runners would be able to only access the allowlisted hosts.

This approach with a single proxy node will simplify origin policy for frontend and we will be able to get rid of the need to generateconfig.g.dart completely.

This issue, however, I think, is beyond the scope of this PR and we can address it separately.

[damondouglas]

This PR at commit d645408223 has the following test failures:

1. go test beam.apache.org/playground/backend/internal/code_processing

=== RUN Test_getRunOrTestCmd/Get_run_cmd
code_processing_test.go:632: getExecuteCmd() = runCommand arg1, want runCommand arg1
--- FAIL: Test_getRunOrTestCmd/Get_run_cmd (0.00s)
=== RUN Test_getRunOrTestCmd/Get_test_cmd
code_processing_test.go:632: getExecuteCmd() = testCommand arg1 , want testCommand arg1 --- FAIL: Test_getRunOrTestCmd/Get_test_cmd (0.00s)

2. go test beam.apache.org/playground/backend/internal/setup_tools/life_cycle

=== RUN TestSetup/Successfully_setup_life_cycle_with_Scio_sdk
life_cycle_setuper_test.go:342: Setup() error = error during create necessary files for the Scio sdk: exit status 1, wantErr false
--- FAIL: TestSetup/Successfully_setup_life_cycle_with_Scio_sdk (0.01s)

[damondouglas]

I looked into this errors, seems that there should be additional measures taken to make tests runnable in Cloud Shell environment.

FYI this was a compute engine instance but nonetheless a clean environment.

I updated Readme to clarify that tests need local Datastore emulator to run and they will not run with real datastore (although this should be easily fixable - but I'm not if this change worth the effort).

Instead of the emulator, have you tried to:

1. Implement a mock google.golang.org/genproto/googleapis/datastore/v1 with only the endpoints needed for the tests? Endpoints not needed could simply return fmt.Errorf("not implemted")
2. In your test you can add https://pkg.go.dev/google.golang.org/api/option#WithEndpoint to the https://pkg.go.dev/google.golang.org/api/option#WithEndpoint after service begins
3. When test(s) complete, shut down the service

If you still want to use the emulator, I recommend the tests that need it to actually activate and shut down the emulator but personally when I test Google api dependent code in the context of Go, I implement the grpc server. This doesn't replace integration tests but serves as a quicker way to detect errors in code.

[TSultanov]

@damondouglas
I agree with your points that unit tests should preferably use mock of the server and integration tests should be able to start and stop their dependencies automatically, but such change certainly goes beyond the scope of this PR. We can estimate efforts for the change to the tests in this task (#25588).

[github-actions]

Reminder, please take a look at this pr: @Abacn

[Abacn]

R: @damondouglas

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[damccorm]

Mostly LGTM other than a few small comments

[damccorm]

I'm waiting on CI checks to pass before merging, probably will check back tomorrow