[BEAM-13008] Create gradle tasks for the Beam Playground by damondouglas · Pull Request #15761 · apache/beam

damondouglas · 2021-10-20T17:24:50Z

This PR creates gradle tasks for the Beam Playground project as well as a bundled playgroundPrecommit task. Future work needs to configure the precommit process to install necessary dependencies to properly run the playgroundPrecommit task. Note that when running ./gradlew playgroundPrecommit in the current state will show multiple issues and errors in the playground code.

I'd like to request the following reviewers or whomever they delegate.
R: @pabloem
R: @tvalentyn

./gradlew playground:tasks

./gradlew playground:backend:tasks

./gradlew playground:frontend:tasks

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-10-20T17:48:36Z

Codecov Report

Merging #15761 (b0ff33f) into master (a05aa45) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #15761 +/- ##
==========================================
- Coverage 83.54% 83.52% -0.02% 
==========================================
Files 445 445 Lines 61185 61185 ==========================================
- Hits 51114 51107 -7 - Misses 10071 10078 +7

Impacted Files	Coverage Δ
sdks/python/apache_beam/utils/interactive_utils.py	`92.68% <0.00%> (-2.44%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (-2.24%)`	⬇️
sdks/python/apache_beam/io/source_test_utils.py	`88.47% <0.00%> (-1.39%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`89.25% <0.00%> (-0.17%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.64% <0.00%> (+0.12%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a05aa45...b0ff33f. Read the comment docs.

pabloem · 2021-10-20T18:43:57Z

have you ran this locally? can you share the output?

damondouglas · 2021-10-20T19:28:23Z

(cc: @pabloem )
The following is the output running:

./gradlew playgroundPrecommit

@ilya-kozyrev would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle. Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the go dependency references.

> Task :playground:backend:tidy FAILED
go: finding module for package beam.apache.org/playground/backend/internal/environment
beam.apache.org/playground/backend/cmd/server imports
beam.apache.org/playground/backend/internal/environment: cannot find module providing package beam.apache.org/playground/backend/internal/environment: unrecognized import path "beam.apache.org/playground/backend/internal/environment": reading https://beam.apache.org/playground/backend/internal/environment?go-get=1: 404 Not Found

@ElessarST would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle.

For the mockito related errors, I realize there is this SOF post about commiting/notcommiting mockito generated code. I personally am in favor of commiting but I'll leave it to your judgement to figure out a way to make the ./gradlew playgroundPrecommit pass either through another gradle task that would generate these dependencies.
Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the dart dependency references.

> Task :playground:frontend:analyze FAILED
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:21:8 - Target of URI doesn't exist: 'package:playground/generated/playground.pbgrpc.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
error - test/modules/editor/repository/code_repository/code_repository_test.dart:31:8 - Target of URI doesn't exist: 'code_repository_test.mocks.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
error - test/modules/editor/repository/code_repository/code_repository_test.dart:54:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
error - test/modules/editor/repository/code_repository/code_repository_test.dart:87:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
error - test/modules/editor/repository/code_repository/code_repository_test.dart:121:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
13 issues found.

damondouglas · 2021-10-20T20:02:01Z

- "playground/frontend/assets/**/*.svg",
- "playground/frontend/assets/**/*.png",
- "playground/frontend/assets/**/*.jpg",
+ "playground/frontend/**/*.svg",


@ElessarST @ilya-kozyrev (cc: @pabloem ) what is the license for these SVGs?

pabloem · 2021-10-20T21:48:57Z

LGTM. This is wonderful.

pabloem · 2021-10-20T21:49:04Z

Run Java PreCommit

ElessarST · 2021-10-21T09:39:09Z

@damondouglas I created PR with frontend changes damondouglas#2

ilya-kozyrev · 2021-10-21T13:30:27Z

(cc: @pabloem ) The following is the output running:

./gradlew playgroundPrecommit

@ilya-kozyrev would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle. Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the go dependency references.

> Task :playground:backend:tidy FAILED
go: finding module for package beam.apache.org/playground/backend/internal/environment
beam.apache.org/playground/backend/cmd/server imports
beam.apache.org/playground/backend/internal/environment: cannot find module providing package beam.apache.org/playground/backend/internal/environment: unrecognized import path "beam.apache.org/playground/backend/internal/environment": reading https://beam.apache.org/playground/backend/internal/environment?go-get=1: 404 Not Found

@ElessarST would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle.

For the mockito related errors, I realize there is this SOF post about commiting/notcommiting mockito generated code. I personally am in favor of commiting but I'll leave it to your judgement to figure out a way to make the ./gradlew playgroundPrecommit pass either through another gradle task that would generate these dependencies.
Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the dart dependency references.

> Task :playground:frontend:analyze FAILED
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:21:8 - Target of URI doesn't exist: 'package:playground/generated/playground.pbgrpc.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
error - test/modules/editor/repository/code_repository/code_repository_test.dart:31:8 - Target of URI doesn't exist: 'code_repository_test.mocks.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
error - test/modules/editor/repository/code_repository/code_repository_test.dart:54:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
error - test/modules/editor/repository/code_repository/code_repository_test.dart:87:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
error - test/modules/editor/repository/code_repository/code_repository_test.dart:121:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
13 issues found.

@damondouglas Regarding this error

> Task :playground:backend:tidy FAILED
go: finding module for package beam.apache.org/playground/backend/internal/environment
beam.apache.org/playground/backend/cmd/server imports
beam.apache.org/playground/backend/internal/environment: cannot find module providing package beam.apache.org/playground/backend/internal/environment: unrecognized import path "beam.apache.org/playground/backend/internal/environment": reading https://beam.apache.org/playground/backend/internal/environment?go-get=1: 404 Not Found

It happened because we are still waiting for #15654 to get merged. Can we merge Environments first of all to resolve it?

damondouglas · 2021-10-21T16:09:10Z

@pabloem

./gradlew playgroundPrecommit

has the following output:

BUILD SUCCESSFUL in 21s
5 actionable tasks: 5 executed

pabloem · 2021-10-21T16:16:55Z

LGTM. Merging.

… for the Beam Playground * Implement Beam Playground gradle tasks * Fix trailing whitespace * Remove format from precommit * Fix order of includes and add descriptions * [BEAM-13008] add generated files and fix analysis issues (apache#2) * Reconcile errors with playgroundPrecommit output Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>

damondouglas added 2 commits October 20, 2021 17:17

Implement Beam Playground gradle tasks
efd1ff4

Fix trailing whitespace
b940ca0

pabloem reviewed Oct 20, 2021
View reviewed changes

Comment threadsettings.gradle.kts Outdated
Comment threadbuild.gradle.kts Outdated

damondouglas added 2 commits October 20, 2021 19:36

Remove format from precommit
f7fc854

Fix order of includes and add descriptions
b8d9cd2

damondouglas commented Oct 20, 2021
View reviewed changes

tvalentyn approved these changes Oct 21, 2021
View reviewed changes

ElessarSTand others added 2 commits October 21, 2021 07:46

[BEAM-13008] add generated files and fix analysis issues (#2)
9e240cf

Reconcile errors with playgroundPrecommit output
b0ff33f

pabloem merged commit dc0548e into apache:masterOct 21, 2021

Navigation Menu

Uh oh!

[BEAM-13008] Create gradle tasks for the Beam Playground - #15761

[BEAM-13008] Create gradle tasks for the Beam Playground#15761
pabloem merged 6 commits into
apache:masterfrom
damondouglas:BEAM-13008-playground-build-gradle

damondouglas commented Oct 20, 2021•
edited
Loading

Uh oh!

codecovBot commented Oct 20, 2021•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

damondouglas commented Oct 20, 2021•
edited
Loading

Uh oh!

damondouglasOct 20, 2021

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

ElessarST commented Oct 21, 2021

Uh oh!

ilya-kozyrev commented Oct 21, 2021

Uh oh!

damondouglas commented Oct 21, 2021

Uh oh!

pabloem commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

damondouglas commented Oct 20, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

codecovBot commented Oct 20, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

damondouglas commented Oct 20, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damondouglasOct 20, 2021

Choose a reason for hiding this comment

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

ElessarST commented Oct 21, 2021

Uh oh!

ilya-kozyrev commented Oct 21, 2021

Uh oh!

damondouglas commented Oct 21, 2021

Uh oh!

pabloem commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

damondouglas commented Oct 20, 2021•
edited
Loading

`ValidatesRunner` compliance status (on master branch)

codecovBot commented Oct 20, 2021•
edited
Loading

damondouglas commented Oct 20, 2021•
edited
Loading