[BEAM-13812] Integrate DataprocClusterManager into Interactive Environment

[victorplusc]

• Integrates the DataprocClusterManager previously introduced in:
  • [BEAM-13799] Create an Interactive Beam Dataproc package for users to manage clusters with
• Adds code to automatically create a Dataproc cluster configured with Docker and Flink if a user instantiates an InteractiveRunner using underlying_runner='FlinkRunner'
• By default, the created Dataproc cluster will be called 'interactive-beam-cluster' and will be cleaned up when the Interactive Environment is interrupted
• Introduces the Clusters class in interactive_beam.py, which acts as a user interface that provides the user with the following capabilities:
  • Changing the default name of the clusters created by Interactive Beam
  • Cleaning up a cluster by providing a corresponding pipeline, an option to force the deletion is provided, in the case where more than one pipeline use the same cluster
  • Getting all pipelines that use the provided master_url
  • Receiving a description back for either the cluster corresponding to a specific pipeline, or the descriptions of all pipelines
• ASF Jira thread: [Beam-13812]

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---					---
Java
Python	---					---
XLang						---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[victorplusc]

R: @kevingg
R: @rohdesamuel

[codecov]

Codecov Report

Merging #16741 (9d9554d) into master (f87f146) will increase coverage by 8.98%.
The diff coverage is 93.12%.

@@ Coverage Diff @@## master #16741 +/- ##
==========================================
+ Coverage 74.65% 83.63% +8.98% 
==========================================
Files 656 453 -203 Lines 82424 62470 -19954 ==========================================
- Hits 61532 52249 -9283 + Misses 19892 10221 -9671 + Partials 1000 0 -1000 

Impacted Files	Coverage Δ
...s/interactive/dataproc/dataproc_cluster_manager.py	81.42% <78.04%> (ø)
...che_beam/runners/interactive/interactive_runner.py	92.48% <92.00%> (-0.12%)	⬇️
...pache_beam/runners/interactive/interactive_beam.py	81.25% <100.00%> (+3.89%)	⬆️
...eam/runners/interactive/interactive_environment.py	90.74% <100.00%> (+0.34%)	⬆️
...ks/python/apache_beam/runners/interactive/utils.py	95.23% <100.00%> (+0.37%)	⬆️
setup.py	0.00% <0.00%> (ø)
sdks/go/pkg/beam/core/runtime/coderx/int.go
sdks/go/pkg/beam/pcollection.go
sdks/go/pkg/beam/coder.go
sdks/go/pkg/beam/core/runtime/xlangx/namespace.go
... and 201 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f87f146...9d9554d. Read the comment docs.

[nika-qubit]

LGTM

[nika-qubit]

retest this please

[victorplusc]

retest this please

[victorplusc]

Run Python PreCommit

[aaltay]

colab users are reporting error logs showing "No cluster_manager is associated with the provided pipeline!". Could you please file a bug / fix the problem?

[victorplusc]

Hi @aaltay,

Thanks for letting me know. I am working on a fix in this PR: #16936. Here is the link to the Beam ASF thread: BEAM-13996.