Add flink-conf-dir to flink on portable runner, fix some flink test suites

[damccorm]

In #24228 we applied a general fix for some classes of flink failures - this cleans up some previous attempts to work around the issues and fixes them. It also allows users to pass in a flink-conf-dir arg to the portable runner.

Fixes#21094

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #24254 (e789c16) into master (60caac6) will decrease coverage by 0.08%.
The diff coverage is 0.00%.

@@ Coverage Diff @@## master #24254 +/- ##
==========================================
- Coverage 73.46% 73.37% -0.09% 
==========================================
Files 714 717 +3 Lines 96497 96855 +358 ==========================================
+ Hits 70889 71066 +177 - Misses 24286 24467 +181 
Partials 1322 1322 

Flag	Coverage Δ
python	82.99% <0.00%> (-0.18%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...dks/python/apache_beam/options/pipeline_options.py	93.97% <ø> (-0.94%)	⬇️
...on/apache_beam/runners/portability/flink_runner.py	55.35% <0.00%> (-4.26%)	⬇️
sdks/python/apache_beam/utils/interactive_utils.py	95.12% <0.00%> (-2.44%)	⬇️
...eam/runners/portability/fn_api_runner/execution.py	92.49% <0.00%> (-0.64%)	⬇️
...ache_beam/runners/portability/local_job_service.py	82.00% <0.00%> (-0.50%)	⬇️
...on/apache_beam/runners/dataflow/dataflow_runner.py	82.69% <0.00%> (-0.21%)	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	89.08% <0.00%> (-0.17%)	⬇️
sdks/python/apache_beam/pipeline.py	92.12% <0.00%> (-0.15%)	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	93.42% <0.00%> (-0.13%)	⬇️
... and 11 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[damccorm]

Run Go Flink ValidatesRunner

[damccorm]

Run Python Examples_Flink

[damccorm]

Run Python Examples_Flink

[damccorm]

Run Python Examples_Flink

[damccorm]

Run Python Examples_Flink

[damccorm]

This change fixes the Insufficient number of network buffers but causes Jenkins workers to OOM. I'm going to close and just go with a sickbay approach for now.

[Abacn]

More Examples_Python_Flink tests failing with Insufficient number of network buffers after I fixed a test in #24159 where two more pipelines ran. Looks like this test suite can run a finite number of pipeline. If one adds more tests, then this error occurs on some other irrelevant test. Note that these tests are all run on flink minicluster (a built-in flink implementation when flink-master not set). Not sure if this is a limitation of minicluster. If so we could run it on Dataproc as we're doing for load tests.

[damccorm]

More Examples_Python_Flink tests failing with Insufficient number of network buffers after I fixed a test in #24159 where two more pipelines ran. Looks like this test suite can run a finite number of pipeline. If one adds more tests, then this error occurs on some other irrelevant test. Note that these tests are all run on flink minicluster (a built-in flink implementation when flink-master not set). Not sure if this is a limitation of minicluster. If so we could run it on Dataproc as we're doing for load tests.

I think that would probably work, I won't have bandwidth to take it forward though. In the meantime, I added #24342 to at least sickbay some test and get us green