[BEAM-12114] Eliminate beam_fn_api from KafkaIO expansion

[boyuanzz]

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go		---	---		---		---
Java
Python		---			---		---
XLang		---			---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[kennknowles]

This is a useful step for moving this experiment out of the core SDK and into runners. Nice!

[kennknowles]

Mark @Internal just to be clear that it is not for pipeline authors. Would be good to document why this exists and when to use it.

Perhaps it could be in runners-core-construction, but I actually want to merge that back into the core SDK so no need.

[boyuanzz]

runners-core-construction cannot depend on kafka-io because expansion-service introduces circular dependency, just like pubsub

[kennknowles]

.

[boyuanzz]

Good catch!

[kennknowles]

run java precommit

[boyuanzz]

Run Java KafkaIO Performance Test

[boyuanzz]

Run Java KafkaIO Performance Test

[boyuanzz]

Run Java PreCommit

[boyuanzz]

Run Java PreCommit

[boyuanzz]

Run Java PreCommit

[boyuanzz]

I got build errors from :sdks:java:testing:tpcds module as:

14:50:07 * What went wrong:
14:50:07 Execution failed for task ':sdks:java:testing:tpcds:compileJava'.
14:50:07 > Could not resolve all files for configuration ':sdks:java:testing:tpcds:compileClasspath'.
14:50:07 > Could not find io.confluent:kafka-avro-serializer:5.3.2.
14:50:07 Searched in the following locations:
14:50:07 - https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
14:50:07 If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
14:50:07 Required by:
14:50:07 project :sdks:java:testing:tpcds > project :runners:google-cloud-dataflow-java > project :sdks:java:io:kafka
14:50:07 > Could not find io.confluent:kafka-schema-registry-client:5.3.2.
14:50:07 Searched in the following locations:
14:50:07 - https://repo.maven.apache.org/maven2/io/confluent/kafka-schema-registry-client/5.3.2/kafka-schema-registry-client-5.3.2.pom
14:50:07 If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
14:50:07 Required by:
14:50:07 project :sdks:java:testing:tpcds > project :runners:google-cloud-dataflow-java > project :sdks:java:io:kafka

It seems like if I change :sdks:java:testing:tpcds to apply applyJavaNature, this error will be gone but more complaint from the sdks:java:testing:tpcds code.

[boyuanzz]

Commented on the PR #12436 (comment) to see whether it's intended to not apply applyJavaNature

[kennknowles]

When I look at https://github.com/apache/beam/blob/master/sdks/java/testing/tpcds/build.gradle I do not really understand the root cause. I think using applyJavaNature would be an improvement, so it would gain dependency analysis (and others).

[boyuanzz]

Fix the build makes this PR extremely huge: d05cd78. Especially applying spotless introduced a lot of formatting changes.

[kennknowles]

I am OK doing more than one commit in a PR, if each commit makes sense. Doing the spotless formatting in its own commit makes sense. Why is it so huge? Is it not applied already?

[kennknowles]

Oh, I see. Yes, I think it is probably worth separating the large-scale change of adding applyJavaNature to a separate PR, just to keep things simpler for people reading the history.

[boyuanzz]

Is it not applied already?

No. Spotlesscheck was not applied because this module didn't apply applyJavaNature. The same as the checker framework and errorprone

[kennknowles]

Yea, I see the problem. If we change to a more standard style of Gradle configuration we may avoid this pain in the future. I am OK with multiple commits but maybe a separate PR would be most clear.

[boyuanzz]

Here we go: #14516

[boyuanzz]

Run Java PreCommit

[boyuanzz]

Run Java PreCommit

[boyuanzz]

I'm going to merge this PR and I'll monitor the post commit for next 2 days.