[Task]: Java SDK, make BigTableIO compatible with BigTable library 2.71.x

[aprochko]

What needs to happen?

Hello dear team,

It appears there's a breaking change when using Apache Beam 2.70 with Bigtable 2.71.0, specifically impacting BigtableIO. The system fails to start under this configuration.

However, it functions correctly with Bigtable 2.70.1.

The current version of Bigtable integrated into BigtableIO seems to be 2.68.0.

Exception in thread "main" java.lang.IllegalStateException: java.lang.NoSuchMethodError: 'com.google.api.gax.core.FixedExecutorProvider com.google.api.gax.core.FixedExecutorProvider.create(java.util.concurrent.ScheduledExecutorService, boolean)' at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:823) at org.springframework.boot.SpringApplication.run(SpringApplication.java:328) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1361) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1350) at de.bonprix.package.app.main(App.java:13) Caused by: java.lang.NoSuchMethodError: 'com.google.api.gax.core.FixedExecutorProvider com.google.api.gax.core.FixedExecutorProvider.create(java.util.concurrent.ScheduledExecutorService, boolean)' at com.google.cloud.bigtable.data.v2.stub.BigtableClientContext.create(BigtableClientContext.java:88) at com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStub.createBigtableClientContext(EnhancedBigtableStub.java:224) at com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStub.create(EnhancedBigtableStub.java:205) at com.google.cloud.bigtable.data.v2.BigtableDataClient.create(BigtableDataClient.java:179) at org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceFactory.checkTableExists(BigtableServiceFactory.java:204) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$WriteWithResults.validateTableExists(BigtableIO.java:1289) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$WriteWithResults.validate(BigtableIO.java:1265) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Write.validate(BigtableIO.java:1174) at org.apache.beam.sdk.transforms.PTransform.validate(PTransform.java:180) at org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:673) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:581) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:585) at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$500(TransformHierarchy.java:240) at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:214) at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:477) at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:609) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310) at de.bonprix.package.apprunner.run(AppRunner.java:226) at org.springframework.boot.SpringApplication.lambda$callRunner$5(SpringApplication.java:788) at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:82) at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:86) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:796) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787) at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:772) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:772) at org.springframework.boot.SpringApplication.run(SpringApplication.java:325) ... 3 more

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[damccorm]

@igorbernstein2 or @mutianf could you please take a look?

[alisha-1000]

Just following up on this — since it’s been a couple of weeks after the initial ping.

This looks like a hard incompatibility between Beam 2.70’s BigtableIO and Bigtable client 2.71.x (likely via gax), which prevents pipelines from starting. Please let us know if there’s a recommended next step here (e.g., dependency alignment, exclusion guidance, or a fix in progress).

Happy to help test or validate a change if needed.

[mutianf]

@alisha-1000 @aprochko Sorry missed the ping. This is because of dependency misalignment. If you are using beam 2.70.0, the bigtable version should be the same from the beam-sdks-java-io-google-cloud-platform dependencies https://central.sonatype.com/artifact/org.apache.beam/beam-sdks-java-io-google-cloud-platform/2.70.0/dependencies, which is 2.68.0. What's the reason to use bigtable 2.71.0 with an older version of beam?

Looks like libraries bom was just updated #37473, which means beam will have the newer version of bigtable (2.72.0) in the next release.

[aprochko]

Hi @mutianf, thank you for the feedback!

We used latest version of BigTable because of the bug of the version which was presented in Beam. Right now, I assume, there is no issues for us to use the supposed one.

I think I tested with bom 26.74.0 which includes Bigtable 2.71.0, maybe I did it wrong or proper gax updated was not included yet.

Thank you and looking forward to Beam 2.72.0 then :)

[github-actions]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.