Uh oh!
There was an error while loading. Please reload this page.
[BEAM-151] Work towards moving Dataflow pipeline runner to separate maven module - #87
[BEAM-151] Work towards moving Dataflow pipeline runner to separate maven module#87lukecwik wants to merge 14 commits into
Conversation
This allows for moving the Dataflow specific portion of the test to the Dataflow runner maven module.
This allows for moving the Dataflow specific portion of the testing to the Dataflow runner maven module.
This prevents moving the Dataflow runner code to its own separate maven module.
…rface This prevents moving DataflowPipelineOptions to the Dataflow runner maven module.
This prevents moving DataflowPipelineOptions into a Dataflow runner maven module.
This prevents moving DataflowPipelineOptions into a separate maven module.
Removed DatastoreIO dependence on numWorkers from DataflowPipelineWorkerHarnessOptions. This prevented moving DataflowPipelineOptions to a separate maven module.
This prevents moving DataflowPipelineRunner and subclasses to runner specific maven module.
This prevents moving DataflowPipelineOptions to its own runner specifc maven module.
lukecwik
commented
Mar 29, 2016
R: @dhalperi |
dhalperi
commented
Mar 30, 2016
Random attempt: |
dhalperi
commented
Mar 30, 2016
Jenkins, retest this please. |
lukecwik
commented
Mar 30, 2016
Merged in apache/master to force re-run of tests |
| * Properties that can be set when using Pubsub with the Dataflow SDK. | ||
| */ | ||
| @Description("Options that are used to configure BigQuery. See " | ||
| + "https://cloud.google.com/bigquery/what-is-bigquery for details on BigQuery.") |
davorbonaci
commented
Mar 30, 2016
R: @davorbonaci |
| /** | ||
| * Helpers for cloud communication. | ||
| */ | ||
| public class DataflowTransport { |
There was a problem hiding this comment.
Dataflow or Gcp?
I assume these are used outside of Dataflow Service, e.g., PubSubIO on Flink.
There was a problem hiding this comment.
Oh I see you only moved Dataflow and Clouddebugger. Why only these?
There was a problem hiding this comment.
Yeah, I left all the GCP IO transports in Transport.java and moved out to the two clients which we use only with Dataflow. In the future, cloud debugger could move out to be generic for others as well.
davorbonaci
commented
Mar 31, 2016
No particular comments from me. |
| * {@link DataflowPipelineRunner} specific tests for TextIO Read and Write transforms. | ||
| */ | ||
| @RunWith(JUnit4.class) | ||
| public class DataflowTextIOTest { |
There was a problem hiding this comment.
Remove; duplicate exists elsewhere.
davorbonaci
commented
Apr 5, 2016
LGTM (self-merge). |
dhalperi
commented
Apr 6, 2016
@lukecwik bumping that you should self-merge this PR when ready. |
Break out tests to Dataflow specific variants to make move of Dataflow pipeline runner to new maven module.
Also, swap usage of DataflowPipelineOptions for other options interfaces which were better suited. For example, creating PubsubOptions.